Nicholas Tacik

Doing Data Science on Jeopardy Data

Nick Tacik — Thu, 24 Sep 2026 00:00:00 GMT

Introduction

I like to think of myself as a bit of a Jeopardy aficionado. It’s a ritual for my wife and me to watch an episode as we sit down for lunch or dinner. However, there are definitely some categories that I’m a bit hopeless at — Opera, musicals, Civil War history, biblical names, architecture, and so on. I’d like to get better at these categories, but I’m not even sure where to start.

A hypothesis starts to form as you watch more and more Jeopardy.

There are not really that many total types of categories. The different category types get repeated frequently from game to game.
Within each category there are certain entities that get asked about far more frequently than others.
Those entities often have certain “fingerprints” - specific words used in the clue that signify what the answer is going to be. If I hear “Toreador” in an opera category, I’m betting the answer is “Carmen”, even knowing nothing more about operas.
If I can determine those categories, entities and fingerprints, I’ll have a good basis to study and hopefully get good enough to get on the show one day myself.

In this post, I’ll document my process for building a Jeopardy study tool: scraping the data from J-Archive, clustering the data into types of categories, reconstructing the most common entities in those categories (and the cleaning that goes into it), finding the clue fingerprints for those entities, and finally building the study tool itself.

Scraping the Data

J-Archive does not have an API, so the data comes from a polite, cached crawl.
A fetch stage that does one request per second with every page cached to disk.
A parse stage that reads each game’s board out of the HTML.
A resumable crawl that walks through every season and game.
This is assembled into a single clues.parquet file with one row per clue.
Ultimately, we find 563,266 clues across 9,485 games spanning from 1984 to 2026.

fig, axes = plt.subplots(1, 3, figsize=(13, 4))

per_year = clues["air_date"].dt.year.value_counts().sort_index()
axes[0].bar(per_year.index, per_year.values, color="#4C72B0")
axes[0].set_title("Clues per year")

gt = clues["game_type"].value_counts().head(6)[::-1]
axes[1].barh(gt.index, gt.values, color="#55A868")
axes[1].set_title("Clues by game type (top 6)")

rounds = clues["round"].value_counts()
axes[2].bar(rounds.index, rounds.values, color="#C44E52")
axes[2].set_title("Clues by round")
axes[2].tick_params(axis="x", rotation=20)

plt.tight_layout()
plt.show()

Figure 1: The dataset: clues per year, game types, and rounds.

Clustering in “Types of Categories”

To start clustering, we need to convert each category into a short “document”. The choice here is to take the category title, followed by its clue → answer pairs in a shuffled order (so no clue’s position carries special weight).
Each document is embedded into a 384-dimensional vector with a local sentence-transformer (bge-small-en-v1.5). This step is run offline and commits its output once.

Running a sentence-transformer locally

A sentence-transformer turns a chunk of text into a fixed-length vector where semantically similar texts land near each other. bge-small-en-v1.5 is a small, fast one (384 dimensions) that runs comfortably on a CPU. With the sentence-transformers library it’s essentially two lines:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
vectors = model.encode(documents, batch_size=64, show_progress_bar=True)

The model downloads once (~130 MB) and then embeds entirely offline — no API, no per-call cost. Encoding the ~123k category documents takes a few CPU-only minutes, which is exactly why it runs once and commits its output rather than re-running on every build.

Running K-means (k = 50) over those vectors groups the ~123k category instances into recognizable types. Projected to 2D with UMAP, the map looks like Figure 2. Hover any point to see its type, an example category, and that type’s most common entity. I chose the hyperparameter k=50, with a bit of experimentation. When k was too low, US presidents got merged together with Geography, but when it was too high, it was a bit too hard to interpret the meaningful differences between the categories.

UMAP

UMAP (Uniform Manifold Approximation and Projection) is a nonlinear dimensionality-reduction technique — it squashes the 384-dimensional embeddings down to 2D while trying to keep nearby points nearby, purely so we can see the structure. The clustering itself (K-means) runs on the full 384-dimensional vectors; UMAP is only for drawing this picture.

frac = min(1.0, 4500 / len(clusters))
samp = clusters.groupby("cluster_id", group_keys=False)[clusters.columns.tolist()].apply(
    lambda g: g.sample(frac=frac, random_state=42)
)
samp["type"] = samp["cluster_id"].map(labels)
top1 = tokens[tokens["rank"] == 1].set_index("cluster_id")["phrase"].to_dict()
samp["top_entity"] = samp["cluster_id"].map(top1).fillna("—")

fig = px.scatter(
    samp, x="umap_x", y="umap_y", color="type",
    hover_name="type",
    hover_data={"category": True, "top_entity": True,
                "umap_x": False, "umap_y": False, "type": False, "cluster_id": False},
    render_mode="svg", opacity=0.55,
)
fig.update_traces(marker={"size": 4})
fig.update_layout(showlegend=False, height=620,
                  margin={"l": 0, "r": 0, "t": 10, "b": 0},
                  xaxis_title=None, yaxis_title=None)
fig.show()

Figure 2: 123k category instances, grouped into 50 types (hover to explore; ~4.5k-point sample).

This methodology assumes a category can be neatly clustered, which isn’t always true. Picture a category whose only through-line is that every answer’s last name is “Johnson” — those Johnsons might come from politics, sports, and the arts all at once. Because the unit of clustering is the whole category (not the individual clue), such a category isn’t torn apart; it’s embedded as one document and assigned wherever its overall text points. And categories that don’t sit near any clean type aren’t dropped — the worst ~5% by distance to their cluster’s centre are swept into a synthetic 51st type, Misc: Hard to Classify.
The 50 types are also far from equal in size — Table 1 lists all of them, sorted by how many category instances each contains. The largest few (two flavours of wordplay, plus music, books, and movies) are also the broadest and most conceptually loose.
To name each of the 50 types, each cluster gets an automatic three-part summary — its most frequent real category names, the instances closest to its centre, and its most distinctive words (via TF-IDF over the pooled category text — see below). That summary is handed to an LLM (gpt-4o-mini), which returns a short 2–4 word name for each; the names are committed to a CSV the pipeline only reads, so nothing calls the model at build time.

How the “distinctive words” are scored

To find each type’s signature vocabulary, I pool all of a type’s categories into one big document — 50 documents, one per type — and run an ordinary TF-IDF (Term-Frequency–Inverse-Document-Frequency) over them. A word scores high for a type when it’s frequent inside that type (the TF part) but appears in few of the 50 pooled documents (the IDF part). So inaugural, which lives almost entirely in one type, floats to the top, while a word spread across many types gets downweighted — with scikit-learn’s smoothed IDF, log((1 + n) / (1 + df)) + 1, ubiquitous words are pushed down but never fully cancelled to zero.

(This is the plain-vanilla version. BERTopic popularized a “class-based” c-TF-IDF with a different reweighting designed exactly for this pool-per-class setup; I stuck with stock scikit-learn TF-IDF, which is simpler and good enough to read a name off of.) The same idea, pointed at a single answer instead of a whole type, is how the clue fingerprints below are built.

Table 1: The 50 category types, ranked by number of category instances, with example entities.

	Type	Categories	Example entities
0	Wordplay & Vocabulary	5064	Latin, Greek
1	Pop Music & Songs	4230	Beatles, Grammy
2	Books & Authors	4113	Pulitzer Prize, Charles Dickens
3	Rhymes, Anagrams & Common Bonds	4077	Shakespeare, God
4	Movies	3773	Oscar, Disney
5	Notable People & Awards	3760	Supreme Court, White House
6	Word Origins & Foreign Words	3669	Latin, Greek
7	Food & Drink	3426	France, Italy
8	U.S. Geography	3344	California, Texas
9	Animals & Zoology	3124	Australia, South America
10	Literature & Fictional Characters	3111	Charles Dickens, Moby Dick
11	World Geography & Waters	3100	Canada, South America
12	American History	2987	Union, Virginia
13	Television	2979	Emmy, Michael J. Fox
14	Potpourri / Grab-Bag	2911	Christmas, Guinness
15	World History & Leaders	2858	France, China
16	Museums, Landmarks & Travel	2828	London, Washington
17	Puzzle & Grab-Bag Categories	2825	Earth, God
18	Royalty & European History	2794	England, France
19	Oscars & Movie Quotes	2713	Oscar, James Bond
20	Names & Nicknames	2702	Oscar, Abraham Lincoln
21	Countries & Languages	2654	India, France
22	U.S. Presidents	2631	Senate, Richard Nixon
23	Sports	2425	World Series, New York Yankees
24	Medicine & The Body	2395	Latin, Greek
25	Law, Economics & Abbreviations	2222	Latin, Congress
26	Fashion & Furnishings	2186	Christian Dior, China
27	Religion & The Bible	2172	Jesus, God
28	World Capitals & Cities	2094	Canada, Paris
29	Explorers & Ships	2094	Canada, Christopher Columbus
30	Business & Brands	2067	Procter & Gamble, Johnson & Johnson
31	Chemistry & Earth Science	2050	Earth, Sun
32	Opera & Classical Music	2046	Mozart, Beethoven
33	Children's Literature	1936	Alice, Disney
34	TV Sitcoms & Families	1930	Emmy, Fox Broadcasting Company
35	Famous Women & First Ladies	1880	Susan B. Anthony, White House
36	Quotes & Proverbs	1825	God, Plato
37	Inventors & Scientists	1755	Nobel Prize, Thomas Edison
38	Mythology & The Ancient World	1704	Zeus, Rome
39	Theatre & Broadway	1605	Broadway, Tony Award
40	Poetry	1602	Robert Frost, John Keats
41	Plants & Botany	1598	Japanese, Christmas
42	Art & Artists	1592	Pablo Picasso, Leonardo da Vinci
43	Games & Play	1489	Olympic Games, Monopoly
44	Transportation & Cars	1349	Ford, Chevrolet
45	Astronomy & Space	1194	Earth, Mars
46	Holidays & The Calendar	1154	Christmas, Easter
47	Math & Measurement	1082	Earth, Celsius
48	Colleges & Universities	952	Harvard, William & Mary
49	Shakespeare	883	Hamlet, William Shakespeare

Where is the Study Material?

Wordplay seems to be a big, unusual slice. The two pure letter/anagram types - Wordplay & Vocabulary and Rhymes, Anagrams & Common Bonds - are 7.4% of all categories between them, and folding in Word Origins & Foreign Words brings the wordplay-and-language family to 10.4%, about one category in ten. It’s unusual because it rewards a skill you can drill through repetition more than the facts you memorize. It also splits into two tracks — pure letter games versus the genuinely studyable etymology and idioms.
Types differ quite a bit in how much recurring material they even offer — and if a type has little of it, my whole hypothesis about studyability falls apart. As a rough gauge, I count how many distinct phrases recur 5 or more times in a type, drawn from both clues and answers. It really is rough: the phrases aren’t cleaned yet (so stray first names, presenters, and near-duplicates are all still counted), and breadth isn’t payoff — a phrase repeating a lot doesn’t prove that studying it is worth your time.

eras = pd.read_parquet("category_eras.parquet")
eras = eras[eras["era"] == 1980]
ap = eras[["cluster_id", "n_qualifying_phrases"]].copy()
ap["type"] = ap["cluster_id"].map(labels)
ap = ap.sort_values("n_qualifying_phrases")
top = ap.tail(12); bottom = ap.head(8)
show = pd.concat([bottom, top])
plt.figure(figsize=(8, 9))
colors = ["#C44E52"] * len(bottom) + ["#55A868"] * len(top)
plt.barh(show["type"], show["n_qualifying_phrases"], color=colors)
plt.xlabel("distinct recurring phrases (count >= 5)")
plt.tight_layout()
plt.show()

Figure 3: Breadth of recurring material: distinct phrases (from clues and answers) recurring 5+ times, per type.

And for the material-rich types, here are the phrases that recur most, counted across clues and answers. These recurring names give me a concrete starting list to study — a ranked “if this type comes up, know these.” (They’re truncated mention counts, not a coverage calculation, so I won’t claim what fraction of a type’s clues they account for — just that they come up a lot.)

Table 2

	U.S. Presidents	count
0	Senate	363
1	Richard Nixon	315
2	Ronald Reagan	266
3	Abraham Lincoln	249
4	George Washington	241
5	Theodore Roosevelt	219
6	Harry Truman	211
7	Congress	211
8	White House	192
9	Gerald Ford	189

	World Geography & Waters	count
0	Canada	473
1	South America	345
2	Africa	293
3	Australia	260
4	Mediterranean Sea	228
5	Italy	197
6	Pacific	196
7	Europe	196
8	France	196
9	Atlantic	195

	Books & Authors	count
0	Pulitzer Prize	259
1	Charles Dickens	249
2	Ernest Hemingway	201
3	Mark Twain	180
4	Sinclair Lewis	167
5	John Steinbeck	151
6	Edgar Allan Poe	144
7	F. Scott Fitzgerald	126
8	Nathaniel Hawthorne	123
9	William Faulkner	123

Cleaning the Data

To actually make sense of the data, we need to clean it up. Hand-written regex rules get a lot of the way there, but there are a lot more judgment calls to make that require interpreting the context. Is “John” a studyable entity or a stray first name? John Adams is studyable, John isn’t. Is “May” the month or “Theresa May”? “Sarah of the Clue Crew” comes up a lot, but that’s just a clue presenter, not a studyable entity.
With this in mind, the lists were handed off to an LLM for a second look. For each type, it judged every candidate entity - keep, drop, or merge into a canonical form - with the type’s context in view. It dropped the noise (bare first names, nationalities like “British”, months, Clue Crew presenters, etc.) and stitched fragments back together - “Grey” + “Anatomy” -> “Grey’s Anatomy”. These verdicts live in a committed CSV that the pipeline only reads, so the render stays deterministic.
A second cleaning pass asks a sharper question - not just “is this a real entity” but “does it belong to this topic?”. This was able to pull valid but off-topic answers out of the wrong lists. For example, “New York City” is a fine entity, but it belongs in, say, Geography not in Books & Authors, so it no longer clutters the authors list.

The Study Tool

The study tool is live at research/. Pick a category type, see its most common (cleaned) entities, and jump straight to Wikipedia for each.
It’s filterable by a cumulative window - all-time or restricted to clues aired since 1990, 2000, 2010, 2020. This lets you see how a type’s recurring entities shift as you narrow to more recent years.
The Sample Clue button draws a real clue from that category. Optionally, if an entity is highlighted, it will draw a clue related to that entity.
Clue Fingerprints: For each entity, we look for the handful of terms that Jeopardy keeps reusing to ask about that entity, showing an N-of-M count of how many of that answer’s clues the term shows up in, plus real example clues. Tap a cue, and, where it could be pre-computed, a one-sentence note explains why that word points at the answer. The cues were found using the same TF-IDF trick from naming, but pointed at a single answer instead of a whole type. Filters do the rest of the work - drop the answer’s own name, throw out type-generic terms that show up across most of the type’s answers, and require a cue to appear in at least two of the answer’s clues.
For actual drilling, there’s a Practice mode: pick your topics and it deals you a run of distinct real clues, one at a time with the answer hidden, then a quick tally at the end. (Text-only, so it skips clues that lean on an image or audio you can’t see.)

Playing Codenames Duet with an AI partner

Nick Tacik — Wed, 09 Sep 2026 00:00:00 GMT

Play it

Open the game → — bring your own OpenAI API key. It stays in your browser and is sent directly to OpenAI; it never touches a server of mine.

What this is

Codenames Duet is a two-player, co-operative variation of the popular board game, Codenames.
In the game, there is a 5x5 grid of word cards. Each player also sees a hidden 5x5 key-card, specifying if each location is an agent (green), a bystander (beige), or an assassin (black). The goal of the game is to take alternating turns with your partner, giving each other clues, and trying to find all 15 of your agents together before the shared 9 turn timer runs out. Once all agents on one player’s key card have been found, the other player gives every remaining clue.
A clue is given by saying a number and a word. For example, if I want you to guess “Monkey”, “Elephant” and “Zebra”, I can say “3 Animal”.
After a clue is given, the partner starts guessing. Finding an agent lets them continue, touching a bystander immediately ends the turn, and touching an assassin immediately loses the game. My implementation currently caps guessing at words for a clue numbered , and allows the guesser to stop at any time.
If after all turns are used, there are still agents remaining, the game enters a sudden-death phase. Players may discuss a guessing strategy, and then guess words in any order according to their strategy. Getting all the greens results in a win, while getting anything else results in a loss.
Because of the key-card asymmetry, it’s possible that a bystander for me might be an agent for my partner, so I may want to guess that word in the future (therefore it’s not covered up upon a bystander guess).
There are some reasonable guidelines about what kind of clues are allowed - it must be one word, and it cannot be part of one of the other words visible on the table.
It’s a fun game to play but my wife is sometimes too busy for a game! So I thought playing with an LLM would be a reasonable way to get practice, and still have fun. It also gives me the chance to learn some of the basics about putting an LLM into production.
As the clue-giver, the LLM sees its own key card and has to describe its agents while trying to not touch a bystander or assassin. As the guesser, the LLM sees your clue and the board, and has to infer what you meant.

Interfacing with the LLM

I’ve decided to have players bring their own API key for simplicity. Requests go directly from the browser to OpenAI, so the key never passes through a server of mine. If you choose to remember it, the app stores it in the browser’s sessionStorage. I’ve also limited things to OpenAI for simplicity.
The model picker uses model-name filters to select families expected to support Structured Outputs, so compatibility is best-effort. Structured Outputs constrains the response to our JSON schema, but we still need to handle refusals, incomplete responses, and responses whose contents violate the game rules.
The calls to the API all look roughly like this

client.chat.completions.parse({
  model: selectedModel,
  messages: [
    { role: "system", content: instructions },
    { role: "user", content: currentGameInformation }
  ],
  response_format: zodResponseFormat(responseSchema, responseName),
  max_completion_tokens: 8192
});

In general, the “system” role is to set the context, tone, rules and boundaries of the LLM’s response, while the “user” role submits the immediate task.
Each instructions string starts with the rules, which also convey the goals of the game.

const RULES = `You are an expert cooperative partner in Codenames Duet.
You and your human partner each hold a DIFFERENT key card. You win together by
contacting all 15 agents (the union of both cards) before the shared turn timer runs out.
Two dangers: guessing a BYSTANDER ends the turn; guessing an ASSASSIN loses the game instantly.
If the timer runs out with agents still hidden, no more clues can be given — you both
make final guesses from the clues already given (sudden death), where a single wrong
guess loses. So use each clue turn to convey as much as you safely can while it lasts.`;

The instructions for the clue giver role continues with

export const CLUE_SYSTEM = `${RULES}

Your job now: give ONE single-word clue and a NUMBER for the agents you want your
partner to guess. Rules for a legal clue:
- exactly one word, no spaces or hyphens;
- must NOT be any word on the board, nor contained in one, nor contain one
  (e.g. do NOT clue "HERO" if SUPERHERO is on the board);
- must NOT be another form of a board word (e.g. do NOT clue "PIRACY" if PIRATE
  is on the board);
- the NUMBER must equal how many target words you list.

Strategy: you will be told which remaining words are your AGENTS, your ASSASSINS,
and your BYSTANDERS. Prefer familiar meanings and direct associations your partner
can recognise from the clue alone — not elaborate links that only make sense once
explained. Before committing, weigh every assassin and bystander against your clue:
if your partner could plausibly read the clue as pointing at one of them, or it fits
one as well as your agents, choose a safer clue. An assassin match is fatal and
outweighs any number of agents, so when a clue is even somewhat risky, cluing fewer
agents — or a different pair — is usually better. In "reasoning", briefly note the
main associations and any competing dangerous words, then commit to "clue", "number",
"targets".

Plan for the whole game, not just this turn. You have a limited number of clue turns
to get EVERY one of your agents found, so weigh safety against coverage: group several
agents under one clue when you can, and don't keep deferring a hard, isolated agent —
if the timer runs out with an agent that was never clued, your partner has no way to
find it. Near the end especially, before settling for a safe single-agent clue, check
whether another agent would then be left with no clue at all; a broader clue may be
worth the risk to avoid stranding it.

Example — board has APPLE, ORANGE, BANANA, KING; your agents are APPLE and ORANGE, and
BANANA is your ASSASSIN. "FRUIT" for 2 is tempting, but BANANA is just as much a fruit — a
fatal match. Better: find a clue that fits BOTH agents yet clearly not BANANA — APPLE and
ORANGE are round while a banana is long and curved, so "SPHERICAL" for 2 keeps both agents
and dodges the assassin. Only if no such separating clue exists should you retreat to a
safe single like "CIDER" for 1 (APPLE alone) — cluing fewer still beats risking the
assassin. Good answer: reasoning "SPHERICAL fits round APPLE and ORANGE, and a banana is
plainly not round, so my partner won't reach for BANANA", clue "SPHERICAL", number 2,
targets ["APPLE","ORANGE"].`;

The instructions for the clue guesser role continues with

export const GUESS_SYSTEM = `${RULES}

Your job now: your partner gave a one-word clue and a number. Choose which words on
the board they most likely mean, RANKED best-first. You do NOT see any key card —
infer from the clues.

Reading the history: each player has a different key card, and every outcome shown
refers to the card of whoever GAVE that clue. Your guesses this turn are judged
against your PARTNER's card — so look for leftover agents among your PARTNER's earlier
clues, never your own (your own clues described YOUR card). A word that ended a turn
as a bystander for one of you may still be an agent on the other card. Only 9 of the
25 words are agents on your partner's card; the other 16 are bystanders or assassins
(3 of them fatal), so an unclued word is far more likely to be a miss — guess only
what the clues actually support.

In "reasoning", briefly explain the main associations and any competing words, then
list "guesses" (exact board words).

How many to guess:
- The number is how many words THIS clue points to. Guess those, most-confident first.
- You may make ONE extra "bonus" guess (number+1 total), but ONLY for an agent you are
  confident your PARTNER pointed at in an EARLIER clue and left unfound — never to
  gamble on a loose association with the current clue.
- If you have no such confident leftover, STOP after the clue's number.
- A wrong guess ends the turn, and the assassin loses the game outright — so caution
  beats greed. When unsure, guess fewer.`;

For the clue giver, the currentGameInformation will look something like this. The turn count is the team’s shared timer; the two agent counts describe the clue-giver’s remaining agents and the total still needed to win.

Words still in play: [remaining board words]

Your agents ([agent count] left to get your partner to find, in ~[turn count] clue turns) — each is +1 toward the 15: [agents]
ASSASSINS on YOUR key card — [assassins]. A touch here loses the game instantly; reject any clue your partner
could plausibly read as pointing at one.
BYSTANDERS on YOUR key card — [bystanders]. A touch here ends the turn with nothing found;
avoid clues that fit one as well as your agents.
Turns remaining (shared timer): [turn count]. Agents still to find in total: [total remaining agents].
Make sure every one of your agents gets a clue before time runs out.

Game so far:
[formatted history]

Give your clue now.

The formatted history in the above will look something like this.

human clued "OCEAN" 2 -> WHALE=green, SHIP=bystander

For the clue guesser role, the currentGameInformation will look something like this. The bystander reminder is included only when earlier guesses have identified bystanders on the card currently being guessed against.

Words still in play: [remaining board words]

Your partner's clue: "FRUIT" for 2.
Already shown to be BYSTANDERS on the card you're guessing against — they CANNOT be agents here, so never guess them again: [known bystanders]
Turns remaining: [turn count]

Game so far:
[formatted history]

Make your guesses now: up to 2 for this clue, best-first —
plus a 3rd bonus guess ONLY if you're confident about an agent left over from an earlier clue.

zodResponseFormat is a function supplied by the OpenAI SDK. It converts our Zod schema into the JSON Schema response format sent to OpenAI. Using it with chat.completions.parse() also lets the SDK parse and validate the returned JSON against that schema. For clue-giving, the schema requires reasoning, the clue, the number and the targets.

const ClueSchema = z.object({
  reasoning: z.string(),          // Text explaining the choice
  clue: z.string(),               // The clue, as text
  number: z.number().int(),       // An integer
  targets: z.array(z.string()),   // A list of text strings
});

// ...and pass it to the API call as the response format:
response_format: zodResponseFormat(ClueSchema, "clue")

For clue guessing, we extract the guesses and the reasoning.

const GuessSchema = z.object({
  reasoning: z.string(),
  guesses: z.array(z.string()),
});

When sudden death begins, we ask the AI once for a ranked list of remaining guesses, each with a confidence score from 0 to 1. We reuse that list for subsequent AI guesses, skipping words already found. The human can make their own guess or ask the AI to make its next ranked guess after seeing its confidence score. These scores are the model’s own estimates: the code clamps them to 0–1, but they have not been calibrated against actual success rates. The idea is to mimic the strategy discussion that the rules allow for.

export const SuddenDeathSchema = z.object({
  reasoning: z.string(),
  guesses: z.array(z.object({
    word: z.string(),
    confidence: z.number()
  }))
});

Validating the LLM Response

We still have to check that the AI gave a valid response when giving a clue. The clue must be non-empty and contain no whitespace or hyphens. It cannot equal a board word, contain a board word, or be contained in one. This check includes all 25 original board words, even covered ones. The clue’s number must equal the number of targets, the targets must be distinct, and every target must be an uncovered agent on the current clue-giver’s key card. In the browser game, this is the AI’s card; in self-play, it depends on whose turn it is to give a clue.
If the clue passes validation, then it is passed on to the human. If it fails, the LLM gets a message saying that the response was illegal, for what reason, and to try again. If it fails three total times, the turn is passed.
It’s worth noting that we are using a simplified version of the validation process. HERO when SUPERHERO is on the board is invalid, but MAN when GERMANY is on the board is totally valid, so our substring approach is technically too rigid. An approach using lemmatization to establish word relationships, and escalating to an LLM referee when a verdict cannot be made might be a reasonable future approach.
The prompt also tells the AI not to use another form of a board word, such as PIRACY when PIRATE is on the board. Our validator does not detect those relationships, so that restriction currently relies on the AI following the instruction.
For AI guesses, we normalize the returned words, remove unavailable words and duplicates, and exclude words already confirmed as bystanders on the key card currently being guessed against. A bystander on the other player’s card remains a possible guess. We then apply the remaining guesses in order, stopping when the turn or game ends.

Running evals to calibrate the prompts

After ensuring that the API calls were working, it was time to check whether or not the AI could actually play well.
Essentially, this is kind of a new type of “debugging”, where we are assessing if the prompts we’ve given the AI are sufficient for it to actually achieve its goals. Can two AIs actually play the game together and win before the timer runs out?
I wrote a self-play evaluation script that lets AI control both players, using the same game engine, prompts and validation code as the browser version. The intended targets and the clue-giver’s explanation are recorded for debugging and analysis, but aren’t actually passed in to the guesser.
The script lets me choose the clue-giving and guessing models separately. A seed can be passed in to run the same game for consistency, although it by no means fixes the LLM’s responses.
For example, I can run:

export OPENAI_API_KEY="your-key"
EVAL_LOG=1 \
OPENAI_EVAL_CLUE_MODEL=gpt-4.1 \
OPENAI_EVAL_GUESS_MODEL=gpt-5.6 \
OPENAI_EVAL_SEED=MYSEED \
npm run eval -- 10 | tee /tmp/codenames-verification-trace.log

The evaluation log reports wins, agents found, assassin hits, turns used, rejected clue attempts, and how often games reach sudden death.
For simplicity, the evaluator currently stops when sudden death begins, so its win rate measures wins during normal play.
Initially, we went through a few passes with cheap models, which were always losing initially. This helped find some bugs in the structure of the logic of the self-play, and what information was being passed into the prompt. These bugs led to lots of wasted guesses, but were solved fairly quickly.
After these were fixed, we bumped up to GPT-5.6 for both roles to see how well things were going with a stronger model. What I found here was that the agents were giving solid clues, and not hitting the assassins, but it also wasn’t winning the games. In some cases, some of the words on the board were never even targeted. This told me the agents weren’t being strategic enough in their planning.
I added a reminder to plan across the remaining turns, group agents where possible, and avoid repeatedly postponing difficult words. I also made the remaining-agent counts explicit in the clue-giver’s input. After this change, the agents were able to win 3/3 boards, while previously just winning 1/3.
I then ran 10 evals on a new seed, and the AIs won 9 out of ten games, with the last game reaching sudden death. A pretty solid achievement I would say. Now this isn’t enough to statistically establish just how good the agents are, or how this depends on model, but these evals can start to get pricey, and I’m now satisfied that my goal of having a competent AI to play with has been satisfied. Or at least as satisfied as I can be from watching AIs engage in self-play.

Testing a couple games end-to-end

The above shows me that the agents can competently play with each other, but how does a game with a human go? Can they interpret human clues properly, and give clues that humans can understand? The clues given in the eval logs looked solid, but let’s step into a couple games and see.

Game 1:

Seed: rhino
Model: GPT-5.6 sol
Immediately I see a solid line with plane, cycle and car all being different modes of transportation. I give transportation 3, and the AI gets them. 3/15
The AI gives me prototype 2. My thoughts are you build a prototype in a lab, and a draft could be an iteration in the same way a prototype does. Draft is correct but lab is a bystander. 4/15
I now have litter, string, card, apple and lab as my remaining clues. I try kitten 2 as they are born in a litter and like to play with string. The AI gets both right. 6/15
The AI gives me regression 2. Well a linear regression is a line and it is also a data model, so I guess both of those. I suspect model was the remaining idea for prototype, so I stop guessing. 8/15
With apple, card and lab as my remaining three. I try Newton 2, to try to get the association with apple, and being a scientist working in a lab. The AI gets both right! 10/15
They give the clue surgeon 2. doctor is an obvious one and plastic fits with the the idea of a plastic surgeon. Both are right! 12/15
I just have card left, so I give 1 deck. 13/15
The AI gives acer 1. I actually had to look up that acer is the genus of the maple tree. Surely something like syrup or canada would have been better here? 14/15
Finally, the AI gives bedtime 1, a totally reasonable clue for night. Win!
Worth noting that based on my API token usage, this game cost about $0.10.

Figure 1: “First Game”

Game 1.1:

Seed: rhino
Model: GPT-4.1
I wanted to play the same game back with a different (older, cheaper) model, to see if there were any noticeable differences.
transportation 3 hits all three again. 3/15
The AI now gives me kit 2 as the second clue. With hindsight, I suspect that the association it was is model and plastic. 5/15
I give kitten 2 again, and it guesses litter and string again. 7/15
The AI gives me shift 3 this time. Night shift totally makes sense. Stick shift is also a thing. Stick is a bystander, so we’re now at 8/15.
I go back to Newton 2, and the AI gets them. 10/15
The AI gives me Sketch 2 this time. I suppose that’s a draft with lines. 12/15.
Deck 1 exhausts my agents. 13/15
The AI gives syrup 2. Maple is obvious, but now I have a previous clue relating to shift as well. Without hindsight I’m not sure it would be obvious that doctor was the remaining agent, closing out the victory.
Based on API token usage, this game cost about $0.02.
It does really seem like GPT-5.6 gave stronger, more specific clues, though also their acer clue could have been less esoteric.

Game 2:

Seed: platypus
Model: GPT-5.6 sol
This time the AI gives the first clue, and it is sole 2. fish and foot seem like great candidates. 2/15
I try hiking 2 to connect pole and park, and the AI catches on. 4/15
The AI gives obituary 2 as its second clue. Interesting clue - it’s obviously about a death, but it’s also a recap of the person’s life. I try death, then life, both correct as agents. 6/15
degree, parachute, carrot, microscope and state are my remaining 5 agents. I try thermodynamics 2 to associate state and degree, and, a bit to my surprise, the AI gets it right. 8/15
The AI gives ethics 2 as its third clue. code and fair both clearly fit the bill. 10/15
I just give skydiving 1 as a clue to get parachute, as I can’t think of anything to connect carrot to microscope without also hitting lab. 11/15
The AI gives me jacket 1 as its next clue. It takes me a while to figure it out, but I suppose a book jacket is its cover. 12/15
With 2 turns left, I need to connect carrot and microscope. I try botany 2 and it works out perfectly. 14/15
The AI gives me bunsen 1 as its last clue for lab. Another win!

Figure 2: “Second Game”

Game 2.1

Seed: platypus
Model GPT5.6 luna
Wanted to do another re-do game, but this time with a different lightweight model comparison.
I still get sole 2 as the first clue. 2/15
hiking 2 still connects pole and park. 4/15
Our first divergence - I now get mortality 2 as the clue connecting life and death. Equally valid in my opinion. 6/15
thermodynamics 2 still hits state and degree. 8/15
Instead of ethics 2, we now get computer 2 as the next clue, connecting lab and code. 10/15
I now do biology 2 to connect carrot and microscope, with lab off the board. 12/15
The AI gives me book 2 as its next clue, a great choice to connect fair and cover. 14/15
Skydiving 1 then seals the game.
We see some differences between sol and luna, but they both gave solid clues. Luna came out to about half the price.

Conclusions

This was a fun project! I learned lots of things about putting an LLM into production, setting up and debugging evals, writing good prompts and model cost/performance trade-off. Hope you play a game!

Navier-Stokes Solved?

Nick Tacik — Tue, 08 Sep 2026 00:00:00 GMT

It was announced today by OpenAI that they are sharing a solution to the Navier-Stokes problem. The Millennium Prize Problems are seven complex mathematical problems, selected in the year 2000, with each one representing a prize of one million US dollars to the first correct solution. The first prize was awarded in 2010 to Grigori Perelman for his proof of the Poincaré conjecture, although he chose to decline the prize money. The Navier-Stokes existence and smoothness problem, asks about the properties of the Navier-Stokes equations, which are essentially the governing equations of fluid mechanics. The full paper has been posted here. Now, I’m in no position to give any useful or interesting feedback on this result. But I’m more interested in talking about the apparent way in which the result came about.

The claims going around are that Tristan Buckmaster (NYU) and Levent Alpöge (Anthropic) had been making strides towards solving the problem themselves, and using codex to help their research. After word of their work reached OpenAI, an internal team was set on the problem with an “insane amount of compute”, by OpenAI’s own account. Buckmaster and Alpöge had been putting all of their drafts into Codex, and when Buckmaster asked whether the model had been trained on those sessions, he didn’t get an answer. OpenAI then offered him two deals: post their result and let OpenAI follow with Navier-Stokes the next day, or write up the Navier-Stokes result himself, crediting an internal OpenAI model, on the condition that Alpöge, who works at Anthropic, be removed from authorship. In exchange, OpenAI would say publicly that the pair “deserved the Clay Prize”. He declined both. When he said he’d go public, the reply was “Why would you ruin your career?” and, later, “If you don’t want me to be nice, then I don’t have to be nice.” Obviously, there’s a lot to read into here.

Figure 1: Tristan Buckmaster’s account of the call

Now if that is indeed an accurate version of what happened, the implications are profound. Imagine that every time a researcher is making big progress on an important problem, a big AI company takes notice of it and uses their work to push it foward with their much larger computational capability and take credit themselves. That could be the reality we are heading towards. In that case, is it time for university departments to start hosting their own private language models whose data privacy they can control? Are researchers going to have to start being a lot more secretive about their work, when academia is very much based around collaboration? There’s a lot to think about here.

Update: 24 Fields medal winners have signed an open letter, titled “A Severe Misalignment of AI in Mathematics”. In it, they note that “the push by AI companies to solve mathematical problems as a benchmark is detrimental to the science of mathematics, and to the mathematical community”. The reasoning presented is that “solving problems [is] only a tool and proxy for achieving the primary goal of conceptual understanding and insight”, while also noting that the solutions are announced in a rush. Personally, I am in full agreement with what they are saying here. The beauty of mathematics is in concepts, ideas and ways of understanding, not in specific results. Can AI ever produce these kinds of new concepts? We’ll have to see, I suppose.

Thoughts on the Data Center Backlash

Nick Tacik — Tue, 25 Aug 2026 00:00:00 GMT

A recent poll conducted by Embold Research found that 75% of Americans now oppose building data centers in their areas, including 61% voicing strong opposition. This finding carries across party lines, and opposition is quickly increasing with time.

Figure 1: Data Center Support Over Time

Users on Twitter posted furiously over the weekend in defence of data centers. All of the arguments I read in support of data centers strike me as very unconvincing, and ultimately I think people are going to have a very hard time building support for them if this is the level they are operating at. The arguments seem to break into these few main categories:

“This will be great for jobs and local economy…”
- While the 1-2 year construction phase of a data center might create 1000-1500 temporary construction jobs, afterwards we’re looking at ~50 permanent jobs. For something that costs billions of dollars, this doesn’t really seem like a lot of permanent jobs.
- These kinds of facilities also seem to get incredible amounts of subsidies, tax breaks, and discounted energy rates. A report from the watchdog group Reinvent Albany notes that the New York Power Authority supplied power to data centers in Rockland County at a rate of $12.88/MWh while the average New York household pays $244/MWh.
- People are keenly aware of the kind of deals that these facilities get, and without the permanent jobs behind them, the local economic argument does not seem to resonate.
“If you don’t like data centers, don’t use the internet…”
- A common refrain seems to be that if people are opposed to data centers, then they should avoid the internet altogether. After all, all their favourite apps - DoorDash, Twitter, online banking, Netflix, etc. - all use data centers in some way, and it is therefore hypocritical to use the internet at all. This strikes me as a very disingenuous argument, and assumes people are stupid. We’ve had all these things for years and years, while there has been a recent proliferation in data center construction that has been clearly driven by AI. Shifting the argument like this makes it seem like you are unwilling to defend AI, the thing that the data centers you want to defend are being built for.
Figure 2: Data Center Construction Spend Over Time
“If we don’t do this, we’ll fall behind China…”
- I honestly think this kind of sinophobic propaganda isn’t very effective any more. People are seeing China build mass transit, housing, and the kind of hyper-modern cities they dreamed of as children. Maybe people are okay with them leading the AI revolution. After seeing how American tech has evolved in the last 15 years, I think it’s reasonable for people to assume that more AI built in America will just lead to them being more heavily surveilled and price-gouged.
“Your environmental concerns are completely unfounded…”
- Even if we were to grant that all the concerns about water usage or pollution contribution are completely unfounded, I don’t think people are convinced.
- I think people have both local and global concerns, so let’s try to split them up.
- At the local level, for example, it’s easy to see news reports about how a community with a data center is experiencing awful constant noise. How is this going to affect their quality of life, their property values, the local wildlife? I don’t think they are getting any clear answers on this. What about the effect that the new excessive power demand is going to have on their power bills?
- At the global level, data centers consume ~2.5% of global electricity and this is expected to double in the next few years. While this isn’t necessarily a huge impact alone, people are left to ask if this is something we really need to be doing at a time when we drastically need to reduce emissions to curb climate change.
- Ultimately, I think people are willing to take on environmental harm when they see a clear benefit. Construction of a nearby coal power plant has almost double the support of a data center. I think this is because people ultimately see some benefit - energy generation, and blue-collar jobs. What benefit do they see from AI?
Figure 3: Support For Other Projects
“All the negativity against data centers is a Chinese psy-op”
- Even if that were true, you are still going to have to convince people, and simply saying they have been “psy-opped” isn’t going to do so.
“AI is going to destroy all the jobs and create a permanent underclass”
- Since we’ve established that data center proliferation is the same as AI proliferation, we can now talk about whether AI is good for people or not. People constantly hear this kind of rhetoric about how AI will take all the jobs and create a permanent underclass, not only from tech-bros, but also from the AI CEOs trying to promote the technology. Those CEOs are also constantly talking about how there’s a decent chance that some kind of super-intelligent AI will cause human extinction and we will be completely powerless to stop it. While the latter is easily read as a marketing gimmick, I think people really do clue in to the idea that AI is coming for their jobs. It’s the obvious reason why there is so much money behind AI - your boss replacing your job with an AI that costs a fraction of your salary is great for your boss and for the AI company, and terrible for you. We live in a society where the social contract essentially says you only get to live a tolerable life if you have a job, so it’s obvious that people will try to resist wherever they can to keep their jobs. No one wants to sign up to be part of a permanent underclass.
- It’s common for some people to argue that AI is just a fun little tool that can make little images, or write a song, or help you code. Why all the concern when it’s so benign? But again, people know that the cat is out of the bag - the vast amount of money behind AI is there because of the possibility that it will eliminate many jobs in the future; all the fun little tools are second order.
“Just like the industrial revolution, there will be new jobs”
- This is a common refrain we hear from people who are trying to quell the fears of job loss. However, we never hear any specificity about what these new jobs will be, how well they will pay, how secure they will be, etc. But to me the better question is this - if the AI is good enough to take your existing job, why should I expect that it won’t soon become good enough to replace this hypothetical new job? That’s ultimately what the goal of AI is - doing tasks as well, or better, than humans. If there is a new task that cannot ever be done by AI, then doesn’t that mean the AI has failed?
“Embrace the job loss, there will be UBI and shared riches”
- Another common refrain we hear from people who are trying to quell the fears of job loss is that actually this will be good because working sucks, and you will now be free to use your time fully as you see fit. This will be enabled by some kind of Universal Basic Income (UBI) funded by the gains of AI. Elon Musk, for example, is always tweeting about how AI will bring about “Universal High Income”. This all sounds great, but it leaves out the part of how we actually get there. Are these AI companies going to voluntarily give up their riches? Is the government going to force them to? I think people have good reasons to doubt that this will happen! It is always very suspicious when the greediest people in history who constantly lie start talking like Marxists. The other option, I suppose, is violent class war, but I don’t think your average person wants to sign up as a guerrilla fighter against what would hypothetically be the most powerful entity in human history.
“AI is inevitable, so just relent now please”
- I don’t think this is a very strong argument, and it reads more as bullying. These systems are not bound by physical law, and nothing is inevitable if we collectively decide to stop it.
“AI is curing cancer”
- The final point of the argument acknowledges that a lot of people rightly have existential concerns about the proliferation of data centers and thus AI, but argues that the amount of good that AI will do should greatly outweigh these concerns, and that we should be grateful for data centers. “AI is curing cancer” is a common talking point right now, with the implicit point being that curing cancer would indeed be a sufficient good to outweigh people’s concerns. These claims seem to be resting solely on the recent news that Moderna and Merck are making solid progress towards completing a phase 3 trial for a new skin cancer treatment that combines immunotherapy and mRNA vaccine technology. The news sent Moderna’s stock skyrocketing, and a headline that went around was that “AI played a key role in designing the vaccine”. But what does that actually mean? This work started development in 2017, long before ChatGPT existed. In fact, they use a machine learning pipeline to help with the per-patient bespoke workflow (see here for a nice overview). But this is the kind of machine learning algorithm that existed before LLMs, and doesn’t require a massive proliferation of data centers to exist - it’s very dishonest to conflate the two.
- Now LLM-style AIs have been making incredible progress in one intellectual area in particular - mathematics. They work great in the formal, structured environment that math provides, especially with the recent advent of proof formalization with Lean. Many novel proofs, improvements, and counter-examples have been constructed recently. However, I think it would be a hard sell to convince the average person that disproving the Erdős unit distance conjecture should make them happy about losing their job!

What’s up with terrible AI restaurant pictures?

Nick Tacik — Mon, 24 Aug 2026 00:00:00 GMT

I need to vent about something. I’ve been seeing so many posts lately with menu images from restaurants. Specifically, AI-generated menu images that look absolutely disgusting. Three typical examples are shown below.

Figure 1: “Any Protein”

Figure 2: “Shrimp Scampi”

Figure 3: “Tuna Bagel”

Restaurants have taken pictures of their own food, since the technology to do so has existed. It’s easy to do since they make the food anyway, and now that everyone has a high-quality camera in their pocket, this has never been easier.
Even if you don’t want to do that, here’s the thing. LLMs can now render images of food that actually looks appetizing, very quickly. Below is an image I generated with ChatGPT using the prompt “Give me an image for my restaurant menu of a delicious shrimp scampi. Include spaghetti, broccoli and a lemon wedge.” Doesn’t that look a lot more appetizing than something that seems like I’ll be eating bugs or worse?

Figure 4: “Better Scampi”

I just need to know why so many restaurants are using these menu pictures that immediately make me lose my appetite when they have so many reasons not to do so. Any theories? Maybe there’s a service that doordash (or equivalent) offers where they bulk generate these images for restaurants, and then they print them out for consistency?

Getting Cryptic with Claude

Nick Tacik — Thu, 02 Jul 2026 00:00:00 GMT

I really like this example, because it showcases a simple but practical use of AI in my everyday life, and I actually got a very beautiful solution that I wouldn’t have found without the use of AI. Every week, my parents and my brother get together and solve the Globe and Mail’s Saturday cryptic crossword puzzle. Being a couple provinces away, I can’t join in the fun, but they do send me a photo of the puzzle so I can try it myself. This leaves me with a puzzle of my own - how do I actually work on the cryptic given this photo?

Figure 1: Image of the Globe’s cryptic crossword

As a millennial, I don’t have a printer, so that’s out.
I tried just doing it in an image editor, but it felt very inconvenient. I needed to find a better solution.
Then I realized this would be a good chance to leverage the power of Claude Code. Initially, I prompted Claude to turn the image into an editable google doc, but this was still quite janky to use. The superscript of the clue text was wonky in the same cells as the text from the regular puzzle text, and every time I put in letters, the relative position of each cell shifted. This didn’t seem like a great approach.
At this point, I remembered how good Claude is at making static html webpages. Within a matter of minutes, it spun up a very nice looking webpage (click here!).
It set up the grid, had the clues nicely highlighted, basically everything I wanted for the baseline of a crossword front-end. As someone who has not done much front-end design myself, I really appreciate how easy and quick it is for Claude to get something like this running.
Now I’m no expert at doing cryptic crosswords myself - the clues are tricky, and there are a lot of rules that you need to learn. So at this point, I wanted to put in a hint and reveal system. This was actually the hardest part for Claude, and it took it about 10-15 minutes to solve the crossword itself (a necessary requirement for it to be actually able to give clues). After doing so, we settled on a 2 stage hint system.
I sent the puzzle to my wife to test out the design, and her immediate feedback was that on her phone, the html’s functionality was really wonky. Things would come in and out of focus in a very un-intuitive way, the screen would jump around, etc. This actually took a fair bit of iteration, as Claude seemed to have a bit of trouble simulating a phone-screen environment. The repeated clue locking above the keyboard was a nice touch for mobile navigation.
Finally, I built a Claude skill so that I would be able to invoke this workflow as desired any time in the future. The skill asks where to save the html, dispatches a subagent that detects the grid, transcribes clues, solves the puzzle, writes two-tier hints and runs a builder, then reports back a spoiler-free summary.

Figure 2: Screenshot of the Mobile page

Hello, world

Nick Tacik — Sat, 27 Jun 2026 00:00:00 GMT

This is the first entry. This site is simply a working notebook for myself. Short writeups of things that I’m building, thoughts and musings, maybe Kaggle logs, maybe notes about books, papers or courses.

If you wish, you can contact me at nicktacik@gmail.com .