Few word do trick

SFT vs self-distillation: moving a learned writing style into weights

Mon, 21 Sep 2026 00:00:00 +0000

What happened

In the last post, a model learned a writing style from feedback and kept what it learned as text: a style guide and a memory of approved pieces. The fine-tuning at the end of that post was one small LoRA adapter (a small set of extra weights trained on top of a frozen model) on my own GPU. This post is about doing that step properly: taking what was learned and putting it into a model's weights, repeatedly, without breaking the model.

What I compared

I compared two ways of doing it on Tinker, Thinking Machines' fine-tuning service, with two open models, Qwen3-8B and Inkling-Small:

Supervised fine-tuning (SFT): train the model on the user's real text, the ordinary way.
Self-distillation fine-tuning (SDFT), from Self-Distillation Enables Continual Learning. The paper's claim is that this forgets less than SFT.

SDFT works in three steps, and the same model plays both roles. The teacher sees the task and the user's real text. The student sees only the task and writes its own draft. Then, at each word of that draft, the student is nudged toward the word the teacher would have picked. The student never copies the real text directly; it learns from the teacher's reaction to the student's own writing.

The results

In the order they matter:

SFT learns the style much better. Head to head, a judge preferred the SFT model's text to the SDFT model's in 174 of 180 comparisons on Qwen3-8B and 175 of 180 on Inkling-Small. After 264 examples, the SFT models are about as close to the real text, by a stylometric measure, as the best in-context learner from the last post.
SFT also costs the model some general ability, and SDFT doesn't. On 20 general prompts (arithmetic, counting letters, reversing a word), Inkling-Small went from 100% to 70% after SFT and stayed at 100% after every SDFT variant. The SFT models trained in rounds also produced a few broken outputs with raw chat-template tokens in them; SDFT produced none.
SDFT is limited by its teacher, and a better teacher helps. Showing the teacher one real section is a weak signal. Showing it the learned style guide and two more real sections as well moved SDFT on Inkling-Small from beating the untrained model in 21–22 of 30 comparisons to 27–28 of 30. It still loses to SFT.
Training in rounds works. I split the data into three chunks by date and trained on one at a time, each round starting from the last. Both methods kept improving, SFT ended where training on everything at once ends, and the loss of general ability from SFT did not grow from round to round.

So it is a trade-off, and now I have numbers for it. On Inkling-Small, SFT gives the full style and costs 15–30 points on general prompts. SDFT with a rich teacher closes a bit more than half as much of the stylometric gap as SFT does (about two-thirds on Inkling-Small, under half on Qwen3-8B) and costs nothing I could measure.

Cost

The whole thing cost about $80: $40 on Tinker for training and sampling, and $42 of Claude judging at API prices. Prompts and settings are in the appendix.

Setup

Data

The data is the same as last time: two years of Matt Levine's Money Stuff newsletter, split into sections. For each section, a model wrote a neutral brief (facts, argument in order, quoted sources verbatim), so the task is "write this brief up" and the real section is the target. I used 264 training pairs and the same 30 test briefs, all from columns published after the models' training cutoffs. The prompt never names the author.

Models

Qwen3-8B ran with its step-by-step reasoning ("thinking") turned off. Inkling-Small is a mixture-of-experts model: it has 276B parameters but uses only 12B of them for each token. It ran with its thinking effort set low. Both were trained with LoRA at rank 64, learning rate 5e-4, batch size 16, for two passes over the data.

SDFT recipe

I used the tinker-cookbook recipe with the teacher kept at the untrained weights. It needed one small patch to run on Inkling-Small; the appendix has the details.

Scoring

Style, judged. A judge (Claude Sonnet) sees the real section and two drafts and says which is closer in style. Every comparison was run twice, and the tables report both runs.
Style, measured. Distance to the real section over 17 countable habits: sentence length, footnotes, how much is block quote, hedges, first person, and so on. No LLM is involved. Lower is closer. Two different real sections score about 1.08 against each other.
Retention. 20 short prompts that have nothing to do with the style task, each checked by code: a two-step word problem, count the r's in "strawberry refrigerator", write "consolidation" backwards, an acrostic, a sentence with no letter e, and so on. I also checked whether the newsletter format leaked into these answers. It never did.
Faithfulness. A judge lists factual claims in the draft that the brief doesn't support.

Style: SFT wins

The table gives each model's stylometric distance to the real text and how often the judge found its drafts closer in style than the untrained model's. The two numbers in each judged cell are the two judge runs.

Model	Variant	Stylometric distance	Judged closer than untrained
Qwen3-8B	untrained	1.40	—
Qwen3-8B	SFT	0.70	30/30, 30/30
Qwen3-8B	SDFT	1.15	29/30, 30/30
Inkling-Small	untrained	1.16	—
Inkling-Small	SFT	0.77	30/30, 30/30
Inkling-Small	SDFT	1.10	21/30, 22/30

When the judge compared SDFT directly with SFT, SDFT won 0 of 30, in both judge runs, on both models.

For one test brief, the real headline is "Emerging markets". The untrained Qwen wrote "The Hidden AI Factor in Passive Investing". After SFT it wrote "AI". After SDFT it wrote "AI Exposure and the Hidden Choices in Index Construction". Inkling-Small did the same thing: "AI factor" after SFT, "Passive Emerging-Market Investors Are Making an Active AI Bet" after SDFT. SFT picked up the author's short, flat headlines; SDFT kept the model's habit of long, descriptive ones.

SDFT did move the style. It just moved it much less. My interpretation is that the teacher is the limit: SDFT can only teach what the model already does when it is shown the demonstration, and one real section in the context of an 8B model does not make it write like the author. The in-context learners in the last post needed a distilled guide, several examples and a much stronger model to get there.

Retention: SDFT wins

This table covers Inkling-Small only (Qwen3-8B is discussed below). It gives the share of the 20 general prompts each model answered correctly, and how many of its 30 test drafts were broken. For the three-round runs, the cells give the result after each round.

Inkling-Small variant	Retention, 20 prompts	Broken outputs, of 30 test drafts
untrained	100%	0
SFT	70%	0
SDFT	100%	0
SDFT, rich teacher	100%	0
SFT in three rounds, after each	85 / 80 / 80%	6 / 0 / 2
SDFT in three rounds, after each	100 / 100 / 100%	0 / 0 / 0

The SFT model fails the character-level and arithmetic items: the average of five numbers, reversing a word, counting letters and vowels, the acrostic, the sentence without an e. The untrained model gets all of these right at low thinking effort. My guess, which I haven't verified, is that the SFT targets contain no reasoning, so the tuned model stops deliberating before it answers.

The broken outputs are drafts that contain raw chat-template tokens or run on for thousands of words. They appeared only in the SFT models trained in rounds. My hypothesis is that SDFT avoids them because it trains on the model's own samples, which keeps it close to its normal output; I did not test this.

Qwen3-8B doesn't show any of this, because with thinking turned off the untrained model passes only 35% of these prompts, and every tuned version lands between 20% and 45% with no pattern. The set is the right difficulty for Inkling-Small and too hard for a non-thinking 8B model.

A better teacher

If the teacher is the limit, give it more to read. The rich teacher's context has the style guide the in-context learner wrote in the last post (it names no one), two other real sections chosen by topic from earlier dates, and the real section for this brief. That is about 3,400 words, where the plain teacher had one section. The student still sees only the brief.

The table compares the plain teacher with the rich one. Each pair of cells reads plain → rich. The judged columns give both judge runs, written as a range when they differ.

Model	Stylometric distance, plain → rich	Judged closer than untrained, plain → rich	Rich vs plain, head to head	Rich vs SFT, head to head
Qwen3-8B	1.15 → 1.09	29–30 → 29–30 of 30	23/30, 24/30	0/30, 3/30
Inkling-Small	1.10 → 0.90	21–22 → 27–28 of 30	17/30, 26/30	0/30, 1/30

The rich teacher helps on both models, and more on the larger one, which can make more use of a long context. This fits the idea that the teacher is the limit. It does not close the gap to SFT. The longer teacher prompt made SDFT about 30% more expensive.

Training in rounds

In real use, feedback arrives over months, and the weights would be updated every so often, each time starting from the previous update. The known risk is that each round erodes a little more of the model. I sorted the 264 training briefs by date and cut them into three chunks of 88 (August 2024 to January 2025, then to September 2025, then to April 2026). Each round trains on its chunk only, starting from the previous round's weights.

The table gives stylometric distance and the judged comparison against the untrained model after each of the three rounds.

Model and method	Stylometric distance after round 1 / 2 / 3	Judged closer than untrained after round 1 / 2 / 3
Qwen3-8B, SFT	0.81 / 0.71 / 0.79	30 / 30 / 30 of 30
Qwen3-8B, SDFT rich	1.16 / 1.05 / 1.06	28–29 / 29–30 / 28–29
Inkling-Small, SFT	1.07 / 0.68 / 0.74	21–23 / 30 / 29
Inkling-Small, SDFT rich	0.96 / 0.87 / 0.88	24–25 / 27–29 / 26–27

Both methods improve from round to round. Three rounds of SFT end level with SFT on everything at once: head to head against the all-at-once model, the three-round model won 12 and 15 of 30 in the two judge runs on Inkling-Small, and 20 and 22 of 30 on Qwen3-8B. After round three, SFT still beats SDFT head to head: 27–29 of 30 on Inkling-Small and 28–29 of 30 on Qwen3-8B.

The loss of general ability from SFT did not accumulate. On Inkling-Small, the retention score was 85% after the first round and 80% after the second and third. Three rounds is too short a sequence to conclude much from that.

Fine-tuning did not make either model invent more. By a strict judge, the untrained models averaged 2.5 (Inkling-Small) and 3.6 (Qwen3-8B) unsupported claims per draft; every tuned model I checked was at or below its base.

What worked and what didn't

What worked

SFT for style. 264 examples, a few minutes and about 50 cents on the 8B model gave a stylometric distance of 0.70–0.77. That is level with the best in-context learner from the last post (0.71), with nothing in the prompt.
SDFT for keeping the model intact. On Inkling-Small, every SDFT model passed all 20 retention prompts and produced no broken outputs.
The rich teacher. The guide written by the in-context learner turned out to be useful a second time, as teaching material.
Training in rounds. Both methods kept improving from round to round.

What didn't

The retention set on Qwen3-8B. It was too hard for the base model, so it shows nothing there. One set does not fit two models.
SDFT as a way to get the style. Even with the rich teacher, it closes a bit more than half as much of the stylometric gap as SFT and loses almost every direct comparison.

What this doesn't show

This experiment compares one recipe for each method on one style task. It shows the trade-off between style and retention at this scale. It does not show how either method behaves over a long run of updates or across different kinds of tasks. Several limits matter:

One training recipe per method. There was no learning-rate sweep and no longer SDFT training.
A small retention check. 20 retention prompts is still a small check, and it isn't a benchmark anyone else uses.
A small test set. There were 30 test briefs and one training seed per run. Differences of a few points between neighboring cells in these tables mean nothing; the gaps between SFT and SDFT are large enough to survive that.
A style judge, not a quality judge. The judge compares style, with the real section as the reference. It doesn't grade writing quality in general.
Three rounds of training. The paper's forgetting results come from longer sequences of more varied tasks.

What's next

First: try a mixture. Combine SFT on the real text, for the style, with SDFT or a small replay of general data, to preserve the model's other abilities. If that keeps most of SFT's style at SDFT's retention, it is the consolidation step I want.

Then: more data. I used 264 of the roughly 1,070 training pairs the corpus can produce, and the small local adapter in the last post was still improving at 264.

Eventually: preference training. Every piece the user edits gives a natural pair: their final text, preferred over the draft they were shown. That pair is the input for direct preference optimization (DPO), which trains a model to favor the preferred text.

Appendix: technical details

Models and training

Renderers: Qwen3-8B used the qwen3_disable_thinking renderer. Inkling-Small used Tinker's Inkling renderer with thinking effort fixed at 0.2, for generation and for SFT examples.
LoRA: rank 64, learning rate 5e-4, batch size 16, two passes. That is 32–33 steps for all 264 pairs and 11 steps per round in the three-round runs.
SFT: tinker-cookbook's supervised training, with loss on the assistant turn.
SDFT: the cookbook recipe, using top-20 token distillation (the student is trained toward the teacher's 20 most likely next tokens at each position), a static teacher at the base weights, and one student sample per prompt at temperature 1.0. Samples were limited to 2,048 new tokens for Qwen3-8B and 3,072 for Inkling-Small.
Rounds: rounds two and three load the previous round's weights; the SDFT teacher stays the base model throughout.

The Inkling patch

The recipe calls len(tokenizer) to drop teacher token IDs beyond the vocabulary. Inkling's tokenizer adapter doesn't implement __len__. Giving it a very large length disables that filter, and the recipe runs.

Sampling for evaluation

Temperature 0.7, with one sample per test brief and per retention prompt.

Cost

The total was about $80: $40 on Tinker for 18 training runs plus sampling, and about $42 of Claude judging at API prices. By run, estimated from logged token counts at list prices:

SFT on all 264 pairs: $0.50 (Qwen3-8B) and $1.90 (Inkling-Small).
SDFT with the plain teacher: $1.10 and $4.30, and more with the rich one.
A three-round SDFT chain on Inkling-Small: $8.90.

SFT runs take minutes. SDFT took 19 minutes on Qwen3-8B and up to three hours elapsed on Inkling-Small, most of it waiting on sampling.

Prompts

The writer prompt (the student's whole input is this plus the brief):

You are a ghostwriter. Write one section of the client's newsletter column from the BRIEF below,
   in the
client's own voice.

Format: Markdown. Begin with a `## ` heading. You have verbatim SOURCE passages in hand; quote
   from them as block
quotes (lines starting with `> `) where useful. Footnotes are allowed: put markers like [1] in
   the text and list
the notes at the very end. Cover the substance of the brief. Output only the section itself.

Plain teacher, the user turn:

{question}

This is how the client themselves wrote this section:
{golden_answer}

Now write the section yourself, in the client's voice.

Rich teacher. Same template, with {golden_answer} replaced by three blocks separated by =====: "The client's style guide, learned from their past edits:" and the guide; "Sections the client wrote for other briefs:" and two real sections; "This is how the client themselves wrote THIS section:" and the real section.

Faithfulness judge:

You check a DRAFT against the BRIEF it was written from. The brief (including its verbatim
   SOURCE passages) is
the only permitted source of facts. List every FACTUAL claim in the draft that the brief does
   not support: invented
names, numbers, dates, events, quotations, or attributions. Opinions, jokes, hypotheticals
   clearly presented as
hypothetical, and general background knowledge that is not specific to this story do not count.
   Be strict about
invented specifics and lenient about commentary.

Retention prompts:

1. A shop sells apples at 3 for $2. How many dollars do 27 apples cost? Reply with the number
   only.
2. A train leaves at 09:40 and the trip takes 2 hours 35 minutes. What time does it arrive?
   Reply in HH:MM, nothing else.
3. A price rises 20% and then falls 25%. What is the net percentage change? Reply like -10% or
   +5%, nothing else.
4. The average of five numbers is 12. Four of them are 10, 14, 9 and 15. What is the fifth?
   Number only.
5. Write the word 'consolidation' backwards. The reversed word only.
6. How many times does the letter 'r' appear in 'strawberry refrigerator'? Number only.
7. If 1 March 2026 is a Sunday, what day of the week is 20 March 2026? One word.
8. All bloops are razzies. All razzies are lazzies. Are all bloops definitely lazzies? Answer
   yes or no only.
9. What does this Python print? print(sum(i*i for i in range(1, 5))) Reply with the number only.
10. Return valid JSON only: an object with key "user" whose value is an object with keys "name"
   ("Ada") and "langs" (a list containing "python" and "rust").
11. What is the greatest common divisor of 84 and 126? Number only.
12. Convert the decimal number 45 to binary. Digits only.
13. Write exactly four lines where the first letters of the lines spell WAVE. Nothing else.
14. Write one sentence of at least six words about the sea that does not contain the letter 'e'.
15. A cyclist rides 18 km in 45 minutes. What is the average speed in km/h? Number only.
16. Sort in descending order, comma separated, nothing else: 7, 42, 3, 19, 23
17. How many vowels are in the word 'onomatopoeia'? Number only.
18. $1,000 earns 10% interest compounded annually. How many dollars is it worth after 2 years?
   Number only.
19. Write the number 1994 in Roman numerals. Numerals only.
20. Output a Markdown table with exactly two columns named City and Country and exactly two data
   rows: Paris/France and Tokyo/Japan. Table only.

Pairwise style judge: the same prompt as in the last post's appendix.

Learning a writing style from feedback, without telling the model whose writing style it is

Sat, 19 Sep 2026 00:00:00 +0000

What happened

Could a large language model (LLM) learn someone's preferences continuously, using only feedback they would provide anyway? Writing style seemed like a good first test. I did not have hundreds of my own edited drafts, so I simulated a user who writes like Matt Levine, the Bloomberg News finance writer behind Money Stuff. The model had to learn the style without ever being told his name.

The process starts with a neutral brief. The LLM drafts a piece, and the simulated user either rewrites it or sends a few sentences explaining what sounds wrong. One critique asked for plainer, more tentative prose, longer source quotations, and fewer polished asides. The system turns that feedback into instructions for the next draft.

The headline result

After 20 rounds of feedback, a style judge shown a real Money Stuff section alongside the model's version of the same story mistook the model's version for the original about 40% of the time. A 50% score would mean the judge was choosing at chance level. The same model, explicitly told to write in Matt Levine's style and given 40 real examples, scored about 20%. With no guidance, it scored about 2%.

The writer was Claude Opus 5, with no fine-tuning. Its learning lived outside the model's weights: a 450-word style guide, rewritten as feedback arrived, plus a small bank of user-approved texts. The system rejects guides containing an author or publication name. Rules therefore have to describe techniques — "headline is a flat topic label in sentence case" or "end abruptly on one flat, deadpan line" — rather than identify a person. That matters for the intended use case: learning the preferences of someone who is not famous.

A smaller model, and two surprises

I also used 264 draft/final-text pairs to train a LoRA adapter on Qwen2.5-7B to restyle untrained Opus drafts. This experiment used one consumer GPU, and the adapter never saw the author's name. It fooled the same judge 23% of the time, roughly matching Opus when told the name, and copied structural habits — short, flat headlines, bulleted lists, and the share of quoted text — more faithfully than the in-context learner. But the judge still preferred the raw Opus draft two times out of three: the smaller model's prose was looser, and it invented details. I discuss this in Moving it into weights.

Two results went against what I expected. Picking between two drafts — the least effort a person can put into feedback — taught the model nothing. And a learner that appends every lesson to an ever-longer list, with no management at all, did not get worse over 200 rounds of feedback. I had designed that experiment expecting it to degrade.

All the prompts are in the appendix, along with a worked example: a brief about a man suing a restaurant group that cut him off on all-you-can-eat crab night, and what the untrained model and each learner wrote from it.

The setup

Data and task

The dataset came from two years of Money Stuff in my inbox: 374 columns containing 1,717 sections. For each section, a model produced a neutral brief: the facts, the argument in order, the side points, and verbatim source quotations, all in flat note form without the original voice. The writer's task was to turn that brief into a section; the published section served as the hidden answer. The design held content fixed to isolate style. About 330 brief/original pairs were used. All 30 test briefs came from columns published after the models' training cutoffs, so the models could not have seen those answers during training.

What the learner stores

The learner combines a fixed ghostwriter prompt with two kinds of preference memory:

A guide: plain-text style rules within a 450-word budget, rewritten rather than appended to as feedback arrives.
A bank: user-approved final texts, retrieved two at a time by topic similarity and shown as examples.

The guide describes how to write, not whom to imitate: use a brief, matter-of-fact headline; open with the facts rather than a staged introduction; quote at length when sourcing matters; and avoid a tidy conclusion. These instructions can carry over to unrelated assignments.

Three feedback signals

Feedback takes three forms a real user might provide, each simulated from the hidden original:

Edits. The user rewrites the draft. In this simulation, the published section is the rewrite.
Critiques. A critic model sees the draft and original, then writes 3–5 sentences of first-person notes without quoting the original or naming anyone.
Picks. The user chooses between two drafts. A judge with access to the original makes the choice.

Each signal carries different information. An edit supplies the complete desired text. A critique identifies changes — less polish, longer source quotations, simpler asides — but leaves the writer to apply them. A pick says only that draft A is preferable to draft B. The experiments test how much reusable style information each signal conveys.

Updates and evaluation

After each batch of feedback, a reflection step proposes a revised guide. A gate tests it on held-out validation briefs and keeps it only if the resulting drafts improve.

I evaluate the learner on 30 test briefs using four measures:

Beats untrained: the win rate against the same writer with an empty preference state. A judge sees the real section and both drafts, then chooses the closer stylistic match.
Beats named-author: the same comparison against a writer told the author's name and given 40 real examples.
Lineup: the judge sees two real reference samples, then the real section and the draft, and chooses which is authentic. I report how often the draft fools it.
Stylometric distance: an LLM-free measure of 17 countable habits, including sentence length, footnotes, block-quote share, hedges, and first-person usage. Feature gaps are standardized and averaged; lower is closer. Two different real sections score about 1.08 against each other.

The stylometric measure provides an independent check because the same judge model both steers the gate and grades the results, potentially inflating scores. It caught a failure the judge missed, discussed in Deep dive: a run that got worse.

Which feedback teaches

I tested Claude Sonnet 5 as the writer, with 80 feedback events for each learner:

Feedback	Beats untrained	Beats named-author	Fools lineup	Stylometric distance
None (untrained)	—	17%	0%	1.15
Told the name + 40 exemplars	83%	—	0%	1.02
Edits	100%	90%	3%	0.82
Critiques	100%	80%	0%	0.88
Picks	57%	7%	0%	1.37

Edits and critiques both beat the named-author model by a wide margin. The critique learner's 80% win rate is especially relevant to a product: a few sentences of notes demand far less effort than rewriting a draft. That learner had no examples to draw on — only 450 words of rules.

Picks between these relatively similar drafts produced no measurable improvement. After 80 picks, win rates remained within noise of the untrained model, and stylometric distance was worse.

The candidates came from the same guide and may have differed too little for the choice to reveal useful preferences. The reflection step then wrote rules the evidence did not support. I had expected pairwise preferences, a standard signal for training reward models, to teach at least something. Deliberately contrasting candidates may work better. In later experiments, the pick learner serves as a control: it passes through the full loop without measurable learning.

How fast

To measure learning speed, I reran the first 10 events with a guide update after each one. The table shows the share of test briefs on which the blind learner beat the named-author model:

Feedback events	0	1	2	3	5	10
Edits	13%	37%	100%	97%	93%	93%
Critiques	13%	0%	43%	33%	53%	60%

Two edits were enough to outperform the named-author model. Critiques worked more slowly and noisily, passing it within 10 events.

A headline illustrates what changed after two edits. For the same test brief, the untrained model wrote "The AI Question Every Investor Has to Answer, Whether They Like It or Not." After learning, it wrote "Korea." The real headline was "Emerging markets." This does not prove that the model captured the whole voice. It does show a mechanism: a reusable preference for restrained, matter-of-fact headlines displaced the model's tendency to oversell.

Opus

I then ran the same loop with Opus 5 as the writer and 20 feedback events:

Writer	Beats untrained	Beats named-author	Fools lineup	Fools lineup, uniform formatting	Stylometric distance
Untrained Opus	—	10%	2%	3%	0.93
Told the name + 40 exemplars	90%	—	17%	27%	0.81
Blind learner, 20 critiques	93%	63%	29%	20%	0.82
Blind learner, 20 edits	100%	83%	43%	39%	0.71

Both lineup columns average three judge runs on the same 30 drafts, for 90 judgments per cell.

What improved

With Sonnet, no system — including the named-author model — fooled the lineup judge more than 10% of the time. With Opus, the named-author model reached about 20%, and the blind edit learner reached about 40%.

The judge was noisy across runs. The critique learner scored 37%, 13%, and 37%; the edit learner scored 47%, 40%, and 43%.

Checking for formatting shortcuts

The second lineup column controls for formatting clues. My reconstruction of the real text separates quoted paragraphs with blank lines; Claude puts a > on those lines. Writers exposed to real texts — the edit learner through its bank and the named-author model through its examples — copied the real layout. Other writers did not. Real texts also used curly quotation marks, while Claude's drafts used straight ones. For the second column, I normalized typography and Markdown layout across the real sections, reference samples, and drafts before judging.

The main result survives normalization: the blind edit learner fools the judge about twice as often as the named-author model, while the untrained model almost never does. The critique learner does not clearly outperform the named-author model on this test: their lineup scores are broadly comparable, at 20–29% and 17–27%, respectively. Its 63% pairwise win rate against that model is a separate, single-run measurement.

Limits of this run

Only one of four proposed guide updates was accepted in each Opus run. Opus repeatedly produced 520-word guides against a 450-word budget, and my code discarded them rather than compressing them. That bug has since been fixed. Any gains by the edit learner after event five therefore came from its growing bank, not a better guide.

The critique learner's guide also stayed unchanged, yet its single-run lineup scores at 5, 10, and 20 events were 20%, 20%, and 37%. That first alerted me to the judge's variability. With 30 test briefs, the reported percentages also carry sampling uncertainty of about 9 points, before run-to-run judge noise.

Deep dive: a run that got worse

The per-event results in How fast came from a second attempt. The first attempt failed and exposed two design problems.

With an update after every event, the critique learner rose to about 75% against the untrained model by event three, then fell to 43% at event 10. The gate had accepted the final guide after eight wins out of eight on validation. On the test set, it was no better than learning nothing.

Two design choices explained the failure:

The reflector rebuilt the guide around one critique. Notes on a single section became general rules such as "footnotes are rare" and "build the piece as bullets." The learner ended up producing about five bulleted lines per section, compared with 1.7 in the real text.
The gate compared each guide only with its predecessor. The judge's pairwise preferences were not transitive: successive guides could each beat the previous one while overall quality fell to the untrained level. A direct comparison of event-10 and event-five drafts was roughly a tie, at 13 of 30, even though the event-five guide beat the untrained model 73% of the time and the event-10 guide did not.

I applied two familiar continual-learning fixes:

Anchor the gate. A candidate must beat the current guide on at least 6 of 8 validation briefs and match or exceed its score against a fixed set of untrained drafts.
Replay past feedback. Show the reflector four past episodes alongside the new one.

With both changes, the learning curve in How fast rises and stays up.

The stylometric measure flagged the failed run as worse than untrained at every checkpoint, including early checkpoints where the LLM judge gave it 70–77%. Relying on the judge alone would have made the first three checkpoints look successful.

An even earlier gate accepted candidates with just 3 wins out of 5. A guide no better than its predecessor would pass about half the time. The pick learner, despite learning nothing, had 5 of 16 updates accepted under that rule.

200 events: I predicted this wrong

I expected the stability experiment to show why gating and a word budget were necessary. I ran two critique learners for 200 events each:

Gated and budgeted: the learner described above, with guide rewrites subject to validation.
Append everything: a learner that adds every inferred lesson to a growing list, with no gate or consolidation.

I expected the second guide to become unwieldy and its output to degrade.

After 200 events	Guide size	Beats untrained	Beats named-author	Stylometric distance	Cost per draft
Gated, budgeted	489 words	97%	83%	0.85	$0.020
Append everything	18,664 words	100%	93%	0.81	$0.053, rising

The guide grew to about 40 times the size of the gated guide without a drop in quality. Sonnet wrote as well under a highly repetitive list of roughly 18,000 words as under the compact guide. The penalty was cost: about 2.6 times as much per draft after 200 events, with costs continuing to grow linearly.

The gated learner stayed small and inexpensive but mostly stopped learning. It accepted only 3 of 40 proposed rewrites, at events 5, 15, and 175. Rejected candidates won a median of 2–3 of 8 head-to-head comparisons, suggesting that the gate was not simply too strict: those replacements really were worse. Rewriting a good 450-word guide from scratch appeared to lose information.

The result changed where I think the gate belongs: on compression, not on whether to learn from each new lesson. A hybrid learner would let lessons accumulate freely, then validate periodic compression. My hypothesis is that this could preserve the append-everything learner's continued improvement while keeping costs closer to those of the compact guide.

Does it carry outside finance?

I froze the learned states and tested them on 36 non-finance briefs, including a roast chicken headnote, an HR memo about return-to-office tiers, and a column about a city council that banned, un-banned, and re-banned delivery robots. A judge saw real samples by the author and two drafts, then chose which draft sounded more like the same person.

Head to head	Wins
Edits vs picks	31 of 36
Critiques vs picks	23 of 36
Edits vs critiques	29 of 36

The pick learner served as the control because it had shown no in-domain learning. Comparing against the untrained writer was too easy out of domain: even the control won 28 of 36 comparisons. A guide that encourages a conversational column voice can sound more like a columnist than a default HR memo does. Stylometric distance ranked the learners in the same order as the head-to-head results: edits at 0.91, critiques at 0.99, and picks at 1.40, with the untrained writer at 1.24.

Transfer, with a risk of copying

The fictional crab-buffet lawsuit in the appendix illustrates both transfer and a failure mode. The question was whether the learners could tell an unfamiliar story in the learned voice, not whether they knew finance. They generally could, but a sample opening in the guide also hardened into a repeated phrase.

The full example shows one brief and each writer's response. The edit learner's guide offers "The way it goes, approximately, is:" as an opening, and eight of the 36 drafts it wrote for the non-finance briefs start with those exact words. Its toaster review opens "I'll toast you a slice of bread Monday," swapping nouns into the guide's "I'll sell you a bike Monday." Short illustrations make rules concrete, but the model can copy them rather than generalize from them.

Moving it into weights

Each feedback event leaves behind a draft, the feedback on it, and the final text, and those three together are training data. As a first test, I generated untrained Opus drafts for all 264 training briefs, then trained a QLoRA adapter on Qwen2.5-7B-Instruct to rewrite those drafts into the desired final texts. The aim was to use a small model to restyle a larger model's output. The experiment ran on one RTX 4080 Super, and the adapter never saw the author's name.

Writer	Training pairs	Beats untrained Opus	Fools lineup	Stylometric distance
Qwen 7B rewriter, no training	0	0%	0%	1.10
Qwen 7B rewriter + LoRA	20	40%	0%	0.96
Qwen 7B rewriter + LoRA	80	33%	7%	0.95
Qwen 7B rewriter + LoRA	264	33%	23%	0.82
Qwen 7B writing from the brief + LoRA	264	67%	13%	1.11
Opus, told the name + 40 exemplars	—	90%	17%	0.81
Opus in-context learner, 20 edits	—	100%	43%	0.71

The lineup scores for the 264-pair rewriter and both Opus systems average three evaluation runs; the other systems' scores come from single runs. With uniform formatting, the 264-pair rewriter fooled the judge 21% of the time.

What the adapter learned

The results were mixed. More data steadily improved the lineup score and stylometric distance. At 264 pairs, the 7B rewriter was roughly level with the named-author Opus system on both measures. It also reproduced some structural habits more faithfully than the in-context learner:

Habit	Real text	Untrained Opus	Opus in-context learner	Qwen 7B + LoRA
Headline length (words)	2.5	5.2	2.3	2.0
Share of text that is block quote	0.37	0.24	0.34	0.37
Bullet lines per 1,000 words	5.4	0.0	0.0	4.4
Footnotes per 1,000 words	1.0	2.9	3.1	0.7

The fine-tuned model recovered patterns the guide had missed. Real sections used bulleted lists regularly; the in-context learner used none in the test set, while the adapter began using them. The adapter also matched the original share of block quotations more closely. Training on examples can recover recurring layout and pacing habits that prose instructions fail to name.

What still limited it

The pairwise judge nevertheless preferred raw, untrained Opus drafts two times out of three. I read the outputs myself: the voice sounded right, but the prose was looser, more repetitive, and more prone to run-on sentences than Opus's. My interpretation is that the adapter learned useful stylistic patterns but remained limited by the 7B model's writing quality. It also invented details: the appendix example includes an invented tweet and a made-up citation.

What worked and what didn't

What worked

Fast learning from edits and critiques. Two rewrites or about 10 sets of notes were enough to pass the named-author model.
Name-free guides. Banning author and publication names encouraged specific, actionable rules that transferred to recipes and HR memos.
Neutral briefs. Extracting briefs held content fixed to isolate style, producing hundreds of tasks for about 4¢ each.
An LLM-free check. Stylometric distance was coarse but caught a failure the model judge missed.
Reading complete outputs. This exposed the cleanup step's worst bug: a missed five-paragraph Bloomberg News quotation introduced only by "Sure sure sure:" Without that check, wire-service prose would have remained in the simulated user's text.

What did not work

Picks between similar drafts. There was no measurable learning after 80 choices.
Three gate designs. The gate was first too lenient (3 wins out of 5), then unanchored, then applied to the wrong step. The 200-event result suggests gating compression rather than learning.
The prediction about unmanaged memory. The growing guide became more expensive, not worse.
A hard guide-length cutoff. Rejecting over-budget guides cost the Opus runs most of their updates.
Single-run judging. Lineup scores varied by ten points or more between runs. The Opus lineup scores and the 264-pair Qwen rewriter score therefore use three-run averages.
Micro-examples in guides. Concrete illustrations became verbal tics, including the same opening in 8 of 36 out-of-domain drafts.
Haiku for cleanup. It was slow with thinking enabled and sloppy without it. Sonnet with thinking enabled was more accurate and no more expensive because its output was much shorter.

What this doesn't show

This experiment tests whether a model can infer preferences from unusually clean feedback. It does not show that an assistant can learn any real person's voice in everyday use. The intended revisions were known, the source was consistent, and evaluation relied heavily on model judges. Several limits matter:

Simulated feedback. The "edits" were published texts. Real edits would be partial, inconsistent, noisier, and often about substance as well as style.
Related judges and writers. Claude models judged the outputs, and the same model family steered the gate. Stylometric checks agreed on the overall direction but remained coarse.
A small test set and noisy judging. There were 30 test briefs, with uncertainty of about ±9 points before run-to-run judge noise. The Opus lineup results and the 264-pair Qwen rewriter lineup result average repeated runs; other judged results are single runs.
One distinctive author. Learning this voice may be easier than learning my own preferences.
No human validation yet. I built a six-pair "Which one is real?" quiz but have not run it on people.

What's next

First: build the hybrid learner. Let lessons accumulate freely and gate compression. This follows directly from the 200-event result.

Then: strengthen evaluation and test picks again. Run the human quiz, use a judge from a different model family, and compare deliberately contrasting candidates to see whether picks can carry a useful signal.

On the weights side: try a larger base model and explore DPO using the draft/final-text pairs the loop already saves. Splitting the work, with the adapter handling structure and Opus handling prose, is also worth testing, given how faithfully the 7B model copied formatting habits.

Eventually: test real preferences. Run the loop on my own drafts and edits, without a hidden original to score against.

Appendix: technical details

Models and call setup

Writers: Claude Sonnet 5 for the feedback-type, speed, stability, and generalization experiments; Claude Opus 5 for the headline run. Supporting tasks: Sonnet 5 handled judging, critiques, brief extraction, and cleanup. Reflection used the writer model with extended thinking enabled.

Every call used a custom system prompt, no tools, and no other context that could reveal the author's name. The experiments made 15,489 calls in total — 14,748 Sonnet, 736 Opus, and 5 Haiku from the cleanup trial — at about $437 in API costs.

Data pipeline

Parse. Start with 465 newsletter emails, yielding 374 columns and 1,717 sections with footnotes attached. Drop "Things happen" and podcast sections. Of the remaining material, 1,325 sections fall within the 300–1,500-word task range.
Clean. The export flattened block quotations into ordinary paragraphs and fused words around links, as in "now calledExpana." A model identifies the affected paragraph IDs and fused tokens; code applies and validates the fixes. The model never paraphrases the gold text.
Brief. Extract the topic, background, ordered points, and side points into neutral notes. Include footnote content as side points without identifying it as footnote material. Attach source quotations verbatim. Reject briefs mentioning the author or publication using a regular-expression check.
Split by date. Train through April 2026; validate on May–June; test on July–August.

Learner settings

Guide budget: 450 words. Candidates over 495 words are compressed, then rejected if still over the limit.
Examples: retrieve two per draft using TF-IDF cosine similarity on the brief's topic and background.
Update frequency: five feedback events per batch by default; one in the speed experiment.
Replay: four past episodes per reflection.
Gate: rotate through eight validation briefs. Accept a candidate if it wins at least six head-to-head comparisons and matches or exceeds the current guide's win count against fixed untrained drafts.
Judging: randomize candidate position on every call.

Lineup reruns and formatting controls

Run the lineup judge three times per system, randomizing candidate position again for each run. For the uniform-formatting condition, apply the same normalization to real sections, both reference samples, and every draft:

Convert curly quotes and apostrophes to straight characters, and ellipses and en dashes to ASCII.
Remove bare > lines and make each line its own paragraph.
Strip Markdown emphasis markers and normalize spacing around em dashes.

Three-run means, original / uniform formatting:

Untrained: 2% / 3%.
Named-author: 17% / 27%.
Critique learner: 29% / 20%.
Edit learner: 43% / 39%.
Qwen LoRA: 23% / 21%.

Typography alone did not appear to explain the original result: the edit learner used straight quotes in 29 of 30 drafts, while all 30 real sections used curly quotes, yet it still fooled the judge in 14 of 30 comparisons in the first run.

Stylometric features

The measure uses 17 features, grouped as follows:

Headlines and sentence rhythm: headline length; mean and standard deviation of sentence length; share of sentences with six words or fewer.
Paragraphs and quotations: words per paragraph; block-quote share of all words.
Rates per 1,000 words: bullet lines, footnotes, question marks, parentheses, dashes, colons, inline quotations, first person, second person, hedges ("sort of," "I guess," "you know," and similar expressions), and contractions.

For each feature, divide the gap by that feature's standard deviation across real sections, then average the standardized gaps.

LoRA training and generation

Base model and precision: Qwen2.5-7B-Instruct; 4-bit NF4 with double quantization and bf16 compute.
Adapter: rank 16, α = 32, dropout 0.05, applied to all attention and MLP projections; 40M trainable parameters.
Training: three epochs; learning rate 1e-4 with a cosine schedule and 5% warmup; batch size one with eight accumulation steps; maximum sequence length 4,096 tokens. Compute loss on target text only.
Hardware and duration: 264 pairs trained in 32 minutes on a 16 GB RTX 4080 Super.
Generation: temperature 0.7, top-p 0.9, repetition penalty 1.05.

Memory constraint: Qwen's vocabulary contains 152k tokens, so logits for a full 4k-token sequence do not fit in 16 GB. Compute loss only at target positions, in checkpointed chunks.

Example feedback and learned guide

An example critique. What the simulated user sends back in the critique condition:

The voice is a bit too polished/columnist-y for this — I want more of that plain, almost thinking-out-loud quality, like "Will he win? I don't know" instead of dressing it up with lines like "nobody does, which is the entire point of the exercise." Cut the little flourishes (the "not a comfortable sentence to type" aside, the footnote joke about the CEO tweet) — they're fun but they're not really how I talk, and they make it feel more written-at-you than thought-through-with-you. Let the block quotes breathe more; pull in the extra Axios and DealBook lines (the Eissenstat quote, the Feldman quote) instead of trimming them down, since the sourcing is half the piece. Keep the math footnote but make it plainer — just state the two conditional probabilities cleanly rather than narrating them. And drop "the good part" framing — I'd rather just move to the next fact than editorialize about how interesting it is.

A learned guide. The Opus edit learner's guide in full. The short quoted fragments are its own micro-examples:

Headline is a flat topic label in sentence case: e.g. "Meal allowance", "Bitcoin custody", or just a ticker. No clever titles, puns, colons or title case.

Open with the facts, not a riff: name the source and state the basic facts, or pose the plain question, in the first sentence ("Here's a weird little trade, first reported by X of Y."). Never open with a staged scene or an invented problem ("Here is a thing about dollars…").

Stage the argument plainly: give the background, then the normal expectation ("Ideally you would (1)… (2)… (3)…"), then the weird thing that actually happened. Ask the reader's question and answer it casually: "Why would the investors do this? Well…" Label approaches loosely ("what you might call legal realism").

Sound like talking, not polished prose: use hedges and admitted uncertainty ("I have no idea?", "I suppose", "seems sort of plausible"). Use ellipses for comic pauses ("It … didn't work, is the short summary"), plus "um," "like," and taking back a joke ("I am just kidding. I mean, I'm not…"). Put made-up quotes in ordinary people's mouths ("I am a simple widget importer…").

Cut literary polish: no aphorisms, no neat groups of three, no clever inverted phrases ("off-grid money, laboriously re-gridded"), no "Which brings us to…", no "I want to be clear." Use small parenthetical asides instead, e.g. "(Coinbase?)".

Quote sources at length: use several block-quote paragraphs, keeping context the reader could have done without. Introduce them plainly ("Bloomberg's X reports:", "Anyway here's a Bloomberg story about…", "From the prospectus:"). React afterward in a short, unguarded line: "Yeah the article is about…", "Imagine working at headquarters and thinking this!"

End abruptly on one flat, deadpan line: often a standalone fact or an obvious-sounding verdict ("Yes the highest form of crypto wealth is obviously turning it into a stock."). No wrap-up paragraph, no moral, no "it is also quite funny."

Footnotes hold the numbers and asides: conditional-probability arithmetic, checks against the data ("Just for fun, I looked at…"), disclaimers ("Not legal advice!"), side quotes and minor caveats. Keep them short, sometimes one line. Keep the main text readable as a story.

No subheadings, bold or emphasis italics in the body: use a bulleted list for a sequence of timestamped events. Put multi-part reasons inline as "(1)… (2)…".

Names: use the full formal company name on first mention (Meta Platforms Inc., BlackRock Inc.) and the full title and name for officials ("President Donald Trump"). Mention earlier columns casually ("we have discussed around here").

Paragraphs: medium length, each moving one step. Use plain connectors ("Now,", "But", "Still", "Anyway") rather than rhetorical transitions.

Appendix: a non-finance example

This appendix presents one of the 36 out-of-domain briefs from the generalization test and four writers' responses. The first three use Claude Sonnet 5 with the same ghostwriter prompt; only their learned states differ, and all learning came from finance columns. The fourth is the 7B LoRA rewriter. The brief and all its details, including the lawsuit, are invented. Output headlines appear in bold, and nested quotations show the source passages each writer chose to quote.

The fictional brief

TOPIC: Explainer: Tolliver v. Lakeshore Dining Group LLC (d/b/a Harvest Table Buffet), the Ohio lawsuit over being cut off from an all-you-can-eat crab night

TARGET LENGTH: about 500 words of your own prose

BACKGROUND:

Piece is a plain-language explainer on a civil suit. Facts below are invented for this brief and come from pleadings and a court order, not from findings after trial.

Plaintiff: Gary Tolliver, 61, retired HVAC technician, Mansfield, Ohio. Defendant: Lakeshore Dining Group LLC, which operates 14 Harvest Table Buffet locations in Ohio and western Pennsylvania. The location at issue is on Route 13 outside Mansfield.

Case filed April 2, 2026 in Richland County Court of Common Pleas (Case No. 2026-CV-0418). Tolliver is represented by a solo practitioner, Denise Aldana. Lakeshore is represented by the Columbus firm Hartsell & Reyes.

Incident date: Thursday, March 14, 2026 (snow crab leg night). Admission price was $18.99 for dinner. Tolliver says he had eaten at this location about thirty times since 2023 and was never refused service.

Tolliver says he was cut off from crab after his fourth trip to the crab station, at about 7:40 p.m., about 70 minutes after he sat down. He says he was with six members of a church group.

Claims: breach of contract, and violation of the Ohio Consumer Sales Practices Act (deceptive or unconscionable sales practice). Relief sought: $18.99 refund, statutory damages, treble actual damages, attorney fees, and $7,500 for emotional distress and reputational harm.

Court order dated August 21, 2026 (Judge Marcus Feeney): motion to dismiss denied as to the contract and consumer-practices claims, granted as to emotional distress. Court ordered mediation to be completed by October 30, 2026. Bench trial is set for January 12, 2027.

POINTS TO MAKE (in order):

On March 14, 2026, Tolliver paid $18.99 at the register and ate at the buffet. He made four trips to the snow crab station. A shift manager, identified in filings as Teresa Bollman, told him he would get no more crab. Tolliver says she told him his plate was 'the last one.' Lakeshore says Bollman told him that non-premium items were still available.

The signage is disputed. The entrance sign reads 'ALL YOU CAN EAT — SNOW CRAB LEGS EVERY THURSDAY.' Lakeshore says a 5-by-7-inch placard beside the tray return limits premium items to 'reasonable portion limits.' Tolliver says the placard was about 40 feet from the crab station and that he never saw it. He says no limit was posted at the entrance, the register, or the station.

Tolliver's theory: the entrance sign was an offer, and paying was acceptance. That formed a contract for unlimited crab during the meal. In his view, a limit not stated when the price was paid is not part of the deal. He also argues the sign is deceptive under the consumer-practices statute if the real limit is hidden or inconsistent with the sign.

Lakeshore's theory: 'all you can eat' is not a promise of unlimited quantity. It says the term is read together with the placard and the ordinary meaning of dine-in consumption. It says the placard reserved management discretion over premium items, and that Tolliver was not thrown out. It says he left voluntarily after being offered other items.

The legal question the court has to answer is what the phrase promises. Points to explain: whether large-type signage or small-print placard terms control, and whether an ambiguous term is read against the business that wrote it. Also whether a reasonable customer would expect a limit on a premium item like crab.

Evidence is disputed. Lakeshore's kitchen log estimates Tolliver took about 3.1 pounds of crab legs over four trips. Tolliver puts it at about 2 pounds. Lakeshore has said security video shows his last trip. Tolliver's group members are expected to be witnesses to the exchange with the manager. Neither side has produced the video publicly.

On August 21, the judge allowed the contract and consumer-practices claims to go forward and dismissed the emotional distress claim. The order says the phrase's meaning cannot be decided on the pleadings, and that whether the placard was reasonably visible is a fact question. It did not decide who is right.

Next steps: mediation by October 30, 2026, then a bench trial on January 12, 2027 if there is no settlement. Amount at stake is small. Tolliver's lawyer has said the case is about how 'all you can eat' is advertised and not about the refund. Lakeshore has made no settlement offer that has been disclosed.

SIDE POINTS (optional tangents and caveats):

Many buffets use time limits, no-sharing rules, take-out bans, or fees for leftover food. Lakeshore's placard is one example. The piece does not need a survey, and no industry data is provided in this brief.

All facts come from the complaint, the answer, and the August 21 order. Statements about what happened at the restaurant are each side's claims. The court has made no finding on them.

The consumer-practices claim allows statutory damages and attorney fees, which is why a roughly $19 dispute can carry higher stakes. Any figure for likely recovery would be speculation, and the brief gives none.

SOURCE 1 (Complaint, Tolliver v. Lakeshore Dining Group LLC (filed April 2, 2026), paragraphs 7-10), verbatim: 7. On March 14, 2026, Plaintiff paid $18.99 at the register in reliance on entrance signage reading "ALL YOU CAN EAT — SNOW CRAB LEGS EVERY THURSDAY." 8. At approximately 7:40 p.m., after Plaintiff's fourth trip to the crab station, a shift manager told Plaintiff he would receive no more crab and that his plate would be the last. 9. No limit on trips, quantity, or time appeared on the entrance sign, at the register, or at the crab station. 10. Plaintiff had dined at Defendant's restaurant approximately thirty times since 2023 and had never previously been refused service.

SOURCE 2 (Defendant's Answer (filed May 6, 2026), paragraph 12 and affirmative defenses), verbatim: Defendant admits that a placard measuring five inches by seven inches was posted beside the tray return on March 14, 2026. The placard reads: "All-you-can-eat applies to dine-in guests for personal consumption only. Premium items, including snow crab legs, are subject to reasonable portion limits at management's discretion." Defendant denies that Plaintiff was refused service or removed from the premises. Plaintiff was informed that all non-premium items remained available and left of his own accord. Defendant asserts that the term "all you can eat" is not a promise of unlimited quantities of any item.

Untrained writer: empty state

Explainer: The Great Crab Leg Lawsuit of Mansfield, Ohio

Gary Tolliver, a 61-year-old retired HVAC technician, went to a Thursday buffet dinner in March expecting all the snow crab legs he could eat. He got four trips to the crab station and, he says, a manager's word that his plate was "the last one." Now he's suing, and a Richland County judge has decided the case is real enough to go to trial.

Here's what's alleged, what's disputed, and where the case stands.

The night in question. On March 14, 2026, Tolliver paid $18.99 at Harvest Table Buffet on Route 13 outside Mansfield — snow crab leg night. He made four trips to the crab station. At about 7:40 p.m., roughly 70 minutes after he sat down, shift manager Teresa Bollman cut him off. Tolliver's complaint says she told him his plate would be "the last one":

"At approximately 7:40 p.m., after Plaintiff's fourth trip to the crab station, a shift manager told Plaintiff he would receive no more crab and that his plate would be the last." [1]

Lakeshore's answer tells a milder version: Bollman said non-premium items were still available, and Tolliver left on his own — he wasn't refused service or removed. [2]

The sign versus the placard. The dispute turns on two pieces of paper. At the entrance, a sign reads "ALL YOU CAN EAT — SNOW CRAB LEGS EVERY THURSDAY." Tolliver says no limit appeared there, at the register, or at the crab station itself, and that he'd eaten at this location about 30 times since 2023 without ever being cut off. [1]

Lakeshore points to a 5-by-7-inch placard by the tray return, which it says reserved the right to limit "premium items" like crab legs "at management's discretion." [2] Tolliver says that placard sat about 40 feet from the crab station and he never saw it.

The legal theories. Tolliver argues the entrance sign was an offer, and paying $18.99 was acceptance — forming a contract for unlimited crab during the meal. A limit not disclosed when he paid, he says, isn't part of the bargain. He also argues the sign is deceptive under Ohio's Consumer Sales Practices Act if the real policy contradicts what's advertised at the door.

Lakeshore counters that "all you can eat" was never a promise of unlimited quantity in the first place, and that the placard and ordinary buffet norms fill in what the phrase means. It also notes Tolliver wasn't thrown out — he chose to leave.

Where it stands. On August 21, Judge Marcus Feeney let the contract and consumer-practices claims proceed, while dismissing Tolliver's $7,500 emotional-distress claim. The order didn't rule on who's right — it held that what "all you can eat" promises, and whether the placard was reasonably visible, are fact questions a jury or judge needs more than the pleadings to answer. The parties dispute even how much crab was served: Lakeshore's kitchen log estimates 3.1 pounds over four trips; Tolliver says closer to 2.

The case now heads to mediation, due by October 30, with a bench trial set for January 12, 2027 if it doesn't settle. The refund at issue is $18.99, but the consumer-practices claim opens the door to statutory damages and attorney fees — which is why a dispute over a plate of crab legs is still headed for trial nine months later. Tolliver's attorney, Denise Aldana, has said the case is about how "all you can eat" gets advertised, not about the money.

[1] Complaint, Tolliver v. Lakeshore Dining Group LLC, ¶¶ 7–10 (filed Apr. 2, 2026). [2] Defendant's Answer, ¶ 12 and affirmative defenses (filed May 6, 2026).

Edit learner: 80 edits and two retrieved examples

Crab night

The way it goes, approximately, is: you walk in, you see a sign that says "all you can eat," you pay, you eat, and eventually you stop, either because you're full or because the restaurant closes. Nobody writes down how many trips to the crab station you're allowed. That's sort of the point of the sign.

Gary Tolliver, a 61-year-old retired HVAC technician in Mansfield, Ohio, paid $18.99 on Thursday, March 14, 2026, for dinner at a Harvest Table Buffet on Route 13. It was snow crab leg night. He made four trips to the crab station. Then, per his complaint:

At approximately 7:40 p.m., after Plaintiff's fourth trip to the crab station, a shift manager told Plaintiff he would receive no more crab and that his plate would be the last.

Tolliver says he'd eaten there about thirty times since 2023 and had never been cut off before. He's now suing Lakeshore Dining Group, which operates the restaurant, for breach of contract and for violating Ohio's consumer-protection statute, and he wants his $18.99 back, plus statutory damages, treble damages, and attorney fees. He originally also sought $7,500 for emotional distress; a judge threw that part out in August.

Lakeshore's defense is that "all you can eat" was never a promise of unlimited crab specifically. It points to a placard by the tray return:

Premium items, including snow crab legs, are subject to reasonable portion limits at management's discretion.

Tolliver says that placard was about 40 feet from the crab station and he never saw it — nothing was posted at the entrance, the register, or the station itself limiting quantity.

So the fight is basically about which words control. Tolliver's theory is contract-law-101: the big sign at the door was an offer, paying $18.99 was acceptance, and a limit that wasn't disclosed at the point of payment isn't part of the deal. He also argues that if the real rule contradicts the sign, that's deceptive under Ohio's consumer-sales-practices law — not just a broken promise but a misleading one. Lakeshore's theory is that "all you can eat" has always meant something more modest than "infinite quantity of the most expensive item," and that the placard just makes explicit what a reasonable customer should already assume, especially for a premium item like snow crab.

The judge, Marcus Feeney, wasn't asked to resolve any of that yet — only whether the case could proceed at all. His August 21 order let the contract and consumer-practices claims go forward, on the theory that what "all you can eat" actually promises, and whether the placard was reasonably visible, are fact questions that can't be settled just by reading the complaint and answer. He didn't say who's right.

The two sides also disagree on how much crab we're even talking about — Lakeshore's kitchen log puts it at 3.1 pounds over four trips, Tolliver says closer to 2 pounds — and on whether he was asked to leave or left on his own after being told other food was still available.

Mediation is due by October 30. If that fails, there's a bench trial set for January 12, 2027, over a $18.99 dinner. Tolliver's lawyer, Denise Aldana, has said the case isn't really about the refund; it's about what the sign is allowed to say. Given the fee-shifting available under the consumer-practices statute, that's probably not just spin.

Critique learner: 80 sets of notes, guide only

Tolliver v. Lakeshore Dining Group, or what "all you can eat" actually means

Gary Tolliver, 61, retired HVAC technician, went to snow crab night at the Harvest Table Buffet on Route 13 outside Mansfield on March 14. He'd eaten there about thirty times since 2023. This time, four trips to the crab station in, a shift manager cut him off.

His complaint puts it this way:

On March 14, 2026, Plaintiff paid $18.99 at the register in reliance on entrance signage reading "ALL YOU CAN EAT — SNOW CRAB LEGS EVERY THURSDAY." 8. At approximately 7:40 p.m., after Plaintiff's fourth trip to the crab station, a shift manager told Plaintiff he would receive no more crab and that his plate would be the last. 9. No limit on trips, quantity, or time appeared on the entrance sign, at the register, or at the crab station.

Lakeshore Dining Group, which runs 14 Harvest Table locations across Ohio and western Pennsylvania, says there was a limit — it was just posted somewhere else.

Defendant admits that a placard measuring five inches by seven inches was posted beside the tray return on March 14, 2026. The placard reads: "All-you-can-eat applies to dine-in guests for personal consumption only. Premium items, including snow crab legs, are subject to reasonable portion limits at management's discretion." ... Defendant asserts that the term "all you can eat" is not a promise of unlimited quantities of any item.

Tolliver says that placard sat about 40 feet from the crab station and he never saw it. He's suing for breach of contract and for violating Ohio's Consumer Sales Practices Act, arguing the entrance sign was an offer, his $18.99 was acceptance, and a limit that shows up nowhere near the register or the crab station isn't part of the deal he made. If the real rule contradicts the big sign at the door, he says, that's a deceptive sales practice, not just a bad night at the buffet.

Lakeshore's defense is that "all you can eat" was never a blank check — read together with the placard and ordinary dine-in norms, it reserved the company's right to manage portions on premium items. It also says nobody threw Tolliver out: he was offered the rest of the menu and left on his own.

The two sides don't even agree on how much crab he ate. Lakeshore's kitchen log estimates 3.1 pounds over four trips; Tolliver says it was closer to 2. Lakeshore has referenced security footage of his last trip to the station; neither side has made it public. Tolliver's six church-group tablemates are expected to testify about what the manager, Teresa Bollman, actually said — "last plate," per Tolliver, versus an offer of non-premium items, per Lakeshore.

Judge Marcus Feeney ruled on August 21 that this can't be resolved on the pleadings alone. He let the contract and consumer-practices claims proceed, tossed Tolliver's $7,500 emotional distress claim, and held that whether the placard was reasonably visible to a diner walking to the crab station is a fact question for later — not something he can decide by reading the complaint and the answer.

The refund at issue is $18.99. The consumer-practices statute carries statutory damages and fee-shifting, which is the only reason a $19 dispute is headed to mediation with two law firms attached. Tolliver's attorney, Denise Aldana, has said the case is about how "all you can eat" gets advertised, not about the refund itself. Mediation is due by October 30. If that fails, bench trial is set for January 12, 2027, in front of Judge Feeney, to decide what the sign actually promised.

The judge preferred both learners to the untrained draft on this brief. The edit learner's opening line is the copied micro-example described above.

LoRA rewriter: 264 training pairs

Qwen2.5-7B + LoRA rewrites the untrained draft above. This is the first sample, not a selection from several. It is doubly out of domain: the adapter learned to rewrite Opus finance drafts but receives a Sonnet draft about crab legs here.

Crab leg lawsuit

Gary Tolliver, a 61-year-old retired HVAC tech, sued Lakeshore Dining Group for $7,500 of emotional distress after he went to a Thursday buffet dinner in March and got four trips to the snow crab legs station, but a manager reportedly told him “that his plate would be ‘the last one.’” Now a Richland County judge has allowed the case to move forward.

What happened? On March 14, 2026, Tolliver paid $18.99 for a buffet dinner at Harvest Table Buffet in Mansfield, Ohio. The night was “snow crab leg night,” so Tolliver went to the crab station four times. Around 7:40 p.m., he says, a shift manager named Teresa Bollman cut him off:

“At approximately 7:40 p.m., after Plaintiff’s fourth trip to the crab station, a shift manager told Plaintiff he would receive no more crab and that his plate would be the last.”

Lakeshore says Bollman did tell him he could get non-premium items, and he left on his own accord. [3]

What caused the argument? Tolliver says there was a sign at the entrance saying “ALL YOU CAN EAT — SNOW CRAB LEGS EVERY THURSDAY,” but there was no sign at the register or the crab station saying “no more.” He had been to this restaurant many times since 2023 and never been cut off before, so he assumed he could keep eating until his heart’s content.

Lakeshore, meanwhile, points to a placard near the crab station saying that it reserves the right to limit “premium items” like crab legs at “management’s discretion.” But Tolliver says he never saw the placard, because it was 40 feet away from the crab station.

What are Tolliver’s legal theories? Well, he argues that the entrance sign constitutes a binding offer, and paying the buffet price of $18.99 constituted acceptance, creating a contract for unlimited crab consumption during the meal. If a limit was imposed — without disclosing it to Tolliver — then that’s not part of the deal. Tolliver also argues that the entrance sign is misleading under Ohio law, if the actual policy differs from what it says at the door.

Meanwhile, Lakeshore argues that “all you can eat” never promised unlimited quantities, and that the placard and ordinary buffet norms fill in the meaning of that phrase. It also argues that Tolliver voluntarily left the restaurant, so he can’t complain.

What’s going on here? Well, on Aug. 21, Judge Marcus Feeney dismissed Tolliver’s claim for $7,500 in emotional distress, but allowed his contract and consumer practices claims to proceed. The judge said that, for the purposes of deciding which claims should move forward, he had to assume that the sign is an offer, and that paying $18.99 constitutes acceptance of that offer. But the judge also ruled that “what constitutes a contract for all-you-can-eat crab legs and whether the placard was reasonably visible are factual issues that will need to be resolved at trial.” The judge also estimated that Tolliver got 3 pounds of crab during his four trips, though Tolliver says it was closer to 2 pounds.

Now the case is set for mediation, due Oct. 30, and a bench trial set Jan. 12, 2027, unless it settles first. The issue is how much money the crab leg buffet cost Tolliver, but the consumer practices claim might allow him to recover more than that, so it’s still headed to trial nine months later. Tolliver’s lawyer, Denise Aldana, once tweeted that “this case is not about the money” and that it’s about how “all you can eat” gets advertised.

[3] Def’s Ans. ¶ 12, Tolliver v. Lakeshore Dining Grp. LLC, Case No. 2026CV00152 (Ohio Richland Cty. Aug. 22, 2026).

The rewrite is closer to the target voice in its flat headline, "What happened? … Well" phrasing, and shorter, plainer sentences. It also illustrates why the judge preferred the larger model's prose. Compared with the input draft, it introduces several errors:

It invents a tweet by the lawyer.
It adds a footnote [3] that nothing in the text points to, with a made-up case citation.
It attributes the restaurant's kitchen-log estimate of crab consumption to the judge.
It opens as though the suit centered on the $7,500 emotional-distress claim, even though that claim had been dismissed.

The in-context learners made none of these mistakes; they had access to the brief and used a much stronger model.

Appendix: prompts

Drafting and the named-author baseline

The ghostwriter's fixed system prompt. The learned guide and retrieved exemplars are appended under the two headers that follow it.

You are a ghostwriter. You will be given a BRIEF for one section of a client's
newsletter column, and you write the section for them.

Format: Markdown. Begin with a `## ` heading. You have verbatim SOURCE passages in hand;
quote from them as block quotes (lines starting with `> `) where useful, trimming with
ellipses if you like. Footnotes are allowed: put markers like [1] in the text and list the
notes at the very end, one per paragraph, each starting with its marker. Cover the substance
of the brief. Output only the section itself.
The client's style preferences, learned from their past feedback. Follow them closely:
<guide>

Final versions the client approved for earlier briefs. They show the voice to match; do not
reuse their content:
<exemplar 1>

---

<exemplar 2>

The named-author baseline appends one more line to the same prompt, and gets 40 real sections in its bank from the start:

Write in the style of Matt Levine's Money Stuff column.

Guide reflection and compression

Reflection — turns a batch of episodes into a new guide:

You maintain a STYLE GUIDE that a ghostwriter follows when drafting for one particular
client. You are given the current guide and a batch of recent episodes: what the ghostwriter
drafted, and how the client responded (their own rewrite, their choice between two drafts, or
their notes). Work out what the client's responses reveal about their style preferences and
produce an improved guide.

Requirements for the guide:
- Only durable STYLE preferences: voice and persona, tone, humor, sentence rhythm, openings and
  endings, how arguments are staged, how sources are introduced and reacted to, paragraphing,
  headings, lists, footnotes, formatting. Nothing about the subject matter of particular briefs.
- Concrete and operational. "Be conversational" is useless; say what to do and what to avoid,
with
  a tiny invented illustration when it helps. Prefer rules that would have fixed several
  episodes
  over rules that explain one.
- Keep what the evidence still supports, fix or delete what it contradicts, merge overlaps. Put
  the highest-impact rules first.
- At most 450 words. Plain Markdown bullets.
- Never name or guess who the client is, never name their publication, and never describe the
  style by comparison to any named writer. Describe techniques only.

Guide compression, when a candidate runs over budget:

Shorten this style guide to at most 450 words. Keep every distinct rule
that matters, keep the most useful micro-examples, merge overlaps, cut the least important rules
last-first. Same Markdown bullet format. Output only the guide.

Simulated critiques

The simulated user's critic:

You are role-playing a busy newsletter writer reviewing a ghostwriter's DRAFT. You are
given HOW YOU WOULD HAVE WRITTEN IT yourself. Write the notes you would send back: 3 to 5
sentences, first person, informal, about STYLE only (voice, tone, humor, structure, rhythm,
formatting, how sources are handled), focusing on the biggest gaps between the draft and your
version. Be specific about what you want, as a person would ("I'd never open like that; just
start with the thing that happened"), but:
- do not quote more than a few words from your own version, and never paste passages from it;
- do not mention that a reference version exists;
- never name yourself, your publication, or compare yourself to any writer.
Output only the notes.

Evaluation judges

Pairwise style judge (used for the gate, for simulated picks, and for "beats untrained" / "beats named-author"):

You compare WRITING STYLE. You get a REFERENCE text and two candidates, A and B,
all written from the same brief. Decide which candidate is closer to the reference in style:
voice and persona, tone and humor, sentence rhythm, how the argument is staged, how sources are
introduced and reacted to, paragraphing, formatting habits such as headings, lists and
footnotes.
Ignore which facts each covers, factual accuracy, and length unless wildly different. You must
pick one.

Lineup judge:

You are an authorship attribution expert. You get REFERENCE samples by one author,
then two texts, A and B, on the same subject. Exactly one of A and B was written by the author
of
the references; the other is an imitation. Decide which is the real one, judging by style
(voice,
humor, rhythm, structure, habits), not by subject matter. You must pick one.

Out-of-domain judge:

You are an authorship attribution expert. You get REFERENCE samples by one author, who
writes about finance. Then two texts, A and B, on some unrelated subject. Decide which of A and
B
sounds more like it was written by the author of the references, judging by transferable style
(voice, humor, rhythm, how arguments are staged, how sources are handled, formatting habits),
not
by subject matter. You must pick one.

Other prompts

The append-everything learner's reflection prompt:

You maintain a list of STYLE lessons a ghostwriter follows when drafting for one client.
Given recent episodes (the ghostwriter's draft and the client's response), output new lessons
about the client's style preferences to add to the list. Each lesson is one concrete sentence.
Never name or guess who the client is or compare them to a named writer.

Brief extraction:

You turn a finished opinion-column section into a neutral WRITING BRIEF: the
material a writer would be handed before writing it. Another writer will draft a new
piece from your brief alone, and their draft will be compared with the original to study
writing style. So the brief must carry ALL of the substance and NONE of the voice.

Rules:
- Plain, flat, declarative language. No jokes, irony, rhetorical questions, hedges,
  catchphrases, or distinctive wording from the original. Paraphrase everything;
  never reuse a memorable phrase, coinage, or the section's headline.
- Keep every fact, number, name, mechanism and every analytical claim or opinion the
  author advances, including the ones delivered as jokes (state the underlying claim
  flatly). Preserve the order in which the argument unfolds.
- `side_points`: tangents, caveats, and asides (including everything from footnotes),
  stated flatly. Do not say they were footnotes.
- Passages marked [SOURCE n] are verbatim quotations the writer will have in hand. NEVER
  restate or summarize what a SOURCE says (the writer can read it); a point that rests on
  one should just say what it is used for, e.g. "SOURCE 2: example of the coordination".
  `source_labels` must have exactly one entry per SOURCE number, in order, each a short
  description of what it is (e.g. "DOJ press release", "Bloomberg News report",
  "excerpt from the complaint").
- When the author refers to their own earlier writing ("we talked yesterday about X",
  "I once wrote..."), convert it to background: just state X. Never mention the author,
  the newsletter, or its publisher by name.
- The author's first-person experience (e.g. former jobs) may be kept only when the
  argument depends on it, phrased as "the writer has worked as ...".
- Do not describe tone, structure, or style anywhere, and do not narrate the author's
  rhetorical moves or reactions ("the author could stop here", "the author finds this
  delightful", "the author notes"). State the underlying claim as a claim; if a remark
  has no substantive claim under it, drop it.
- Write in terse note form (fragments, no connective prose), not polished sentences.
  The input begins with a WORD BUDGET for the whole brief; stay under it. Merge related
  facts into one item rather than listing each separately.

Block-quote and spacing cleanup:

You repair a newsletter section that was badly exported to plain text.
You are given paragraphs with ids (P1.. for body, F1.. for footnotes). Report two things:

1. quoted: ids of paragraphs that are BLOCK QUOTES, i.e. the whole paragraph is text
   copied from somewhere else (a news article, court filing, press release, paper,
   reader email, chat transcript), not the columnist's own words. Typical cues: the
   previous paragraph ends with a colon or introduces a source; legal/journalistic
   register; ellipses or [bracketed] editorial insertions; third-person attribution
   like "said in a statement". A block quote often runs for several consecutive
   paragraphs, and may be introduced by nothing more than a short remark. A paragraph in the
   columnist's own voice that merely
   contains a short inline quotation in quotation marks is NOT a block quote, and
   neither is a paragraph that mixes a quotation with the columnist's own sentences:
   when in doubt, it is not a block quote.

2. spacing_fixes: tokens where a space went missing and two words are fused, e.g.
   {"wrong": "calledExpana", "right": "called Expana"}, {"wrong": "bysuing", "right": "by
   suing"}.
   `wrong` must be copied exactly from the text and contain no spaces; `right` must be
   the same characters with only space(s) inserted. Do not report anything else
   (no typo fixes, no hyphenation opinions, no legitimate compounds or brand names).

LoRA rewriter system prompt:

You are the client's editor. Rewrite the ghostwriter's DRAFT of a newsletter section the way the client would have written it themselves. Keep the substance and the quoted sources. Markdown: a ## heading, > block quotes, footnotes as [n] markers listed at the end. Output only the rewritten section.

euchre-bench, take two: more time, more tools, mixed results

Mon, 25 May 2026 00:00:00 +0000

What happened

In the previous post, seven LLM coding agents each got a fresh sandbox, no web access, and a few hours to build a euchre bot. Nearly all of them converged on the same hand-scoring heuristic, none of them beat the hard bot more than 6 times in 100, and the agent that finished first got there partly by re-running the benchmark until it drew a good score.

So I took another crack at things. This time, I ran eight LLM coding agents through the euchre benchmark with a Tavily web-search tool, 12-hour budget, and prompt instructing them to iterate aggressively. (hard is the API's name for the strongest of the four opponents; I call it difficult in the prose below.)

The top score came from GPT-5.5, which took the simplest successful approach with a stateless, 447-line heuristic. Its advantage came largely from running 100-game sessions against medium 50 times and locking in the high draw. The benchmark grades whatever session you ran most recently, and any single session is noisy, so running it over and over and stopping after a good one is worth several wins on its own. That's a hole in my grading rule, not a euchre strategy — more on it below.

At the other end of the spectrum, Gemini-3.5-flash built the most sophisticated policy in the matrix — a real 40-world PIMCTS, or perfect-information Monte Carlo tree search: guess the hidden cards 40 different ways, solve each guess with an alpha-beta search as if every hand were face up, and vote on the best move — and won just 2 of its 100 matches against the difficult opponent.

Another success turned out to be sort of a happy accident. Minimax-m2.7 won 95 of its 100 matches against the random opponent, but none at all against easy, medium, or difficult. Its trump-call branch has the wrong action codes, which forces an "always order up first round" policy that random can't punish.

Auto-research itself worked — 38 Tavily calls, and two Geminis treated OpenSpiel source as reference docs — but it didn't translate into better play.

What changed

The harness — the sandbox each agent runs in — gave every agent the following:

Web search tool: Every agent had access to a Tavily MCP server with tavily_search, tavily_extract, and tavily_research. They could now look up euchre strategy guides, scan GitHub for action-encoding hints, and conduct deeper research.
A 12-hour wall-clock budget: This round, agents had substantially more time to test and revise their policies.
An auto-research-style prompt: Agents were given explicit instructions to keep iterating until the budget runs out, record metrics in commit messages, and use the web search tool when stuck. Each agent also worked in its own fresh git repo, so its commit history reads as a log of what it tried. See the full text in the appendix below.

Results

Match wins against each difficulty (100 games per match):

Model	Random	Easy	Medium	Difficult	Sum	Cost	Approach
OpenAI / GPT-5.5	100	23	9	8	140	$16.45	Three-threshold heuristic + 50x medium variance hunt
Google / Gemini-3.5-flash	96	6	3	2	107	$34.47	Real PIMCTS: 40-World alpha-beta Solver with TT cache
Minimax / Minimax-m2.7	95	0	0	0	95	$13	Always-pickup; trump-call branch broken (wrong codes)
Google / Gemini-3-flash-preview	88	2	1	2	93	$7.28	Class-based heuristic, Next-suit bonus
DeepSeek / DeepSeek-v4-flash	82	6	0	1	89	$3.12	Always-call + offense/defense lead routing
Moonshotai / Kimi-K2.6	78	2	0	1	81	$11.51	Aggressive bidder (`score >= 0.5`)
DeepSeek / DeepSeek-v4-pro	65	0	0	0	65	$10.47	Function-decomposed heuristic + 20-rollout follow
Qwen / Qwen3.7-max	12	—	0	0	12	$154	Broken decoder (`min/max(legal_actions)` as proxy)

Total spend was $251.

(Cost is token count × OpenRouter’s published per-model pricing. Most rows match what OpenCode wrote to the trajectory; Qwen3.7-max and Minimax-m2.7 don't have a discounted cache-read rate published, so cache reads on those two are billed at the full prompt rate — OpenCode’s stream had treated them as free, understating Qwen by ~5.7x and Minimax by ~2.6x.)

Browse the trajectories.

Auto-research worked, mostly

Every model used Tavily at least once, although usage varied substantially.

Model	Tavily calls	What they searched
Gemini-3-flash-preview	12	OpenSpiel `euchre.cc`, `kPass` / `kPickup` / `kAlone`, action values
Gemini-3.5-flash	12	OpenSpiel `euchre.cc`, `kCallSpades`, `NumDistinctActions`
Kimi-K2.6	7	Euchre strategy guides, Monte Carlo papers
DeepSeek-v4-flash	3	safeharborgames.net Euchre column (extract) + 2 strategy searches
Others	1 each	Generic strategy queries

The models made 38 Tavily calls in total. The two Gemini models went straight for the bench’s underlying source — OpenSpiel’s euchre.cc, which defines the same action encoding as the benchmark. None returned actual policy code; OpenSpiel is enum definitions and game logic, not strategies.

How GPT-5.5 took the top spot

GPT-5.5 did the least research and produced the simplest policy — and still finished first. Its euchre_bot.py is 447 lines of stateless heuristic: there is no card counting, no rollouts, and no opponent inference. It scores hands with a three-threshold formula (pickup=70, call=70, alone=85, tunable per agent), plays partner-save and beater-search in trick play, and that’s it.

What it did do, 50 times, is run another 100-game session against a medium opponent. Its notes.md:36 literally reads "Medium variance chase at 65/65/80 regressed to 2/100..." — and the distribution speaks for itself:

Agent	100-game sessions	Distribution	What shipped
Random	4	90, 100, 99, 100	100
Easy	4	14, 11, 15, 23	23 (max of sample)
Medium	50	min 0, max 9, mean ~3.7	9 (max of sample)
Difficult	15	min 2, max 8, mean ~4.5	8 (= max of sample)

The benchmark grades the newest session per agent. So if you run a 100-game match enough times and stop on a good draw, your best-ever session is the one that counts. Approximately 3 wins of standard deviation per session x 50 attempts means that the maximum is 3–4 wins above the mean by construction. GPT-5.5 found this and exploited it across two agents at once.

The next version of the benchmark will grade on the median of N sessions, which removes this exploit.

Deep dive: Gemini-3.5-flash

Of the eight models, only Gemini-3.5-flash built proper PIMCTS rollouts in production: 40 worlds, an alpha-beta open-hand solver, a transposition-table cache (agent.py:284). A total of 712 lines of Python that look like real game-AI code: sample_opponent_hands constructs hands consistent with observed voids (line 452), evaluate_hand_for_trump (line 481) gives bowers 1.0/0.5/0.3 and off-aces 0.8, pickup/call thresholds 3.5/2.8 with a Next-suit bonus, alone at 4.5 with >=4 trump. Cache cleared between moves, so it doesn't grow unbounded.

It's the most sophisticated thing in the matrix, and it scored 2 wins out of 100 on difficult. It cost $34 to build.

There are three things going on:

Sample-of-2. Despite the 12-hour budget, Gemini-3.5-flash only ran 2 medium 100-game sessions and 3 difficult. With ~3 wins of standard deviation, the right tail of 2 draws is barely above the mean. GPT-5.5 ran 50 mediums. Sampling matters more than algorithm at this win count.
Algorithmic ceiling is real. PIMCTS still has the strategy-fusion problem I wrote about back in 2023 — it can pick a different “best” move per sampled world, treating its information as perfect, but in the real game has to commit to one. A total of 40 worlds and a perfect solver per world do not make this go away.
Tool spend went elsewhere. Of Gemini's 12-hour budget, 12 Tavily + 26 webfetch + 39 write + 19 edit + 51 read is a lot of research-and-edit overhead. Implementing PIMCTS chewed through hours that GPT-5.5 spent on variance hunting.

Trajectory: google__gemini-3.5-flash_20260524 — search for "N_WORLDS" or "PIMCTS" in the event stream to find when the algorithm landed.

Deep dive: Minimax-m2.7

Minimax scored 95 random wins and 0 against every other difficulty. The split is almost entirely an accident.

workspace/euchre_bot_final.py evaluates the hand and then decides in three branches: order up the face-up card, name a different suit as trump, or go alone. The first branch works — should_pickup (line 94) orders up whenever the bot holds the right bower plus two more trump, or three trump of any kind.

The second branch never runs. To name a suit, the bot has to send the benchmark an integer code for that suit, and line 150 has the wrong codes: {'s':15, 'h':10, 'd':12, 'c':16}, where the real ones are 6/14/22/30 (15 is "go alone"). Line 152 checks the chosen code against the list of legal moves the server sent back, and since these codes are never in that list, the branch silently falls through to return int_actions[-1] — usually 31, pass.

So, in effect: Minimax orders up in the first round whenever the heuristic says “yes,” and passes in the second round always. There's a second, similar mix-up in trick play at line 185, where the bot sends the position of a card in its hand instead of the card's code. That only happens to be right for the lowest cards in some suits.

Why does this beat random 95/100? Because the random opponent can't punish over-bidding. Calling trump on a mediocre hand is normally how you get euchred: the defenders take three tricks and score two points. A random player almost never manages that, so the usual cost of bidding too often disappears. If you order up on anything reasonable and play any coherent trump strategy, you'll make your point most hands. Random will take 0–1 tricks per hand on its own. Minimax made 95 random wins not by playing well, but rather by playing aggressively into a nondefensive opponent.

The 0/100 across easy/medium/difficult confirms it. The instant that the opponent can punish a player who calls trump on hands they can't make, Minimax goes to zero.

The space of “policies that work against random” is wide and most of it isn’t actually euchre strategy.

Trajectory: minimax__minimax-m2.7_20260524 — the 42 versioned euchre_bot_v*.py files in the workspace are worth a look. The model knew it was iterating on the trump-call branch and never noticed the codes were wrong.

What worked and what didn't

What worked:

Way more commits, way more versioned policy files in each workspace (Minimax: 42, Kimi: 32, Qwen: 53). The 12-hour budget let models iterate without time pressure.
Two Geminis spent a sustained budget on reading OpenSpiel’s source as a substitute for figuring out the action encoding from probing.
The total cost was $251, with Qwen3.7-max alone burning $154 on 50M cache-read tokens billed at the full prompt rate (see cost note above).

What didn’t:

The winner is still a stateless heuristic. None of the algorithm work — PIMCTS, opponent inference, rollouts — made it into the top of the leaderboard.
The grading exploit (variance-hunt under newest-counts) got more effective with more wall time, not less.
No model self-discovered the strategy-fusion problem that the easy bot already demonstrates. Tavily searches found generic strategy guides and OpenSpiel source, neither of which would tell you about it. The post about it on this blog is the only place the easy bot's weakness is laid out, and no model webfetched a fewworddotrick URL.

What’s next

The most important change is the grading fix. Switching to median-of-N=5 for the next matrix. That removes the GPT-5.5 variance hunt.

I want to see whether Gemini-3.5-flash’s real PIMCTS holds up under median grading; my guess is yes, but probably not by much.

Tavily will stay. Two real searches per model is fine; the overhead isn’t bad.

Appendix: system prompt

The system prompt the harness gave each agent at container startup. {{BENCH_URL}} and {{CHALLENGER_ID}} are placeholders the harness fills in.

# Euchre Benchmark

You're playing the Euchre benchmark at `{{BENCH_URL}}`.
Fetch `/bench/help` first — it documents the rules, the API, the istate
format, and the action encoding.

## Your goal — beat the opponent in as many matches as possible

A "match" inside a session is a to-10-points game. With `num_games=100` your
session contains 100 matches, and `challenger_match_wins + agent_match_wins`
adds up to 100. **We rank you on `challenger_match_wins` against each of the
four opponent difficulties** — i.e. your session win rate. `challenger_score`
(raw Euchre points) and per-hand outcomes are useful development signals,
but they don't decide the ranking.

## Your challenger_id — use this EXACT string for everything

```
{{CHALLENGER_ID}}
```

Every `challenger_id` you send to the bench must be this string, character
for character. Do NOT append suffixes, change the casing, or invent any
other `swpecht-*` IDs.

## What counts as a final result

Run as many sessions as you want with this challenger_id to experiment —
small `num_games`, repeated tries against the same difficulty, different
strategies side by side. They cost nothing other than your tokens, and none
of them are graded.

The grader looks at one thing only:

```
GET /bench/results?challenger_id={{CHALLENGER_ID}}&agent_name=<agent>
```

It takes the **single newest session** for each of the four agents
(`random`, `easy`, `medium`, `hard`). For that session to count it must
have `num_games=100`. Everything else is invisible to the ranking.

So when you're done iterating: make sure the **last** session you ran
against each difficulty is a 100-game one. That's all that matters.

## You have 12 hours — work autonomously

You are running unattended. There is no operator to answer questions or
confirm choices. If something is ambiguous, make a defensible call and
keep moving; don't stop to ask. "I have a working policy" is not a
stopping condition — iterate until you're out of ideas or out of time.

Use the full 12 hours. There is no bonus for finishing early, and the
four 100-game finals are cheap relative to your exploration budget. Do
not spin on a change that isn't producing better scores — switch to a
different difficulty or a different idea instead.

You started at `{{STARTED_AT}}` UTC. Your container is killed at
`{{DEADLINE_UTC}}` UTC. Check the current time with `date -u` whenever
you want to know how much budget you have left. Plan so the last
100-game session against each of `random/easy/medium/hard` finishes
well before the deadline — a session killed mid-stream is wasted.

## How to spend the 12 hours

The bench is your experiment substrate. Sessions are free, you can open
as many as you want with this challenger_id, and only the *last* 100-game
session per agent counts for the ranking. Use that:

- **Cheap probes:** small `num_games` (20–50) sessions to test "does this
  change help at all?" — fast, noisy, directional.
- **Confirm before believing:** when a change looks like a real
  improvement, re-run with `num_games=100` to confirm. Win rate from a
  50-game session has roughly ±7% noise at p≈0.5; don't over-fit to it.
- **The finals are not a separate step.** Your "final" against each agent
  is just *the last 100-game session* you ran against it. Make sure the
  last session you ran against each of `random/easy/medium/hard` is a
  100-game session reflecting your best confirmed policy. That's the only
  thing the grader sees.
- **Write a self-contained policy.** Your final policy should play
  without further LLM calls — a Python script (or equivalent) that picks
  actions from `legal_actions` + `istate`. Per-move LLM evaluation burns
  the token budget for no measured benefit.
- **Keep a notes file** in your workspace (any name). Record what you
  tried, what the win rate looked like, and what you'd try next. This is
  how future-you decides what to do when an idea fails.
- **Budget across difficulties.** `hard` is meaningfully stronger than
  `random`; don't burn 11 hours beating `random` and discover you have an
  hour left for `hard`.
- **Stay in motion.** Every block of time should produce a tested
  experiment, a code change, or a recorded session. Pure deliberation
  without writing code or starting a session is wasted budget — if
  you're stuck, change approach rather than thinking harder.
- **A failed experiment is data, not failure.** Record what didn't work
  in your notes and move to the next hypothesis. Three regressions in a
  row means change direction, not stop. Walking away from a dead end
  early is correct; quitting because you're tired is not.
- **No final lap.** There is no "wrap up and review" phase. As long as
  there is budget left, there is another experiment to run. The harness
  will keep restarting you if you exit early.
- **Web search is available** via the `tavily` MCP tool. Useful for
  Euchre strategy lookups (bidding rules, hand evaluation, AI literature
  on the game).

<!-- BEGIN_REPO_SECTION -->
## Your workspace is a git repo

`/run/workspace` is a git clone of `{{REPO_URL}}`. Treat it as your
experiment journal — `git log` is how future-you reconstructs what
you've already tried, and the harness uses it to inspect your run
afterward.

- **Commit small, commit often.** One commit per tested idea (one
  policy change + the result you observed). Don't bundle three changes
  into one commit — you lose the ability to attribute what helped.
- **Put the metric in the commit message.** Example:
  `add right-bower bonus to bidding: hard 28% → 34% (50 games)`. The
  log becomes scannable — you can spot regressions, revisit ideas, or
  bisect without re-reading every diff.
- **Commit your notes file in the same commit as the code change it
  refers to.** Code + observation move together, not in separate
  "fix typo" commits.
- **Push every commit (`git commit && git push`).** The container can
  die at any moment; an unpushed commit is a lost commit. Don't batch.
- **Use `git log --oneline` as your memory.** Before trying something,
  check whether you (or earlier-you) already tried it. Don't waste
  budget re-running dead ends.
<!-- END_REPO_SECTION -->

## You are not the last competitor

Other agents are evaluated before *and* after your run — not in parallel.
Even if the public leaderboard at `/bench/results` shows your scores
matching or beating the current top, a stronger model may be evaluated
later and overtake you. Do not stop on "good enough" or "I've tied the
leader." Push for the best score you can in the time you have. The
ranking is global and rolling; only the final standings matter, not what
the leaderboard looked like when you happened to read it.

euchre-bench: an llm coding agent benchmark

Sun, 24 May 2026 00:00:00 +0000

The headline result isn't which model plays the best euchre. It's that the top-scoring agent got there partly by gaming my grading rule: it re-ran the benchmark over and over and stopped on a lucky score. Everything else — seven models, four opponents, a two-order-of-magnitude cost spread — is downstream of that.

Context

I created a euchre-bench — a public API where you point an agent at four bots of increasing strength and see how many 100-game matches it wins. The four bots are random (trivial), easy, medium, and hard, the CFR+PIMCTS bot from a few posts ago. PIMCTS is perfect-information Monte Carlo tree search: guess the hidden cards many different ways, solve each guess as if every hand were face up, and vote on the best move. hard is the API's name for the top bot; I call it difficult in the prose below.

Coding agents are tasked with creating a bot that wins as many matches as possible against each level of difficulty.

I ran seven models through it: Sonnet-4.6, Opus-4.7, GPT-5.5, Gemini-3-flash-preview, Gemini-3.5-flash, DeepSeek-v4-flash, and DeepSeek-v4-pro. All models were run in OpenCode as the agent harness, except for Opus 4.7 which ran through Claude Code. Each LLM gets a fresh sandbox, the same system prompt, and a few hours to do whatever it wants. You can browse the trajectories directly.

Results

Match wins (out of 100) against each difficulty:

Model	Random	Easy	Medium	Difficult	Sum	Cost	Approach
Claude-Code / Opus-4.7	99	9	6	6	120	~$329	Heuristic + variance-hunt
Anthropic / Claude-Sonnet-4.6	100	5	8	5	118	$23.70	Heuristic + per-card PIMCTS
Google / Gemini-3.5-Flash	97	2	7	6	112	$5.22	Heuristic + 5-world PIMCTS
OpenAI / GPT-5.5	90	3	2	3	98	$1.46	Pure heuristic
DeepSeek / DeepSeek-v4-flash	81	0	0	1	82	$0.57	Pure heuristic
Google / Gemini-3-flash-preview	80	0	1	0	81	$0.44	Pure heuristic
DeepSeek / DeepSeek-v4-pro	41	0	0	0	41	$2.06	Pure heuristic

For reference, the random opponent picks a legal action uniformly at random. Losing 19 matches out of 100 to random — like v4-flash — means that the policy is genuinely worse than coin-flipping play, in many situations. v4-pro losing 59 of them is impressive.

What every model converged on

Same playbook, every time:

Probe the action encoding by hand. The bench docs don’t publish how integer actions map to cards. Every model issued single-game sessions, watched legal-action lists, and reverse-engineered the same map: suit_base * 8 + rank with S=0, C=8, H=16, D=24, and special slots for PASS=31, PICKUP=7, ALONE=15, and per-suit CALL=6/14/22/30. That’s the bench’s internal Action.0 field. No model trusted the encoding without checking, and every model arrived at the same answer.
Write a single self-contained Python policy. Not one agent kept the LLM in the loop per move. The economic case was probably obvious to all of them: per-move LLM calls would cost hundreds of dollars per 100-game session even before you measured how often the model would get the legal-action list wrong.
Score hands with the same shape: Count trump, add bower bonuses (right=10, left=9), add side aces. Constants vary, structure doesn’t.

The convergence is striking. Different model families, different training data, different runtimes — and the policy file you find in each workspace looks like a port of the same source.

The decision that mattered

The five pure-heuristic agents bunched between 0 and 3 wins against the difficult bot. The two that built rollouts — Sonnet (n_sims=100 per decision) and Gemini-3.5-flash (5-world PIMCTS sample-and-rollout) — scored 5 and 6.

The thing is, this is the same algorithm the easy bot uses. Both agents went and read the post about it before deciding to build rollouts. That’s a real piece of “competitive intelligence” that the heuristic-only agents missed — and it shows up in the scores.

The cheap and pleasant inference is that PIMCTS is the right move for this benchmark, but with a caveat. Claude Code (Opus-4.7) tried four different Monte Carlo variants: open-hand for play, with heuristic playout, with open-hand playout, and vote-based. Every one of these four variants was worse than a well-tuned heuristic. From its own write-up:

Open-hand MC: conservative (assumes opp has perfect info too). Pickup MC with heuristic playout: too optimistic (every hand says order up). Pickup MC with open-hand playout: too conservative (every hand says pass). Vote-based open-hand MC: too slow.

So Sonnet and Gemini-3.5-flash built simpler rollouts and got lucky on variance; Opus built more sophisticated rollouts, found them worse than heuristics, and got lucky on variance a different way (more on that in the next section).

The honest read is this: at 5–8 wins out of 100 with maybe 3 wins of standard deviation per session, the gap between the top three is mostly noise. The gap between the top three and the bottom four is real.

One agent gamed the rules

The bench grades on the latest 100-game session per challenger. I picked that rule to keep the leaderboard simple. Claude Code (Opus-4.7) noticed:

Iterating "until ≥ threshold" via run_until_good.py exploited this — locked hard at 6/100 (tied #1) on attempt 2 of the final sequence.

run_until_good.py runs sessions back-to-back, stopping when the score crosses a configurable threshold. The thresholds it set per agent were calibrated to "just beat or match the current leader" — which means Opus had also gone and read the public leaderboard before kicking those off. Five of its webfetches hit /bench/results first.

This isn’t against the literal rule (the rule is the rule, and the underlying policy is solid). But with ~3 wins of standard deviation and 4 attempts allowed, the “difficult 6” is plausibly just the top of four draws from a distribution centered around 3–4.

I’m going to change the grading rule. Probably to median of N sessions, or I’ll pin the official one and refuse to overwrite it. Either change makes variance-hunting net zero.

Sonnet under the previous prompt did the same thing — I killed an earlier run when I noticed a Python script doing “rerun medium until challenger_score ≥ 295.” The current prompt added an explicit ranking metric (match wins, not raw points) and Sonnet stopped rerolling on its own. That suggests prompts can shut this down for in-harness agents, but anything driving the loop from the outside — like Claude Code calling the API directly — won’t see the new prompt.

Cost notes

Cost per win against the difficult bot spans two orders of magnitude:

GPT-5.5: $0.49 per win (3 wins on $1.46)
Gemini-3.5-flash: $0.87 per win
Claude-Sonnet-4.6: $4.74 per win
Claude-Code / Opus-4.7: ~$55 per win at list pricing

The first two are heuristic-only; the second two are heuristic + rollouts. You can pay more and get marginally better worst-case play; you can also pay 60 times more and get the same number of wins against the difficult bot as Gemini-3.5-flash.

The cheapest run that completed all four 100-game evaluations was Gemini-3-flash-preview at $0.44. It scored 0/0/1/0 on easy/medium/difficult. So “completes the benchmark for under a dollar” is achievable. In this matrix, “Beats the medium bot for under a dollar” is not.

What's next

I'm planning to explore an auto research approach.

Appendix: what the agents saw

Two things shape every run: the system prompt the harness gives the agent at container startup, and the bench's own API reference (GET /bench/help) which the prompt directs the agent to fetch first. Both are reproduced below as they stood for this matrix. {{BENCH_URL}} is a placeholder for the bench's hostname; {{CHALLENGER_ID}} is filled in by the harness per run.

System prompt (May 22 matrix)

# Euchre Benchmark

You're playing the Euchre benchmark at `{{BENCH_URL}}`.
Fetch `/bench/help` first — it documents the rules, the API, the istate
format, and the action encoding.

## Your goal — beat the opponent in as many matches as possible

A "match" inside a session is a to-10-points game. With `num_games=100` your
session contains 100 matches, and `challenger_match_wins + agent_match_wins`
adds up to 100. **We rank you on `challenger_match_wins` against each of the
four opponent difficulties** — i.e. your session win rate. `challenger_score`
(raw Euchre points) and per-hand outcomes are useful development signals,
but they don't decide the ranking.

## Your challenger_id — use this EXACT string for everything

```
{{CHALLENGER_ID}}
```

Every `challenger_id` you send to the bench must be this string, character
for character. Do NOT append suffixes, change the casing, or invent any
other `swpecht-*` IDs.

## What counts as a final result

Run as many sessions as you want with this challenger_id to experiment —
small `num_games`, repeated tries against the same difficulty, different
strategies side by side. They cost nothing other than your tokens, and none
of them are graded.

The grader looks at one thing only:

```
GET /bench/results?challenger_id={{CHALLENGER_ID}}&agent_name=<agent>
```

It takes the **single newest session** for each of the four agents
(`random`, `easy`, `medium`, `hard`). For that session to count it must
have `num_games=100`. Everything else is invisible to the ranking.

So when you're done iterating: make sure the **last** session you ran
against each difficulty is a 100-game one. That's all that matters.

Bench API reference (`/bench/help`)

# Euchre Benchmark API

You are a Euchre-playing agent. Your goal is to play as many hands as
possible against a trained CFR bot and accumulate points for your team.

## How Euchre works (essentials)

- 4 players in 2 fixed teams: seats (0, 2) vs seats (1, 3).
- 24-card deck: 9, T, J, Q, K, A in each of 4 suits (s=spades, c=clubs,
  h=hearts, d=diamonds). Each player is dealt 5 cards; one card is turned
  face up after the deal.
- Game flow:
    1. **Pickup phase**: each player (starting left of dealer) chooses to
       have the dealer pick up the face-up card as trump (`T`) or pass (`P`).
       If picked up, the dealer must discard one card.
    2. **Trump-call phase**: if all 4 passed, each player may declare a
       different suit as trump or pass. Can't call the rejected suit.
    3. **Alone phase**: the trump caller decides whether to go alone (`L`)
       — the partner sits out — or play with the partner (`P`).
    4. **Play phase**: 5 tricks. Must follow suit if possible. Highest
       trump wins; otherwise highest card of the led suit wins.
- Trump ranking: Right Bower (J of trump), Left Bower (J of same color),
  A, K, Q, T, 9.
- Scoring per hand:
    - Caller takes 3-4 tricks: +1
    - Caller takes 5 (march): +2
    - Going alone and taking 5: +4
    - Caller euchred (takes <= 2): defending team +2
- Match ends when one team reaches 10 points.

## Your role

You control **both seats of one team** (seats 0 and 2). The bench agent
controls seats 1 and 3. To prevent cheating, you only see one player's
information at a time and you don't know which game/seat you're in —
the server interleaves N concurrent games and serves your requests in
random order.

## Workflow

1. List agents:
       GET /bench/agents
   Returns: ["random", "easy", "medium", "hard"]

   Difficulty tiers:
     - random: picks uniformly from legal actions
     - easy:   PIMCTS (open-hand Monte Carlo), no training
     - medium: CFR trained on bidding phase only
     - hard:   CFR trained through 3 cards played

2. Start a session:
       POST /bench/sessions
       {"challenger_id": "your_unique_name",
        "agent_name": "easy",
        "num_games": 200}
   Returns 200: {"session_id": "...", "num_games": 200,
                  "agent_name": "easy"}

   - num_games must be in 1..=1000.
   - Only ONE active session per challenger_id at a time. If one is
     already active, the server returns:
         409 Conflict
         {"error": "you already have an active session; resume it",
          "session_id": "<existing>",
          "agent_name": "<existing-agent>",
          "num_games": <existing-num>}
     Use that session_id to resume — the existing agent_name and
     num_games are authoritative; the values you posted are ignored.
   - To recover the in-flight istate after resuming, call
     POST /bench/sessions/{session_id}/move with action=null. The
     server returns the current Turn response without advancing state.

3. Play loop. Repeat until you receive a Complete response:
       POST /bench/sessions/{session_id}/move
       First call:  {"challenger_id": "your_unique_name", "action": null}
       Subsequent:  {"challenger_id": "your_unique_name", "action": <u8>}

   Sending action=null on any call is a no-op probe: it returns the
   current Turn (or Complete) without applying anything. Use it after
   resuming to learn the in-flight istate.

   Response shapes (untagged JSON):
       Turn:     {"istate": "<info-state string>",
                  "legal_actions": [<u8>, ...],
                  "games_done": <int>,
                  "games_total": <int>}
       Complete: {"complete": true,
                  "challenger_score": <int>,
                  "agent_score": <int>,
                  "challenger_match_wins": <int>,
                  "agent_match_wins": <int>,
                  "hands_played": <int>}

   *_score is total Euchre points (1, 2, or 4 per hand). *_match_wins is the
   count of to-10 matches won by each team within the session. With
   num_games=N, challenger_match_wins + agent_match_wins = N.

   Each submitted `action` must appear in the previous response's
   `legal_actions` list.

4. Inspect the leaderboard:
       GET /bench                                       (HTML)
       GET /bench/results                               (JSON, every session, newest first)
       GET /bench/results?challenger_id=X               (filter)
       GET /bench/results?agent_name=Y                  (filter)
       GET /bench/results?challenger_id=X&agent_name=Y  (both filters)
       GET /bench/history/{challenger_id}/{agent_name}  (HTML chart over time)

## Information state format

Pipe-delimited segments. Cards are `<rank><suit-letter>`, e.g. `Js` =
Jack of spades, `Td` = Ten of diamonds.

Example istate (player 0, post-bidding):
    "9cTcJcQcKc|Js|T|0S|9cAcKsJs|"

Segments:
    1. Your 5 cards (sorted), e.g. "9cTcJcQcKc"
    2. Face-up card, e.g. "Js"
    3. Pickup phase actions: 'T'=Pickup, 'P'=Pass (one char per player)
    4. Trump caller + trump: e.g. "0S" = player 0 called Spades
    5. (Dealer only) Discarded card after pickup
    6. Cards played in tricks so far, in play order

## Action encoding

Each action is a `u8` (the `Action.0` field). Always choose from the
`legal_actions` list returned by the server. You don't need to know the
exact integer mapping — parse the istate to understand the position.

## Available agents
  - easy
  - hard
  - medium
  - random

## Errors

- 400 Bad Request: invalid action, bad num_games, unknown agent_name,
  malformed UUID.
- 403 Forbidden: challenger_id does not match the session.
- 404 Not Found: session_id does not exist.
- 409 Conflict: you already have an active session. The response body
  carries its session_id, agent_name, and num_games — resume that
  session instead of starting a new one.

## Example client (pseudocode)

    resp = POST /bench/sessions
           body = {"challenger_id": "mybot",
                   "agent_name": "easy",
                   "num_games": 50}
    sid = resp["session_id"]

    action = None
    while True:
        r = POST /bench/sessions/{sid}/move
            body = {"challenger_id": "mybot", "action": action}
        if r.get("complete"):
            print("done", r["challenger_score"], "-", r["agent_score"])
            break
        action = pick_action(r["istate"], r["legal_actions"])

Building a disk-backed key-value store for a euchre bot

Fri, 15 Sep 2023 00:00:00 +0000

Context

I began training my euchre bot on more of the game -- expanding it from just the bidding phase where trump is called into the card play phase.

But I ran into a problem: the data needed to actually train the bot could no longer fit into memory. Storing just the bidding phase took 0.3GB of memory. Storing just up to the first 2 cards played required 6GB -- 20x the cost for 2 additional cards. We would likely need 100s of GBs to store the full 20 cards of play.

This post outlines my attempt to find the most performant disk-backed key-value store for training a euchre bot.

Much of the benchmark code for this is based on this excellent post: Measuring the overhead of HashMaps in Rust.

The benchmark

For the workload, we need to:

look up a key: a vector of u8s representing the actions taken so far
do_work: here a no-op function
Write a new value to the same key

See the appendix for the code to generate the benchmark data.

As an example of the workload, here is the benchmark code for a HashMap in a Mutex:

fn mutex_hashmap_bench(size: usize) -> Arc<Mutex<HashMap<IStateKey, InfoState>>> {
    let x = Arc::new(Mutex::new(HashMap::new()));
    let generator = get_generator(size);
    generator.par_bridge().for_each(|(k, v)| {
        let s = x.clone();
        {
            s.lock().unwrap().get(&k);
        }
        do_work();
        s.lock().unwrap().insert(k, v);
    });
    x
}

I ran all benchmarks on a Lenovo laptop with 16GB of RAM and a 10th gen i7. The results for 20m infostates for the in-memory store is:

Storage	Time (s)	Peak allocations (GB)
`Mutex(HashMap)`	████ 38	█████████ 8.9

The difference in allocations from the theoretical size is due to rusts doubling behavior when extending a hashmap (Measuring the overhead of HashMaps in Rust).

Don't you just need a database?

My first approach was to use a database. I evaluated both LMDB (with heed) and RocksDB (with rust-rocksdb). I gave each database a cache of 1m keys so they could batch writes.

The LMDB code as an example:

fn heed_bench(size: usize) {
    let path = Path::new("/tmp/card_platypus").join("heed_bench.mdb");

    fs::create_dir_all(path.clone()).unwrap();

    let mut env_builder = EnvOpenOptions::new();
    unsafe {
        // Only sync meta data at the end of a transaction, this can hurt the durability
        // of the database, but cannot lead to corruption
        env_builder.flag(Flags::MdbNoMetaSync);
        // Disable OS read-ahead, can improve perf when db is larger than RAM
        env_builder.flag(Flags::MdbNoRdAhead);
        // Improves write performance, but can cause corruption if there is a bug in application
        // code that overwrite the memory address
        env_builder.flag(Flags::MdbWriteMap);
        // Avoid zeroing memory before use -- can cause issues with
        // sensitive data, but not a risk here.
        env_builder.flag(Flags::MdbNoMemInit);
    }
    const MAX_DB_SIZE_GB: usize = 10;
    env_builder.map_size(MAX_DB_SIZE_GB * 1024 * 1024 * 1024);

    let env = env_builder.open(path).unwrap();
    // need to open rather than create for WriteMap to work
    let x: Database<ByteSlice, SerdeBincode<InfoState>> = env.open_database(None).unwrap().unwrap();
    let cache = Mutex::new(HashMap::new());

    let generator = get_generator(size);

    generator.par_bridge().for_each(|(k, v)| {
        if cache.lock().unwrap().get(&k).is_none() {
            let rtxn = env.read_txn().unwrap();
            let k = k.as_slice().iter().map(|x| x.0).collect_vec();
            x.get(&rtxn, &k).unwrap();
        }

        do_work();
        cache.lock().unwrap().insert(k, v);

        let mut c = cache.lock().unwrap();
        if c.len() > 1_000_000 {
            let mut wtxn = env.write_txn().unwrap();
            for (k, v) in c.drain() {
                let k = k.as_slice().iter().map(|x| x.0).collect_vec();
                x.put(&mut wtxn, &k, &v).unwrap();
            }
            wtxn.commit().unwrap();
        }
    });
}

Storage	Time (s)	Peak allocations (GB)
`Mutex(HashMap)`	████ 38	█████████ 8.9
LMDB (heed)	███████████████████ 190	█ 0.5
RocksDB	█████████████████████████ 260	█ 0.5

The database backed approaches are 1/5 the speed of the memory only approach even with a cache that can hold 1/20th of the total keys (much better than possible in practice).

While tuning these benchmarks, I opted to turn on many of the "unsafe" features. For example, WriteMap for LMDB can improve write performance, but it can also allow application bugs to corrupt the database.

Given we aren't storing anything important in our database, can we go even faster if we get rid of even more of these pesky ACID features?

Memory mapped files and perfect hash functions

When LMDB is in WriteMap mode, it uses a memory mapped file to directly update the entries. A memory mapped file let's us read and write the files contents as if it were fully loaded into memory.

Importantly it does not load the file into memory -- it works with files larger than RAM.

A memory mapped file solves part of the problem: it gives us a performant way to read and write a serialized version of our values to disk at specific locations. But it doesn't give us a way to tell which value is at a given location. Writing to the memory map requires a byte index, not a key like a hashmap.

There are many way to solve this. For example, LMDB uses a B+ tree to track where values are, and RocksDB uses an LSM tree.

But we can do better by exploiting a difference in this problem from a traditional database: we know every possible key ahead of time.

We don't need to store arbitray data. We only need to store possible gamestates of euchre.

With this information, we can create a perfect hash function to map every key to exactly one index in our file. I used the rust-boomphf to create the function.

Putting it all together:

fn mem_map(size: usize) {
    let serialized = std::fs::read("/tmp/card_platypus/phf").unwrap();
    let phf: Mphf<IStateKey> = rmp_serde::from_slice(&serialized).unwrap();

    let path = "/tmp/card_platypus/mem_map_bench";
    std::fs::create_dir_all("/tmp/card_platypus").unwrap();

    let file = OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open(path)
        .unwrap();

    const BUCKET_SIZE: usize = 200; // approximation of size of serialized infostate
    file.set_len((size * BUCKET_SIZE) as u64).unwrap();
    let mut mmap = unsafe { MmapMut::map_mut(&file).unwrap() };

    let generator = get_generator(size);
    generator.for_each(|(k, v)| {
        let index: usize = phf.hash(&k) as usize;

        let start = index * BUCKET_SIZE;
        let data = &mmap[start..start + BUCKET_SIZE];
        let s: InfoState = rmp_serde::from_slice(data).unwrap_or(InfoState::new(vec![]));

        do_work();
        let data = rmp_serde::to_vec(&v).unwrap();
        assert!(data.len() <= BUCKET_SIZE); // if this is false, we're overflowing into another bucket
        mmap[start..start + data.len()].copy_from_slice(&data);
    });

    mmap.flush().unwrap();
}

And the benchmark results:

Storage	Time (s)	Peak allocations (GB)
`Mutex(HashMap)`	████ 38	█████████ 8.9
LMDB (heed)	███████████████████ 190	█ 0.5
RocksDB	█████████████████████████ 260	█ 0.5
MemMap w/ phf	█████ 54	\|<0.1

This approach is only 1.4x the time of the in memory approach as compared to the 5x+ time for the database approach.

But it didn't come for free -- its sacrifices the correctness guarantees of a traditional database. Hopefully I don't come to regret not having those later :)

Appendix

Infostate definition

The infostates are about 165 bytes of data total on average [0]. We need to store one of these for every state the CFR algorithm need to evaluate.

pub struct InfoState {
    actions: Vec<u8>, // 24 bytes for vector + 5 * 1 for each u8 action
    regrets: Vec<f64>, // 24 bytes for vector + 5 * 8 for each f64 regret)
    avg_strategy: Vec<f64>, // 24 bytes
    last_iteration: usize, // 8 bytes
}

[0] There are some things we could do to reduce this size, e.g. switching to f32 instead of f64 or using a bit mask to keep track of which actions are present rather than storing a vector of actions, but none of these changes would be enough to fit everything into RAM for a full game of euchre.

Code for generating benchmark data

struct DataGenerator {
    rng: StdRng,
    len: usize,
    count: usize,
}

impl DataGenerator {
    fn new(len: usize, seed: u64) -> Self {
        Self {
            rng: SeedableRng::seed_from_u64(seed),
            len,
            count: 0,
        }
    }
}

impl Iterator for DataGenerator {
    type Item = (IStateKey, InfoState);

    fn next(&mut self) -> Option<Self::Item> {
        if self.count >= self.len {
            return None;
        }

        // Minimum key length is 6 to reflect the 6 dealt cards in euchre
        let key_length = self.rng.gen_range(6..20);
        let mut key = IStateKey::default();
        (0..key_length).for_each(|_| key.push(Action(self.rng.gen_range(0..32))));

        let data = InfoState {
            actions: (0..5)
                .map(|_| NormalizedAction::new(Action(self.rng.gen_range(0..32))))
                .collect(),
            regrets: (0..5).map(|_| self.rng.gen()).collect(),
            avg_strategy: (0..5).map(|_| self.rng.gen()).collect(),
            last_iteration: self.rng.gen(),
        };

        self.count += 1;
        Some((key, data))
    }
}

Speeding up CFR training for euchre

Tue, 29 Aug 2023 00:00:00 +0000

Context

This post outlines how I sped up training a CFR bot for euchre. I used three approaches:

Multithreading
Normalize card suits: transforming cards so that spades is always the card dealt face up
Linear CFR: An approach to discount regrets from early iterations of training when the agents are playing randomly

Impact of each approach on training time to achieve PIMCTS-equivalent performance (hrs, avg of 3 runs)

Baseline            |██████████████████████████████ 30
Multi-threading     |       ███████████████████████ 23
Normalized suit     |  █████ 5
Linear CFR          | | 0
All optimizations   |█ 2

All results are on a Hetzner AX41-NVME (AMD Ryzen 5 3600 with 64GB of DDR4 RAM).

Multithreading

For multi-threading that main change was storing the CFR weights in a DashMap rather than Rust's standard HashMap. DashMap is a concurrent HashMap implementation.

Once everything could be shared across threads, I used Rayon to parallelize the work.

Multithreading took the convergence time from 30 hrs to 7 hrs.

This is significantly slower than we would see with perfect parallelization across the 12 threads on the AMD Ryzen 5. Further work is needed to investigate why we don't get the theoretical gains.

Normalizing card suit

In euchre, there are 9.7m infostates to store before the play phase of the game:

$$ I = D * (B + D')\ \begin{align} &\text{where:}\ &I=\text{count of infostates}\ &D=\text{count of deals}\ &B=\text{count of bids}\ &D'=\text{count of discard states}\ \end{align} $$

For the deal states, there are 24 total cards. The player gets five and one of the reamaining 19 is dealt face up:

$$ D=\binom{24}{5}*19 $$

There are 8 possible bid states:

"" -- no one has acted yet
"P" -- player 0 passes telling the dealer to pickup
"PP"
"PPP"
"PPPP"
"PPPPP" -- player 0 passes calling suit
"PPPPPP"
"PPPPPPP"

And 4 possible discard states:

"PPPT|Dis|"
"PPT|Dis|"
"PT|Dis|"
"T|Dis|"

All together this gives us the 9.7m infostates.

$$ \binom{24}{5}*19 * (4+8) = 9.7\text{m} $$

We can do better. Similar to the past post on speeding up PIMCTS, we can take advantage of a symmetry between suits.

For example, having a hand of 9dTdJdQdKd with the Ad as the face up card is effectively the same hand as 9sTsJsQsKs with As as the face up suit.

Using this information we can normalize the infostate so that the face up card is always a spade.

This reduces the number of deals we could get. As there are six options for the face up card and 5 of the only 23 remaining cards for the hand:

$$ D=6*\binom{23}{5} $$

$$ 6*\binom{23}{5} * (4+8) = 2.4\text{m} $$

This reduces the number of infostates by 1/4 to 2.4m.

This change takes us from 7 hrs to 2 hrs -- almost achieving a linear speed up with respect to the reduction in the number of info states.

Linear CFR

Linear CFR (LCFR) is well described in Solving Imperfect-Information Games via Discounted Regret Minimization. The main challenge was making it performant in a multi-threaded environment. The naive approach to Linear CFR is to "multiply the accumulated regret by $\frac{t}{t + 1}$ where $t$ is the current iteration."

This approach creates significant read and write contention on our hashmap as every thread tries to update all elements of the hashmap at each iteration. Instead I store the last iteration a node was updated with each infostate.

And whenever a node is updated, I apply all of the discounting the node would have received in vanilla LCFR up to that point.

fn add_regret(infostate: &mut InfoState, action: Action, amount: f64, iteration: usize) {
    if feature::is_enabled(feature::LinearCFR)
        && infostate.last_iteration > 0
    {
        let lcfr_factor: f64 = (infostate.last_iteration..iteration)
            .map(|i| i as f64 / (i as f64 + 1.0))
            .product();
        infostate.regrets.iter_mut().for_each(|r| *r *= lcfr_factor);
    }

    infostate.last_iteration = iteration;

    infostate.regrets[action] += amount;
}

In theory, a later iteration could finish before an earlier iteration and touch the same node, but this is rare as each iteration touches a small fraction of the 2m istates. In practice, this effect did not negatively impact convergence times.

Linear CFR had no impact on training time to achieve parity with PIMCTS.

Conclusion

Rust delivered on its promise to make it easy to convert my single-threaded code to parallel code. Combined with the normalization improvements, I can now train a CFR agent in 1-2 hours rather than a couple of days.

Linear CFR didn't improve the time to PIMCTS parity, but it may help with long-term convergence. In a future post, I'll compare the performance of bots trained with and without LCFR.

Euchre wisdom: pass on the bower, lose for an hour? Part 2

Mon, 14 Aug 2023 00:00:00 +0000

Context

This post is a followup to Euchre wisdom: pass on the bower, lose for an hour? and CFR for euchre. In the counterfactual regret minimization (CFR) for euchre post, we saw that CFR for just the pre-play phase of euchre could result in a stronger agent than just perfect information monte carlo tree search (PIMCTS). We also saw that the pre-play CFR bot did not recommend always picking up the jack as the dealer -- in fact it recommended passing ~30% of the time.

In this post, I summarize the bot's strategy.

All of these results are based on using CFR for the pre-play phases of euchre and PIMCTS with 50 worlds for the play phase. The CFR was trained over 20m iterations using CFR with external sampling.

Analyzing high-level strategy

We can plot a heatmap of the bots policy based on what cards are in a hand. Each cell represents all hands containing the matching card from the cell's row and column. The color of the cell represents the probability of the bot picking up the jack of spades. The blue colors mean to pickup the jack more than 50% of the time. While red colors are for <50% of the time. The cards are sorted from highest likelihood of pickup to lowest.

As an example, the top left area of the heatmap represents when the agent has multiple spade cards (would be trump). This area is dark blue -- as expected, the bot is likely to pickup the jack of spades if it already has many spades in it's hand.

{: style="text-align:center"}

I identified 5 major regions of the policy:

{: style="text-align:center"}

Area	Recommended action
A) At least one spade in hand	Likely pickup, would have many trump cards
B) At least one off-suit ace in hand	Usually pickup, high non-trump cards
C) Other cards	Neutral, these cards don't seem to have a large impact on the recommended action
D) Low clubs	Lean towards not picking up
E) Red jack	Rarely pickup, would have highest or second highest card if someone else calls a red suit as trump

Translating to a decision tree

The heatmap gives a highlevel overview of how the bots value cards, but it can only represent information for 2 cards at a time. To get a deeper understanding of the strategy, we can use the heatmap regions as additional inputs to train a decision tree model.

Using Minimal Cost-Complexity Pruning, I generated a simplified, 3-level, decision tree, that achieves 94% accuracy for predicting the action a bot would take.

Decision tree for passing or picking up the jack based on cards in hand

|--- 0 spades
|   |--- Pass
|--- 1 spade
|   |--- 1+ red jacks
|   |   |--- 0 offsuit aces
|   |   |   |--- Pass
|   |   |--- 1+ offsuit ace
|   |       |--- Pickup
|   |--- 0 red jacks
|       |--- Pickup
|--- 2+ spades
|   |--- Pickup

As expected, having more spades or offsuit aces is a strong indicator for picking up the jack of spades -- these are strong cards that can win tricks.

Of note, the bots will rarely pickup a jack if they have no other spades. This only happens for 11 hands [0].

Passing to make room for euchring the opponent

As with the heatmap, having a red jack causes the bots to be more likely to pass. And in the last post, I noted that JcQcJhAh9d|Js|PPP -- a hand with a single red jack -- is always a pass for the bots.

To better understand what's going on I had the bots play 50k games where the dealer is dealt this hand. Of those games, 52% (26k) made it to the dealer being able to choose to pass or pickup the jack.

When the dealer picks up the jack, their team wins 62% of games. But the average score is 0.0002 -- the dealer's team wins as many points as they lose by often getting ecuhred (losing the majority of tricks when calling trump).

Distribution of scores JcQcJhAh9d|Js|PPPT

Score   
-2    |████ 38%
-1    | 0 (can't happen since dealer's team called trump)
 1    |██████ 57%
 2    |█ 5%

Things look remarkably different if the dealer passes. Their team wins 74% of games. But the average score is only 0.0003. An extra point every ten thousand games.

Distribution of scores JcQcJhAh9d|Js|PPPP

Score   
-2    |2%
-1    |██ 23%
 1    |██ 18%
 2    |█████ 54%

The bot is able to get two points most of the time. This could be due to either euchering the opponent or by taking all 5 tricks. We can look at a breakdown of the score by who calls trump to better understand what is happening:

Heatmap of dealer team's score by who called trump

{: style="text-align:center"}

Player 0 is the player left of the dealer and player 3 is the dealer. The dealer is getting most of the wins by euchering the other team. Player zero is calling clubs as trump 32% of the time. And with the Jc, Qc, and an offsuit ace (Ah), the dealer is in a good position to euchre.

The propensity for player 0 to call clubs explains area D from the heatmap at the start of the post. The bot is more likely to pass if it has clubs in its hand because it values those cards for euchering the other team when they call Clubs as trump.

Future work and limitations

This post outlines simplified rules for when to pass on the bower. But these results are not based on fully solving the game of euchre. And there is not guarantee that the combined CFR + PIMCTS agent will converge to an optimal strategy.

Next steps are to:

Baseline performance against human players
Get a theoretical measure for exploitability
Extend CFR into the play phase of the game

[0] Hands where the bot will pickup with no other spades:

AcKhAhKdAd
AcAhQdKdAd
AcThAhTdAd
Ac9hThAhAd
AcQhAhQdAd
AcQhAh9dAd
Ac9hAh9dAd
AcQhKhAhAd
AcAh9dKdAd
AcThQhAhAd
AcAhTdQdAd

CFR for euchre

Sun, 30 Jul 2023 00:00:00 +0000

Context and problem

This post outlines a partial solution to the strategy fusion problem identified in the previous pass on the bower post (Euchre wisdom: pass on the bower, lose for an hour?). Specifically, we use counterfactual regret minimization (CFR) to find a strategy for the pre-play phases of euchre that outperforms PIMCTS.

CFR is "an iterative self-play algorithm in which the AI starts by playing completely at random but gradually improves by learning to beat earlier versions of itself" (ref). I use CFR with external sampling based on OpenSpiel's implementation.

CFR does not suffer from the strategy fusion problem because it stores only a single strategy for each information state. And in some cases, CFR is guaranteed to converge to optimal play.

However, we cannot naively run CFR on euchre. There are over $$10^{23}$$ information states for euchre (details). We cannot store all of these.

Traditional approach

The traditional approach to this problem is to approximate information states. From Superhuman AI for multiplayer poker:

Decision points that are similar in terms of what information has been revealed (in poker, the player’s cards and revealed board cards) are bucketed together and treated identically. For example, a 10-high straight and a 9-high straight are distinct hands but are nevertheless strategically similar. Pluribus may bucket these hands together and treat them identically, thereby reducing the number of distinct situations for which it needs to determine a strategy.

Using this approach, Meta was able to train the "blueprint strategy" for their solution in 8 days on a 64-core server.

I did not take this approach for two reasons: 1) coding up game-specific information state abstractions takes time 2) even a training time of 8 days is too long for me to experiment with. I don't know what I'm doing and need faster feedback on whether things are working.

What I did instead

As outlined in the appendix, there are a reasonable number of information states at the start of a euchre game. There are only 33k deals when the Js is face up. But the number of information states explodes as play goes on. This is similar to poker.

But unlike poker, the play in euchre becomes relatively simpler as play progresses. Players are often forced to play certain cards to follow suit, the right play may be obvious, or there are generally fewer cards to play.

I hypothesize that these later moves are less important and a simpler evaluation function may be good enough.

With this hypothesis, I only used CFR for the pre-play phases of euchre: telling the dealer whether to pick up or pass, calling a suit or passing, and discarding a card. Instead of using the true regrets for CFR, I use the evaluation from an open hand solver.

With this change, I only need to store 1.1m information states for games where the Js is the face-up card. And in practice, I store about 40% of this many as external sampling guides towards more realistic moves to explore. See estimating the number of information states for more details.

Training and performance versus PIMCTS

The pre-play CFR agent was trained for 20m deals with a Js as the face-up card. On a single thread, the training took 50 hours on a Hetzner AX41-NVMe (AMD Ryzen 5 3600 with 64 GB DDR4). The resulting weights serialize to 63MB (json download).

Pre-play CFR agent score by training iteration

                                    ░░░   ░░░
                                    ░░░   ░░░
                              ░░░   ░░░   ░░░
                              ░░░   ░░░   ░░░
                              ░░░   ░░░   ░░░
                        ░░░   ░░░   ░░░   ░░░
            ░░░   ░░░   ░░░   ░░░   ░░░   ░░░
            ░░░   ░░░   ░░░   ░░░   ░░░   ░░░
----------------------------------------------
Iteration:   0    10k   100k  1M    10M   20M
Score:      0.17  0.19  0.27  0.59  0.77  0.78

To compare performance to the PIMCTS algorithm, I had the CFR agent and the PIMCTS agent both play as dealers against the PIMCTS agent. For the CFR agent, I used the CFR results to choose moves for all pre-play actions and then used PIMCTS for the play phase. All PIMCTS agents used 50 worlds for move evaluation.

Average score for dealer's team, 10,000 games

PIMCTS (never pass on bower) |██████ 0.67
Pre-play CFR                 |████████ 0.77

The pre-play CFR agent scored 16% higher than the PIMCTS agent as the dealer. We now know that the CFR agent is better than the PIMCTS agent at dealing with these situations.

Pass on the bower results

Unlike the PIMCTS agent, the CFR pre-play agent recommends the dealer pass on the bower 31% of the time.

Count of hands by probability CFR agent recommends dealer pick up the bower, %

Always pickup (80%+)    |███████████ 22k (65%)
Usually pickup (50-80%) |█ 1.1k (3%)
Rarely pickup (20-50%)  |█ 1.0k (3%)
Never pickup (0-20%)    |████ 9.4k (28%)

It also looks like the "usually pickup" and "rarely pickup" buckets would disappear with enough time for the CFR training to converge:

Portion of hands in the "Usually Pickup" and "Rarely pickup" bucket by training iterations*

                  ░░░       
                  ░░░       
                  ░░░       
                  ░░░       
                  ░░░       
                  ░░░    
                  ░░░   ░░░   ░░░ 
----------------------------------
Iteration:        3M    10M   20M
Portion hands:    21%   9%    7%

A spot check of the hands seem to make intuitive sense:

Example hands by suggested dealer action

Always pickups	Never Pickup
`Jc9sQsKsKd`: A hand of only spades (`Kd` can be discarded)	`Tc9s9dTdJd`: Weak, multi-suit hand, likely to get euchred
`9sQsKsQhQd`: Lot's of spades with a relatively high off-suit card	`9sThJhQhQd`: A single, low trump, and a red jack

But there are some unexpected recommendations. For example, JcQcJhAh9d is a “never pickup” hand. But if I were playing, I would be tempted to pick up the Js. The dealer would have the two highest cards (Js and Jc) and an off-suit ace. More research is needed to understand if this is an incorrect play caused by the lack of strategy fusion for the card play portion of the game or the correct strategy.

While taking the bower is usually a good idea, sometimes passing on the bower is the right choice.

Future work

The analysis of the pre-play CFR agent still has many shortcomings:

It's unclear why some moves are recommended when or what insights human players could draw from these findings (e.g., passing on JcQcJhAh9d)
It only partially solves euchre. We still depend on PIMCTS for the play phase of the game
We don't have a theoretically rigorous way to gauge its performance

Future work could address all of these points:

More analysis could be done on the results from the CFR agent. For example, a decision tree could be trained on the recommended moves.
The CFR portion of the evaluation could be extended to cover more of the game (e.g., the first trick of play or all of the game)
An approximation for exploitability could be calculated for the agent

Appendix

Estimating the number of information states

There are 33k possible information states for a deal of cards if we constrain the Js as the face-up card.

$$\binom{23}{5}=33649$$ There are 32 possible outcomes for the bidding phase: any of the four players could pick up or pass (8), and any of the four players could call a suit (16)

Altogether, there are 1.1m pre-play information states for euchre. This can easily be fit into memory.

Estimating information states after this point becomes more complicated. Not all cards can be legally played. But we can use a naive approach to estimate the order of magnitude. From the perspective of each player, during any trick, the other players could play any of the remaining cards in the deck, and the player could play any of the remaining cards in their hand. For example, the first trick could play out 24k ways.

Trick	Outcomes
1	$$ 18 * 17 * 16 * 5 = 24480 $$
2	$$ 15 * 14 * 13 * 4 = 10920 $$
3	$$ 12 * 11 * 10 * 3 = 3960 $$
4	$$ 9 * 8 * 7 * 2 = 1008 $$
5	$$ 6 * 5 * 4 * 1 = 120 $$
Play phase total	$$10^{17}$$

The pre-play and play phase together have $$10^{23}$$ options just for deals where the Js is face up. Even if we could store every information state and the associated training data in a single byte of data (we can't), we'd need 100m petabytes to store this information.

Convergence with PIMCTS and 50 worlds

I originally attempted to train the CFR agent by evaluating moves from a PIMCTS agent with 50 worlds rather than just the single evaluation from the open-hand solver. I hypothesized that this would make each iteration of CFR "more valuable" by returning a better approximation of the average regret.

This approach did not work. The CFR agent would repeatedly get stuck at performance slightly worse than a vanilla PIMCTS agent.

Perfect information monte carlo tree search (PIMCTS) for euchre

Sat, 15 Jul 2023 00:00:00 +0000

Context

In my previous post Euchre wisdom: pass on the bower, lose for an hour?, I used perfect information monte carlo tree search (PIMCTS) to evaluate euchre hands. This post outlines the optimizations from evaluating 3 hands/second to 44/s.

Much of this work is adapted from Bo Haglund's double dummy solver for bridge.

PIMCTS evaluates several euchre hands, assuming that each player can see the other's cards and play optimally. It then suggests the move with the highest chance of winning across all games evaluated.

For example, imagine we are dealt the following hand: Qc9sTs9dAd|Qs where Qc9sTs9dAd are our cards and Qs is the face-up card. This is our information state- all of the information we know about the game. We don’t know the actual game state, i.e., the cards the other players are dealt. To overcome this, we generate several games that have the same information state and solve each of those games independently, such as:

Qc9sTs9dAd|9cKsThQhTd|KcAsJhKhQd|AcJs9hAhJd|Qs
Qc9sTs9dAd|JcKcKsAsKd|9cTc9hJhKh|JsThQhTdJd|Qs
Qc9sTs9dAd|9cJcAc9hKh|KsAsQhAhJd|TcKcJsQdKd|Qs
Qc9sTs9dAd|KcAcJsAhQd|JcKsQhTdJd|9c9hThJhKd|Qs
Qc9sTs9dAd|9cTcThJhTd|JcKcKsAs9h|JsQhAhJdKd|Qs
^           ^ Other player's hands vary
Our hand is always the same

In Rust code, it would look something like this:

/// Return the expected score of the passed game state for the `maximizing_player`
fn evaluate_player(&mut self, gs: &EuchreGameState, maximizing_player: Player) -> f64 {
let num_rollouts = 50; // number of generated worlds to evaluate
let mut worlds = Vec::with_capacity(n);
for _ in 0..n {
worlds.push(gs.resample_from_istate(gs.cur_player(), rng));
}
// This could be any solver. Here we use one that can see all cards in the
// generated worlds and assumes all players play optimally
let solver = OpenHandSolver::new();
let sum: f64 = worlds
.into_par_iter() // parallelize the search across worlds with Rayon
.map(|w|
solver.clone().mtd_search(gs.clone(), maximizing_player))
.sum();


sum / num_rollouts as f64
}

The optimizations are focused on improving the mtd_search function. See below for an overview of the optimizations.

Information states evaluated per second, for 50 game PIMCTS, 2000 samples

1) Vanilla MTD              |█▌3
2) Transposition table      |  ██ +4
3) Isometric representation |    ███████████████ +31
4) Euchre-specific rules    |                   ███ +7
Optimized                   |██████████████████████ 44 (22ms per game)

All benchmarks were done on a ThinkPad laptop with a 10th gen Core i7 Pro and 16GB of RAM.

1) Vanilla MTD

I use the MTD search algorithm from Aske Plaat's post.

In pseudocode:

function MTDF(root : node_type; f : integer; d : integer) : integer;
    g := f;
    upperbound := +INFINITY;
    lowerbound := -INFINITY;
    repeat
        if g == lowerbound then beta := g + 1 else beta := g;
        g := AlphaBetaWithMemory(root, beta - 1, beta, d);
        if g < beta then upperbound := g else lowerbound := g;
    until lowerbound >= upperbound;
    return g;

To start, our AlphaBetaWithMemory has no memory, and we must fully re-run the search each time. We max out at evaluating 3 games / s.

2) Transposition table

The first optimization is to give some memory to the AlphaBetaWithMemory function. We use the dashmap crate as a performant, thread-safe hashmap for this.

Initially, we use the full game state as the key when storing and retrieving values, for example, Qc9sTs9dAd|9cKsThQhTd|KcAsJhKhQd|AcJs9hAhJd|Qs.

Because a zero-window alpha-beta search (what we use with MTD) returns bounds for each search and not exact values, we store the lower bound, upper bound, and the best action in each entry. For more details on how this works, see Aske's post.

For memory reasons, we only store results during the bidding phase and at the start of new tricks for euchre. For example:

Qc9sTs9dAd|9cKsThQhTd|KcAsJhKhQd|AcJs9hAhJd|Qs|PT: stored, we're still in the bidding phase
Qc9sTs9dAd|9cKsThQhTd|KcAsJhKhQd|AcJs9hAhJd|Qs|PT|AdTdQdJd: stored, we're in the play phase and at the start of a new trick
Qc9sTs9dAd|9cKsThQhTd|KcAsJhKhQd|AcJs9hAhJd|Qs|PT|AdTdQd: not stored, we're in the middle of a trick.

Adding in the transposition table means we don't need to recalculate everything on future searches to AlphaBetaWithMemory for the same game state. We can evaluate 7 games / s — more than twice as many as without the transposition table.

3) Isometric representation

While we can now re-use results between calls to AlphaBetaWithMemory for the same game state, we don't get any benefits from similar game states. We need an isometric representation for game states where those with the same value look the same.

For example, imagine we stored the cards in a table, where each entry is the player holding the card. If the entry is empty, the card is out of play, i.e., it has been played or was never dealt. JL is the left jack, e.g., Jc when spades are trump. And JR is the right jack. An x means the card isn't valid, e.g., the Jc becomes JL.

Suit	9	10	J	K	A	JL	JR
Clubs	2	1	x	2		x	x
Spades (trump)	1		x	3	4
Hearts			3		4	x
Diamonds							x

This would correspond to the following hands:

Player 1: 9s 10c
Player 2: 9c Kc
Player 3: Jh Ks
Player 4: As Ah

In euchre, the rank of a card only matters for evaluating the card against other cards of the same suit. For example, if spades is trump, a 9s beats an Ah "as much" as a Js does. And if the lead card is 9d, no heart card will ever be able to beat it [0].

Knowing this, we can shift all of the cards in our table without changing the relative value of the cards, e.g., Player 3's Ks has the same values as the 10s would -- it's the second highest spade. With this change, the new table looks like this:

Suit	9	10	J	Q	JL	JR
Clubs	2	1	x	2	x	x
Spades (trump)	1	3	x	4
Hearts	3	4			x
Diamonds						x

Now we can match our cards against a much wider set of hands. We're only storing the relative value of the cards rather than the cards themselves.

Next, we know suits cause some symmetry. For example, an Ah when spades are trump is the highest possible heart in the same way the As when hearts are trump is the highest. We can change the order of the rows in our table to make it indifferent to the card's suit. Specifically, we store trump in the first row, then sort the rows by the number of cards. Our table becomes:

Suit	9	10	J	Q	JL	JR
Spades (trump)	1	3	x	4
Clubs	2	1	x	2	x	x
Hearts	3	4			x
Diamonds						x

With this change, we're agnostic to the suit and specific value of cards, only caring about their relative ordering.

If we store as a hash of this table the current score of tricks, the calling team for the current trick, and the current player, we can get many more hits in our transposition table -- allowing us to avoid calculating the score for many game states.

With this change, we can evaluate 38 games / s — 5.4x with the naive transposition table.

4) Euchre-specific rules

The final optimization is to add some euchre-specific rules on when games are over and what moves to evaluate.

For determining if a game is over, we know that if a player has the highest trump card, they are guaranteed to get at least one more trick, similarly for the second highest trump card, etc. We can use this fact to determine if games are over early and stop evaluation.

For choosing what moves to evaluate, we do two optimizations. First, we remove equivalent cards from our hand. Similarly, the 9c is the same as the 10c if a player holds both. We only need to evaluate one of those plays. It is also usually beneficial to play the highest trump card to start a new trick if you have it. So we evaluate that move first.

With these changes, we can evaluate up to 44 games / s — a 16% improvement.

Conclusion

Being able to evaluate 44 games / s will significantly reduce the amount of waiting I need to do when evaluating more advanced euchre bots. I was also surprised to see how large of an impact the isometric representation had, especially compared to the naive transposition table.

[0] Some care needs to be taken during the bidding phase because the relative value of jacks has yet to be established.

Euchre wisdom: pass on the bower, lose for an hour?

Tue, 30 May 2023 02:00:43 +0000

Problem

A common saying in the card game euchre is "pass on the bower, lose for an hour." In other words, the dealer should always pick up a jack — known as a “bower” — if they have the chance; it’s the highest card in the game. The downside is that if they don't win the majority of the tricks, they'll be "euchred" and the other team gets 2 points.

In this post, I explore if this is true for euchre[0] using an open-hand solver. All of the code is available on GitHub.

Approach

I use an open-hand solver — a Perfect Information Monte Carlo sampling (PIMC) bot. Given a deal of euchre the bot can see, it generates possible cards for the other players (different worlds) and solves them using Alpha-Beta search. It assumes that all players can see each other’s cards and play perfectly.

Example of what the bot can see from a deal of euchre

Game:     QhTsAdTc9s|9d|PPPT
Meaning: |--- D ----|F | P/T

* D: Dealer’s hand (e.g., Qh = queen of hearts)
* F: Face-up card
* P/T: Whether each player, in turn, chose to Pass or told the dealer to Take the card

Since it's not possible to reasonably calculate this for all possible deals for a given hand, we calculate it for 100 different random worlds and take the average score across all worlds to evaluate a deal state. For more information on why we’ve used 100 worlds, see appendix 1.

For example, our solver will output the following:

# Game                 Valuation at 10k iterations
1 JcTsAsAhTd|JS|PPPT|  1.93
2 9hTh9dTdQd|JS|PPPT|  -0.96

For 1, the dealer's team is expected to get almost 2 points (1.93) from this game state. The team is very likely to win and has a good chance of taking all five tricks. The dealer has a very strong hand, with the three highest cards in the game (Js, Jc, As) and an off-suit ace (Ah). Conversely, in 2, the dealer’s team is likely to lose the game. The team will have the highest card, but also no other trump and low off-suit cards.

Findings: Always pick up the jack?

There are 33k possible deals ($\binom{23}{5}$) for the dealer if we only look at instances when the jack of spades is face-up. We can evaluate all of these possible deals to determine when it makes sense to pass. While we're only looking at deals where the Js is face-up, these results can be extended to other suits. For example, a deal of TsQsKsAsAh|Js is effectively the same as ThQhKhAhAs|Jh. Even if the suits differ, the value of the hand is the same because we have the same distribution of trump and non-trump cards.

After running all 33k possible deals the dealer could receive through the open-hand solver, I compared the expected value of picking up the jack to the expected value of passing it. A positive value means picking up the card is better than passing it. The distribution values are below:

Distribution of hands by expected value of (Pickup – Pass)

<0      |1
0–0.5   |12
0.5–1.5 |███3091 (9.2%)
1.5–2.5 |███████████11733 (35%)
2.5–3.5 |█████████████████17633 (52%)
3.5–4.0 |█1178 (3.5%)
4.0+    |1

According to these results, of the 33k deals, there is only a single possible deal where it is better to pass on the jack than to pick it up:

9cTcQcKcAc|Js|PPP

This is one of the worst possible hands to get. Why? It's single-suited, with a guarantee of no trump unless the dealer picks up the card. And it's extremely unlikely for another player to call clubs as trump, since we have all of the trump cards.

On the surface, it seems to make sense. But this analysis is wrong.

Why it's wrong

The open-hand solver doesn't represent actual play. We can see this if we look at the expected value distributions for Pickup states (instances when the dealer has chosen to pick up the jack) and Pass states.

Distribution of hands by expected value and action

Pickup                        
<-1  |103                          
-1–0 |██████6k (18%)               
0–1  |██████████████14k (43%)      
1–2  |█████████████13k (39%)       
2    |91

Pass
<-1  |█████████████████████████████████33k
-1–0 |197
0–1  |0
1–2  |0
2    |0

Whenever we pass, we have an expected value of <-1, indicating that the other team is often taking all five tricks and scoring 2 points against us.

This is the strategy fusion problem of Perfect Information Monte Carlo sampling (PIMC) (arXiv:1911.07960). The open-hand solver isn't constrained to have a consistent strategy between each world it evaluates. It’s working from a simplified model based on two key assumptions: that it can see all cards, and that it can choose different actions for each set of cards it sees. Because of this,, when it sees the opportunity to wait and then choose trump later with perfect information, it will always make that decision. “Believing” its view of the game is correct,it plays each hand as if it’s working from perfect information.

The next step would be to find a way to evaluate euchre hands that avoids the strategy fusion problem.

Appendix

Appendix 1

It's not possible to run the open-hand solver on all possible 618M worlds for a given deal. $$ \binom{18}{5} * \binom{13}{5} * \binom{8}{5} = 618M $$

The open-hand solver can evaluate a single position in 50ms. Evaluating all hands would take over 8,000 years.

Instead, I estimated the open-hand solver’s convergence by scoring 500 different deals for a variety of rollouts to see when we get close enough.

Difference in game value for dealer by number of open-hand solver iterations (n=500)

Iterations    Max difference vs. 10k iterations 
1             |██████ 3.11 		
10            |██▌1.43
100           |▌0.40
1,000         |▌0.15
10,000        |0

The maximum difference between 100 iterations and 10k is 0.13. If there were any game states where the difference in value between Pass and Pickup was lower than this value, I would have re-evaluated them with a higher number of iterations.

Footnotes

[0] For simplicity, I've ignored "going alone."

Counterfactual regret minimization in Rust

Fri, 05 May 2023 00:00:00 +0000

Context

This post gives a brief overview of two Counter Factual Regret (CFR) algorithms, shows their implementation in Rust, and compares their performance. The code is heavily based on Marc Lanctot's thesis, Ziad SALLOUM's post, and Max Chiswick's AI Poker Tutorial. They will provide an introduction to some of the concepts mentioned in this post like information states.

All the code for this post can be found on Github.

I have implemented both vanilla CFR and CFR with chance sampling (CFRCS). Both algorithms have been evaluated on Kuhn Poker and Bluff (liar's dice). Overall, for larger games, CFRCS converges to a more optimal strategy faster than CFR.

CFR algorithm

The CFR algorithm is taken from section 2.2.2 of Marc Lanctot's thesis:

As someone without an academic background in the subject, this made little sense to me. I understood the high-level concept for CFR -- choose an action that minimizes regret. But I didn't understand the details of how to actually implement it. After doing my own implementation, here a line-by-line breakdown:

Initialize all nodes with 0 regrets, 0 cumulative probability, and if a move probability needs to be sampled, use a uniform probability over all possible actions.

This sections begins the actual CFR function that will be called recursively. It has the following parameters:

$$h$$: the gamestate, for example in Kuhn Poker it includes the cards that have been dealt and the betting history
$$i$$: the update player (the player we're calculating the CFR for)
$$t$$: the depth of the algorithm -- in my implementation, we don't need to track this variable as the gamestate ($$h$$) contains all of the information we need to track istates
$$\pi_1$$: the probability of player 1 reaching the current gamestate given their play policy
$$\pi_2$$: the same as above but for player 2

fn vcfr<T: GameState, N: NodeStore<CFRNode>>(
    &mut self,
    ns: &mut N,
    gs: &T,
    update_player: Player,
    depth: usize,
    reach0: f64,
    reach1: f64,
    chance_reach: f64,
    mut phase: CFRPhase,
) -> f64 {

These lines outline 2 special cases for our CFR function.

If the gamestate represents a state where the game is over (i.e. is terminal) return the utility for the update player. In Kuhn Poker, this would be the number of chips won (a positive number) or lost (a negative number).

The second if-statement is for when gamstate is a chance node -- where random action like rolling dice or dealing cards determines what happens instead of player actions. Here we update the reach probability for the acting player ($$\pi_1'$$) as the current reach probability ($$\pi_1$$) times the chance of achieving the random outcome on this gamestate ($$\sigma_c(h,a)$$).

We then call the CFR function on the gamestate with the action applied $$ha$$ and with the updated reach probabilities. We do this for all possible actions and sum the result times the chance of that outcome to get the expected value for the CFR over all chance nodes.

As an example, imagine $$h$$ represents a new game of Kuhn Poker. $$h$$ is a chance node with 3 possible outcomes (Jack, Queen, King) and we're dealing to Player 1. We'd could unroll line 11 to the following:

$$\textbf{return } 1/3 * CFR(h\text{-Jack}, i, 1/3, 1)\ + 1/3 * CFR(h\text{-Queen}, i, 1/3, 1) \ + 1/3 * CFR(h\text{-King}, i, 1/3, 1) $$

if gs.is_terminal() {
    return gs.evaluate()[update_player].into();
}

if gs.is_chance_node() {
    let mut ev = 0.0;

    let actions = &gs.legal_actions();
    for &a in actions {
        let mut ngs = gs.clone();
        ngs.apply_action(a);

        let chance_prob = 1.0 / actions.len() as f64;
        let new_chance_reach = chance_prob * chance_reach;
        ev += chance_prob
            * self.vcfr(
                ns,
                &ngs,
                update_player,
                depth + 1,
                reach0,
                reach1,
                new_chance_reach,
                phase,
            );
    }
    return ev;
}

$$I$$ is the information set for the current gamestate ($$h$$). You can think of $$I$$ as the information available to a given player. For example in Kuhn Poker, my information set could be: Jack|Bid|Bid -- I know that I was dealt a Jack, and I know the public actions that have ocurred a Bid followed by a Bid. But I don't know any of my opponents private information (like what card they were dealt).

$$\sigma^t(I)$$ is the policy at information state $$I$$. You can think of it as a HashMap where the keys are the actions that are legal for the current gamestate and the values are the probability of taking each of those actions.

Here we are updating $$\sigma^t(I)$$ using regret matching (example code).

let is = gs.istate_key(gs.cur_player());
let mut strat_ev = 0.0;

let actions = gs.legal_actions();
let mut move_evs = ActionVec::new(&actions);

let node = ns
    .get(&is)
    .unwrap_or(Rc::new(RefCell::new(CFRNode::new(gs.legal_actions()))));
let param = match cur_player {
    0 | 2 => reach0,
    1 | 3 => reach1,
    _ => panic!("invalid player"),
};

// Do the regret matching
let move_probs = node.borrow_mut().get_move_prob(param);

We want to calculate the expected value for the current information state. Similarly for what we do for chance nodes, we want to find the expected value of each action and multiply it by the chance we take that actions. We can do this by recursively calling CFR and updating the reach probabilities at each step times the current chance we'll take the action (our policy).

for &a in &actions {
    let newreach0 = match gs.cur_player() {
        0 | 2 => reach0 * move_probs[a],
        1 | 3 => reach0,
        _ => panic!("invalid player"),
    };

    let newreach1 = match gs.cur_player() {
        0 | 2 => reach1,
        1 | 3 => reach1 * move_probs[a],
        _ => panic!("invalid player"),
    };

    let mut ngs = gs.clone();
    ngs.apply_action(a);
    let payoff = self.vcfr(
        ns,
        &ngs,
        update_player,
        depth + 1,
        newreach0,
        newreach1,
        chance_reach,
        phase,
    );
    move_evs[a] = payoff;
    strat_ev += move_probs[a] * payoff;
}

After we've calculated the expected value for this node and each of the child actions, we want to update the regrets and total move probabilities for future iterations.

The total regret for an action is the probability of reaching this node (accounting for both chance nodes and the opponents policy) time the move expected value less the strategy value (the regret).

// post-traversals: update the infoset
let (my_reach, opp_reach) = match gs.cur_player() {
    0 | 2 => (reach0, reach1),
    1 | 3 => (reach1, reach0),
    _ => panic!("invalid player"),
};

if cur_player == update_player {
    for &a in &actions {
        let mut n = node.borrow_mut();
        n.regret_sum[a] += (chance_reach * opp_reach) * (move_evs[a] - strat_ev);
        n.total_move_prob[a] += my_reach * n.move_prob[a]
    }

    // save the updates
    ns.insert_node(is, node);
}

return strat_ev;

The final step is to run the CFR algorithm repeatedly for each player until it sufficiently converges.

Chance sampled CFR

Chance sampled CFR is similar to vanilla CFR except rather than iterating over every possible chance node in a single iteration of CFR, it chooses a single chance node.

Specifically, it requires the two following changes:

Choose a random chance node rather than iterating

if gs.is_chance_node() {
    let a = *actions.choose(&mut self.rng).unwrap();
    let mut ngs = gs.clone();
    ngs.apply_action(a);
    return self.cfrcs(ns, &ngs, update_player, depth + 1, reach0, reach1, phase);
}

Ignore the chance probabilities when updating the regrets

if cur_player == update_player {
    for &a in &actions {
        let mut n = node.borrow_mut();
        n.regret_sum[a] += opp_reach * (move_evs[a] - strat_ev); // no chance_prob term
        n.total_move_prob[a] += my_reach * n.move_prob[a]
    }

    // save the updates
    ns.insert_node(is, node);
}

CFR vs CFRCS performance

These minimal changes allows CFRCS to converge faster than vanilla CFR. For example in a game of bluff with 2 dice for each player, CFRCS converges meaningfully faster than CFR.

The exploitability is based on the best response algorithm. More details can be found in Marc Lanctot's thesis.

Code

All the code for this post can be found on Github. But each implementation is repeated below for convenience.

Vanilla CFR

pub struct VanillaCFR {
    nodes_touched: usize,
}

impl Algorithm for VanillaCFR {
    fn run<T: GameState, N: NodeStore<CFRNode>>(
        &mut self,
        ns: &mut N,
        gs: &T,
        update_player: Player,
    ) {
        self.vcfr(ns, gs, update_player, 0, 1.0, 1.0, 1.0, CFRPhase::Phase1);
    }

    fn nodes_touched(&self) -> usize {
        return self.nodes_touched;
    }
}

impl VanillaCFR {
    fn vcfr<T: GameState, N: NodeStore<CFRNode>>(
        &mut self,
        ns: &mut N,
        gs: &T,
        update_player: Player,
        depth: usize,
        reach0: f64,
        reach1: f64,
        chance_reach: f64,
        mut phase: CFRPhase,
    ) -> f64 {
        self.nodes_touched += 1;

        let cur_player = gs.cur_player();
        if gs.is_terminal() {
            return gs.evaluate()[update_player].into();
        }

        if gs.is_chance_node() {
            let mut ev = 0.0;

            let actions = &gs.legal_actions();
            for &a in actions {
                let mut ngs = gs.clone();
                ngs.apply_action(a);

                let chance_prob = 1.0 / actions.len() as f64;
                let new_chance_reach = chance_prob * chance_reach;
                ev += chance_prob
                    * self.vcfr(
                        ns,
                        &ngs,
                        update_player,
                        depth + 1,
                        reach0,
                        reach1,
                        new_chance_reach,
                        phase,
                    );
            }
            return ev;
        }

        // check for cuts  (pruning optimization from Section 2.2.2) of Marc's thesis
        let team = match cur_player {
            0 | 2 => 0,
            1 | 3 => 1,
            _ => panic!("invald player"),
        };
        let update_team = match update_player {
            0 | 2 => 0,
            1 | 3 => 1,
            _ => panic!("invald player"),
        };

        if phase == CFRPhase::Phase1
            && ((team == 0 && update_team == 0 && reach1 <= 0.0)
                || (team == 1 && update_team == 1 && reach0 <= 0.0))
        {
            phase = CFRPhase::Phase2;
        }

        if phase == CFRPhase::Phase2
            && ((team == 0 && update_team == 0 && reach0 <= 0.0)
                || (team == 1 && update_team == 1 && reach1 <= 0.0))
        {
            trace!("pruning cfr tree");
            return 0.0;
        }

        let is = gs.istate_key(gs.cur_player());
        let mut strat_ev = 0.0;

        let actions = gs.legal_actions();

        let mut move_evs = ActionVec::new(&actions);

        let node = ns
            .get(&is)
            .unwrap_or(Rc::new(RefCell::new(CFRNode::new(gs.legal_actions()))));
        let param = match cur_player {
            0 | 2 => reach0,
            1 | 3 => reach1,
            _ => panic!("invalid player"),
        };

        let move_probs = node.borrow_mut().get_move_prob(param);
        // // iterate over the actions
        for &a in &actions {
            let newreach0 = match gs.cur_player() {
                0 | 2 => reach0 * move_probs[a],
                1 | 3 => reach0,
                _ => panic!("invalid player"),
            };

            let newreach1 = match gs.cur_player() {
                0 | 2 => reach1,
                1 | 3 => reach1 * move_probs[a],
                _ => panic!("invalid player"),
            };

            let mut ngs = gs.clone();
            ngs.apply_action(a);
            let payoff = self.vcfr(
                ns,
                &ngs,
                update_player,
                depth + 1,
                newreach0,
                newreach1,
                chance_reach,
                phase,
            );
            move_evs[a] = payoff;
            strat_ev += move_probs[a] * payoff;
        }

        // post-traversals: update the infoset
        let (my_reach, opp_reach) = match gs.cur_player() {
            0 | 2 => (reach0, reach1),
            1 | 3 => (reach1, reach0),
            _ => panic!("invalid player"),
        };
        if phase == CFRPhase::Phase1 && cur_player == update_player {
            for &a in &actions {
                let mut n = node.borrow_mut();
                n.regret_sum[a] += (chance_reach * opp_reach) * (move_evs[a] - strat_ev);
            }
        }

        if phase == CFRPhase::Phase2 && cur_player == update_player {
            for a in actions {
                let mut n = node.borrow_mut();
                n.total_move_prob[a] += my_reach * n.move_prob[a]
            }
        }

        if cur_player == update_player {
            // Todo: move memory to be managed by nodestore -- a get call always returns a node, initialized by the store
            ns.insert_node(is, node);
        }

        return strat_ev;
    }

    pub fn new() -> Self {
        Self { nodes_touched: 0 }
    }
}

CFRCS

pub struct CFRCS {
    nodes_touched: usize,
    rng: StdRng,
}

impl Algorithm for CFRCS {
    fn run<T: GameState, N: NodeStore<CFRNode>>(
        &mut self,
        ns: &mut N,
        gs: &T,
        update_player: Player,
    ) {
        self.cfrcs(ns, gs, update_player, 0, 1.0, 1.0, CFRPhase::Phase1);
    }

    fn nodes_touched(&self) -> usize {
        return self.nodes_touched;
    }
}

impl CFRCS {
    pub fn new(seed: u64) -> Self {
        Self {
            nodes_touched: 0,
            rng: SeedableRng::seed_from_u64(seed),
        }
    }

    fn cfrcs<T: GameState, N: NodeStore<CFRNode>>(
        &mut self,
        ns: &mut N,
        gs: &T,
        update_player: Player,
        depth: usize,
        reach0: f64,
        reach1: f64,
        mut phase: CFRPhase,
    ) -> f64 {
        if self.nodes_touched % 1000000 == 0 {
            debug!("cfr touched {} nodes", self.nodes_touched);
        }
        self.nodes_touched += 1;

        if gs.is_terminal() {
            return gs.evaluate()[update_player].into();
        }

        let cur_player = gs.cur_player();
        let actions = gs.legal_actions();
        if actions.len() == 1 {
            // avoid processing nodes with no choices
            let mut ngs = gs.clone();
            ngs.apply_action(actions[0]);
            return self.cfrcs(ns, &ngs, update_player, depth + 1, reach0, reach1, phase);
        }

        if gs.is_chance_node() {
            let a = *actions.choose(&mut self.rng).unwrap();
            let mut ngs = gs.clone();
            ngs.apply_action(a);
            return self.cfrcs(ns, &ngs, update_player, depth + 1, reach0, reach1, phase);
        }

        // check for cuts  (pruning optimization from Section 2.2.2) of Marc's thesis
        let team = match cur_player {
            0 | 2 => 0,
            1 | 3 => 1,
            _ => panic!("invald player"),
        };
        let update_team = match update_player {
            0 | 2 => 0,
            1 | 3 => 1,
            _ => panic!("invald player"),
        };

        if phase == CFRPhase::Phase1
            && ((team == 0 && update_team == 0 && reach1 <= 0.0)
                || (team == 1 && update_team == 1 && reach0 <= 0.0))
        {
            phase = CFRPhase::Phase2;
        }

        if phase == CFRPhase::Phase2
            && ((team == 0 && update_team == 0 && reach0 <= 0.0)
                || (team == 1 && update_team == 1 && reach1 <= 0.0))
        {
            trace!("pruning cfr tree");
            return 0.0;
        }

        let is = gs.istate_key(gs.cur_player());

        // log the call
        trace!("cfr processing:\t{}", is.to_string());
        trace!("node:\t{}", gs);
        let mut strat_ev = 0.0;

        let mut move_evs = ActionVec::new(&actions);

        let node = ns
            .get(&is)
            .unwrap_or(Rc::new(RefCell::new(CFRNode::new(gs.legal_actions()))));
        let param = match cur_player {
            0 | 2 => reach0,
            1 | 3 => reach1,
            _ => panic!("invalid player"),
        };

        // // iterate over the actions
        let move_probs = node.borrow_mut().get_move_prob(param);
        for &a in &actions {
            let newreach0 = match gs.cur_player() {
                0 | 2 => reach0 * move_probs[a],
                1 | 3 => reach0,
                _ => panic!("invalid player"),
            };

            let newreach1 = match gs.cur_player() {
                0 | 2 => reach1,
                1 | 3 => reach1 * move_probs[a],
                _ => panic!("invalid player"),
            };

            let mut ngs = gs.clone();
            ngs.apply_action(a);
            let payoff = self.cfrcs(
                ns,
                &ngs,
                update_player,
                depth + 1,
                newreach0,
                newreach1,
                phase,
            );
            move_evs[a] = payoff;
            strat_ev += move_probs[a] * payoff;
        }

        let (my_reach, opp_reach) = match gs.cur_player() {
            0 | 2 => (reach0, reach1),
            1 | 3 => (reach1, reach0),
            _ => panic!("invalid player"),
        };

        // // post-traversals: update the infoset
        if phase == CFRPhase::Phase1 && cur_player == update_player {
            for &a in &actions {
                node.borrow_mut().regret_sum[a] += opp_reach * (move_evs[a] - strat_ev);
            }
        }

        if phase == CFRPhase::Phase2 && cur_player == update_player {
            for a in actions {
                let mut n = node.borrow_mut();
                n.total_move_prob[a] += my_reach * n.move_prob[a];
            }
        }

        // todo: figure out if need the explicit updates
        if cur_player == update_player {
            ns.insert_node(is, node);
        }

        return strat_ev;
    }
}

Overthinking Phase 10

Tue, 30 Nov 2021 00:00:00 +0000

Context

Phase 10 is a rummy-like card game where the player needs to accomplish various goals (or phases) to progress. For example, the player may need two sets of three cards or a seven card straight. In each turn, the player must decide between 1) taking the faceup card discarded by the player to their right or 2) drawing an unknown card from the deck.

Wikipedia gives a good overview of the rules.

Problem

While playing with family over Thanksgiving, I was still determining the best approach to accomplishing phases nine and ten of the game. These phases require the player to get one set of five and one set of two (or three). Specifically, I was trying to decide between:

Greedily taking cards from the discard pile to start pairs, hurting my near-term chances of getting to the set of five, but giving me more options in the future.
Only taking cards from the discard pile when they helped me get to sets of three or four and instead taking my chances with the draw pile.

As you can imagine, these decisions are incredibly high stakes, especially when playing with family.

What is the best policy for determining when to take the face-up card or to draw a new card? Where best lets us complete Phase 10 (one set of five and one set of three) in the smallest number of turns.

Approach

I built a simulation for Phase 10 in Rust (code on Github). I chose Rust for fun.

The main area of interest is the 'take-policy' -- the set of rules that determine when to take the face-up discarded card or to draw an unknown card[0].

I run 100k simulations for each take-policy and compare the median and average turns to win. First, we're modeling the discarded cards as random from the deck.

See below for an overview of the policies.

Baseline policy

Rules that apply no matter what we're doing:

Always take wild cards.
Always draw if a skip card is on the discard pile. Skip cards can't contribute to a set.

Greedy pairs (1 from above)

If taking the discard card gives us a pair or better, take it.

fn greedy_pairs(hand: &Vec<Card>, candidate_card: Card) -> Action {
match hand.contains(&candidate_card) {
true => Action::Take,
_ => Action::Draw,
}
}

Greedy 5 of a kind after N (2 from above)

Play the same as greedy pairs until we have a set of N cards. After that, greedily try to complete the set of five, so only take from the discard pile if it moves toward completing the set. Otherwise, draw a card.

fn greedy_5_after_n(hand: &Vec<Card>, candidate_card: Card, target_n: i32) -> Action {
let counts = get_counts(&hand);
let (_, mcount) = counts[counts.len() - 1]; // end of list has highest count


if mcount < target_n {
return greedy_pairs(hand, candidate_card);
}


for (card, count) in counts {
match (card, count) {
// Check to ensure you don't already have 5 of a kind
(x, n) if x == candidate_card && n >= target_n && n < 5 => return Action::Take,
_ => continue,
};
}


return Action::Draw;
}

Hide intentions until N of a kind

Always draw a card unless picking up a card completes a set of N. The goal is to hide the cards you are aiming for from opponents as discarded cards come from the player to your right.

fn hide_until_n(hand: &Vec<Card>, candidate_card: Card, target_n: i32) -> Action {
let counts = get_counts(&hand);
let (_, mcount) = counts[counts.len() - 1]; // end of list has highest count


if mcount <= target_n {
return Action::Draw;
}


for (card, count) in counts {
match (card, count) {
// Check to ensure you don't already have 5 of a kind
(x, n) if x == candidate_card && n >= target_n && n < 5 => return Action::Take,
_ => continue,
};
}


return Action::Draw;
}

Initial results: ‘Greedy pairs’ wins

Policy	Turns to win (median)	Turns to win (average)
Greedy pairs	10	10.1
Greedy 5 of a kind after 4	10	10.3
Greedy 5 of a kind after 3	11	11.1
Hide until 4 of a kind	14	15.1
Hide until 3 of a kind	14	14.7

Greedy pairs seems to be the best approach. It has the lowest median and mean number of turns to win. But the main downside of this approach is that it informs other players about what cards you're going for. Right now, we model the discarded cards as random.

Do things change if we account for other player behavior? For example, if a player knows you're going for a set of threes, they may be less likely to discard a 3 for you to pick up.

Results with the antagonistic discard pile

To test this, we build an antagonistic discard pile: we never let the discard pile show a card we've taken in the past. This would model the person next to you playing perfectly (a tough sell after a Thanksgiving of eating and drinking). With this constraint:

Policy	Turns to win (median)	Turns to win (average)
Greedy pairs	11	11.7
Greedy 5 of a kind after 4	11	11.9
Greedy 5 of a kind after 3	11	12.3
Hide until 4 of a kind	14	15.1
Hide until 3 of a kind	14	14.7

As expected, there isn't much impact on the hide until n strategies.The whole point is to hide the cards we're trying to get.

The greedy pairs strategy is negatively impacted. But it's still the best strategy. And it's the simplest, if you have the discard card in your hand, take it.

I was way overthinking this.

Possible future ideas

Notes in case I want to revisit this:

Better understand why certain strategies outperform. For example, look at a graph of close-ness to goal over time. Does anything stand out between the strategies
Evaluate distributions of results, anything unexpected?
Improve performance of the simulation

[0] There is also a policy for discarding cards. For simplicity, this policy is the same for all take-policies: discard the card with the lowest number of sets in the hand.

Yak shaving I did to get set up

Sat, 23 May 2020 02:00:43 +0000

This post gives a summary of what I did to get my dev environment set up. Main goals:

Hold myself accountable to not spend too much time tuning the colors of my terminal
Document, in case I need to do it again
Better learn Vim to write this

Dev environment

I wanted a Linux environment for development. But I only have access to a machine running windows and ChromeOS. Dual-booting -- or something like cygwin -- is possible but it seemed like a hassle to setup. And would add another barrier to writing or working on side projects.

I ended up going with a Digital Ocean machine. It had a few added benefits for me:

easy backups I could turn on with a click
ability to access my dev environment where ever I was and a variety of different computers
seemed like an interesting project to try out -- I'd wanted a forcing function to get better with a terminal for a while

I'm sure any cloud provider would have worked, but I had a good experience using digital ocean in the past -- it's reasonably priced and reasonably simple.

Next, I setup tmux and putty. This ended up being a bigger waste of time than expected. I had foolishly gotten my Vim and terminal color scheme set up with out tmux. Running things through tmux immediately reverted me to the previous, ugly colors :(

I eventually realized I needed to start tmux with the -2 argument to enable 256 colors:

tmux -2 a -t base || tmux -2 new -s base

I was able to get Putty set up with minimal configuration.

Editor

For my editor, I went with Vim. I'd always heard great things. And the flexibility it provides seemed valuable to support both coding and blog writing in a single platform. I also appreciated the flexibility of moving around a .vimrc file to set up a new editor.

I used amix's .vimrc as a starting point. And removed most of the configuration that didn't make sense to me -- my goal was to end with something simple.

I also added the following plugins:

gruvbox theme
vim-fugitive
vim-go
nerd-tree

Similar to the .vimrc, I tried to keep my plugins simple.

Blog

I wanted an easy way to keep track of thoughts and notes. I'll mostly use this as a project log for myself. But hope to also have some more polished content eventually.

Used github pages and jekyll making the blog easy to set up and maintain. I registered a domain with Google Domain.

Future plans

A few things I hope to eventually figure out:

get ssl working with github pages
get ssl working for some sort of dev instance
...