The Crucible

每一則訊息都是一個節點，而不是一條線性的聊天紀錄。你可以從任何一點分岔，同時把同一個問題丟給多個模型，讓各自的回答並排成分支直接比較；也可以請另一個模型回頭審查一段對話，審查者會拿到完整的發言者資訊，知道每句話是誰說的。分歧夠大時，再挑出幾個不同的回答，讓模型整合成一份綜合結論。 Every message is a node, not a line in a chat log. You can branch off any point, fire the same prompt at several models at once and get their answers as parallel branches to compare directly, or ask a different model to critically review a conversation thread — with full speaker attribution, so it knows who said what. When the disagreement is real, you pick the divergent answers and have a model synthesize them into one cohesive brief.

實際跑一次是這樣的。同一個問題，兩個模型給出的機率相差很大： Here's what that looks like in a real run — same question, very different numbers:

真實節錄 · 2026-05-02Real excerpt · 2026-05-02

“Will AGI arrive by 2030?”

Gemini 3 Flash

85%

2030年前，AI能自主完成八成以上的知識工作AI autonomously doing 80%+ of knowledge work by 2030

“…the timeline to AGI moved from linear to exponential.”

GPT-5.2

15–30%

同樣以「幾乎不需人工監督」為標準；若允許有人把關，才是50–70%Same bar, minimal oversight; 50–70% only if humans stay in the loop

“85% reads like narrative confidence rather than an inference.”

↳ 這裡偵測分歧的技術，我打算之後用在Legwork下一階段的驗證層。 ↳ I'm eyeing the disagreement-detection here as something to fold into Legwork's next verification layer.

研究：對話被壓縮時，丟掉了什麼Research: what gets lost when a conversation is compacted

The Crucible現在也是一個小型實驗平台，用來回答一個問題：長對話超出context上限、被摘要壓縮之後，哪些細節會留下來，其他的又為什麼不見？做法是在合成的長對話裡埋進事實，讓app真正的摘要流程逐輪壓縮，再用一個oracle檢查每個事實還在不在context裡，分清楚「摘要把它丟了」和「模型沒用到它」。 The Crucible now doubles as a small testbed for one question: when a long conversation outgrows the context window and gets summarized, which details survive, and why are the rest lost? Facts are planted in long synthetic conversations, the app's real pipeline summarizes them turn by turn, and an oracle checks whether each fact is still in the context, which separates "the summary dropped it" from "the model didn't use it".

• 摘要只留下當下目標用得到的東西。換了目標之後，OpenAI和Anthropic的摘要只留住新目標需要的32個事實裡的4到13個。 Summaries keep what the current goal needs. After a change of plans, OpenAI and Anthropic summaries had kept only 4 to 13 of the 32 facts the new goal needed.
• 原因是寫摘要的當下不知道接下來要做什麼。事先寫明下一個目標，能留住30個；只讓主題名稱出現一樣多次、但不說它是下一步，只有6個。不知道未來的話，寫入時唯一的補救就是幾乎不壓縮。 The cause is not knowing what comes next. Announcing the next goal in advance kept 30 of 32; naming the subject just as often, without making it next, kept 6. Without that knowledge, the only fix at write time is to barely compress.
• 等問題出現再決定撈什麼，就找得回來。保留原始訊息、等問題出現後再挑相關的撈回，能找回28到32個；負責挑的模型用最便宜的，效果和最強的一樣。 Deciding at read time gets it back. Keeping the original messages and fetching the relevant ones once the question is known recovered 28 to 32 of 32, and the cheapest model chose what to fetch as well as the strongest.

接著比較哪個摘要模型最合適。在app實際的運作方式（開啟召回）下，最便宜的摘要模型就夠了：它的摘要加上召回，在細節密集的情境留住96個事實中的93個，和最好的96個在統計上相當；換目標之後留住64個中的62個，是所有候選裡最多的；每份摘要的成本約是最好那個的1/25。少了召回，它的摘要單獨只留得住42個，最完整的那個則有90個。也就是說，只要原始訊息撈得回來，摘要就不必是紀錄本身，當索引就好。勝出者是照跑之前就寫下的決策規則選出來的，app的預設摘要模型也跟著換了。 Next came which summarizer to use. With recall on, which is how the app actually runs, the cheapest summarizer was enough: its summaries plus recall kept 93 of 96 facts in the detail-heavy scenario, statistically equivalent to the best summarizer's 96, and 62 of 64 after a change of plans, the most of any candidate, at about 1/25 of the cost per summary. Without recall, its summaries alone kept only 42 of 96, against 90 for the most thorough one. Once the original messages can be fetched back, the summary no longer has to be the record, just an index. The winner was picked by a decision rule written down before the run, and the app's default summarizer changed to match.

這些是在合成對話、8,000 token門檻下的小規模試驗結果，不是通用的benchmark。完整的實驗筆記，包括每個數字和做錯的地方，都公開在repo裡。 These are pilot results on synthetic conversations at an 8,000-token threshold, not a general benchmark. The full lab notes, every number and every mistake included, are public in the repo.

結論也回到了app裡：現在預設會在需要時撈回被壓縮掉的原始訊息。我另外做了一個scaffold audit，每次模型更新，就重測每個context機制是不是還贏過不用它的基準。第一次跑就發現，同一個召回機制幫到了一個模型，卻讓另一個較弱的模型拿錯了主題的值。 The findings went back into the app: it now recalls the compacted originals by default when they're needed. I also built a scaffold audit that re-tests, whenever models change, whether each piece of context machinery still beats going without it. Its first run found the same recall mechanism helping one model and leading a weaker one to pick up the wrong subject's values.