原名 adaptive-iteration。Ordal 取自古英文 ordāl，也就是 ordeal 的字根：一場會給出判決的考驗。Formerly adaptive-iteration. The name comes from Old English ordāl, the root of “ordeal”: a trial that yields a judgment.
一個讓 A/B 實驗不自欺的 Python 框架：只在事先定好的檢查點下結論、用信賴區間判斷、缺資料不當成零。提出假設的可以是 LLM，判定輸贏的永遠是資料。已發佈在 PyPI。 A Python framework for running A/B experiments without fooling yourself: verdicts only at fixed checkpoints, judged by confidence intervals, missing data never counted as zero. An LLM can propose the hypotheses; the data decides who wins. Published on PyPI.
$ python examples/quickstart.py
week 1: insufficient n=20/20 interval [-2.7, 19.6] overlaps ±3; covariate adjustment skipped (missing covariates)
week 2: b_better n=40/40 B better: effect +9.33, interval [1.99, 16.7] above +0; covariate adjustment skipped (missing covariates)
cta_style is proposed nextquickstart 範例 · 模擬資料quickstart example · simulated data上面是框架內附範例的實際輸出。第一週兩組已經看得出差距，但信賴區間還太寬，所以判定是「資料不足」，不下結論；第二週資料夠了，才判定 B 較好。結論只在固定的每週檢查點產生，重複查看造成的誤判率有控制；缺漏或還沒成熟的資料會被排除並記錄，不會算成零；「沒有差異」和「還不知道」是兩種不同的結果。 Above is the real output of the example bundled with the framework. In week one the two arms already look different, but the confidence interval is still too wide, so the verdict is "insufficient" rather than a winner; by week two there's enough data to call B better. Verdicts only happen at fixed weekly checkpoints, with the false-positive rate controlled across repeated looks. Missing or immature data is excluded and counted, never treated as zero, and "no difference" and "don't know yet" are different outcomes.
每一筆觀測和每一個判定都寫進只能追加的 JSONL 帳本，任何結論事後都能重算；變數清單會擋下「同一個概念換個名字再測一次」。假設從哪裡來由使用者注入：可以是 LLM、參數網格、規則引擎或人，框架核心不呼叫任何模型。同樣的操作也提供 MCP server 和 JSON CLI 給 agent 使用，並附帶自動迴圈繞不過去的限制：不能根據中途數字行動，也不能在實驗進行中移動標準。 Every observation and every verdict goes into an append-only JSONL ledger, so any conclusion can be recomputed later, and a variable registry stops the same idea from being tested again under a new name. Where hypotheses come from is injected: a language model, a parameter grid, a rules engine, or a person — the core calls no model. The same operations are exposed as an MCP server and a JSON CLI for agents, with guardrails an automated loop can't talk its way around: no acting on interim numbers, no moving the goalposts mid-experiment.
它負責判定一條私人流水線的實驗，那條流水線每天自動產出並上傳短影音。在此之前，流水線用一套簡單的規則自己判定實驗：兩組平均差得夠多就判贏，每組只要幾支影片，也不看信賴區間。拿真實資料重播，這套規則在其實沒有差異的情況下，超過一半的時候會判出贏家；新的預設規則把這個比例壓在 5% 以下。流水線自己定期跑的研究稽核也陸續找出具體問題：缺資料被當成零、平均值被重複觀看灌高、有些被測試的變數流水線其實從來沒有真正改變過。2026 年 9 月 25 日起，判定改由這個框架負責。 It judges the experiments of a private pipeline that produces and uploads short videos every day. Before that, the pipeline judged its own experiments with a simple rule: a winner whenever the two arms' averages differed enough, with only a few videos per arm and no confidence interval. Replayed on real data, that rule declared a winner more than half the time when there was no real difference; the new default keeps that below 5%. The pipeline's own scheduled research audits also turned up specific problems: missing data counted as zero, averages inflated by rewatches, and some tested variables that the pipeline never actually changed. Since 25 September 2026, this framework does the judging.
舊的實驗大多每組只有幾支影片、逐支資料也不完整，沒辦法逐一重判。只有一個舊實驗的資料夠，用新規則重判的結果是「沒有可偵測的差異」，而舊規則會把它判成有贏家。 Most of the old experiments had only a few videos per arm and incomplete per-video data, so they can't be re-judged one by one. Only one had enough data; under the new rule it comes out "no detectable difference", where the old rule would have named a winner.