Variance
用RAG回答SEC財報問題,但真正的重點是另一半:每個帶數字的答案都拿去跟SEC自己的XBRL結構化數據核對,不是自己說了算。目前只做蘋果一家公司,模型全部跑在本機,eval結果老實秀信賴區間,不會只丟一個準確率數字就交差。 Answers questions about SEC filings with RAG — but the real point is the other half: every numeric claim gets checked against SEC's own structured XBRL data instead of just being trusted. Scoped to one company (Apple) for now, runs entirely on local models, and the eval results show honest confidence intervals instead of a single accuracy number pretending to settle it.
背後的假設是「lazy prices」:財報文字(風險因子、MD&A)跟它逐季的變化,帶有市場還沒完全定價進去的資訊,這是有文獻支持的市場無效率。這個工具做兩半——用混合檢索(向量+BM25,倒數排名融合)加本機LLM回答問題並附上原文出處,以及更關鍵的評估半邊:自動從XBRL事實產生問題,跑過RAG流程,拿答案跟XBRL真實數字比對。 The underlying thesis is "lazy prices": filing text (risk factors, MD&A) and how it changes quarter to quarter carries information the market hasn't fully priced in yet — a documented market inefficiency. The tool has two halves: hybrid retrieval (dense + BM25, fused via reciprocal rank fusion) plus a local LLM that answers questions with citations back to the filing text, and — the half that actually matters — an evaluator that auto-generates questions from XBRL facts, runs them through the RAG pipeline, and checks the answers against XBRL's real numbers.
最新一次真實跑分(AAPL,qwen3.6:35b-mlx,55題):單一事實題68.6%準確率(95% CI 52.0–81.4%),複合題(先拆解再分頭檢索)66.7%(95% CI 41.7–84.8%)——兩個信賴區間有重疊,樣本數也還小,所以不能說複合路由比較差。路由本身把每題都拆對了,拆解準確率100%。 Latest real run (AAPL, qwen3.6:35b-mlx, 55 questions): 68.6% accuracy on single-fact questions (95% CI 52.0–81.4%), 66.7% on compound questions that get decomposed and retrieved separately (95% CI 41.7–84.8%). The two intervals overlap and the sample is still small, so this isn't evidence the router makes things worse — and the router itself split every compound question into exactly the right parts, a 100% decomposition match rate.