Ratatoskr

這份路線圖不是事先規劃好的，而是一連串提問推出來的。一開始只想知道這台Mac mini最適合跑哪個本地模型，一路追到記憶體頻寬的瓶頸、HBM是什麼，才冒出真正的問題：身為工程師，我在這件事上有沒有著力點？第一版路線圖（量化 → 投機解碼 → kernel）建立在「MLX的投機解碼生態還是藍海」這個判斷上，後來查證發現Ollama和LM Studio都已經內建投機解碼，原本以為的空白根本不存在。這次修正留下一條篩選原則：如果真的這麼簡單，一定早就有人做了。 This didn't start as a plan — it got forced out by a chain of questions. It began with just wanting to know which local model would run fastest on this Mac mini, which led down into the memory-bandwidth wall and what HBM even is, until the real question surfaced: as an engineer, is there actually a place to push here? The first version of the roadmap (quantization → speculative decoding → kernel work) rested on a bet that MLX's speculative-decoding ecosystem was underexplored. That bet turned out wrong — Ollama and LM Studio already ship it built in. The correction left behind a sharper filter: if it were really this easy, someone would have already done it.

判斷一個優化能否同時讓訓練和推論受益，要看它鎖定的是什麼。鎖定共用的計算核心、並保證輸出不變的優化（FlashAttention是典型例子），通用性很好；利用只有單邊才有的不對稱性（例如投機解碼利用decode的序列特性、KV-cache分頁利用推論才有的跨請求快取），就不會通用，但這本來就是這類優化的價值來源，並非缺陷。分工原則也很清楚：第一次做的事，不管看起來多機械，都自己動手做懂；做過幾輪、能說清楚每一步在做什麼之後，才把重複性的規模化工作交給agent。 Whether an optimization generalizes across training and inference comes down to what it locks onto: locking onto a shared compute kernel while guaranteeing identical output (FlashAttention is the textbook case) generalizes well; exploiting an asymmetry that only exists on one side (speculative decoding exploits decode being sequential, paged KV-cache exploits inference-only cross-request caching) doesn't generalize — and that's not a flaw, it's where those optimizations get their value from. The division of labor is just as deliberate: the first time doing something, no matter how mechanical it looks, gets done by hand; only after a few rounds of being able to explain exactly what each step does does the repetitive scaling-up get handed to an agent.