A self-calibrating wrapper around createModelAwareTokenEstimator
(v3.8).
Character-ratio estimation is ±15% at best and drifts with content
mix (code vs prose vs CJK) and provider tokenizer changes. This
estimator closes the loop: feed it actual usage from completed calls
via observe(), and it maintains a per-model EWMA of
actual input / estimated input, multiplying future input
estimates by that factor (clamped, and only after minSamples
observations). Output reservations are never corrected —
max_tokens / outputCap is a ceiling, not an estimate.
The correction applies to the whole input estimate, including
any opaqueBlockTokens surcharge and extraInputTokens — the
observed ratio necessarily includes those components too, so scaling
the total is the self-consistent choice (the fixed point is
"corrected estimate ≈ actual"). If your extraInputTokens values
are exact, expect the factor to settle slightly differently than a
pure-prose deployment; the clamps bound the damage either way.
observe() always measures against the uncorrected base estimate,
so feedback does not compound: an already-corrected estimator does
not drag its own factor back toward 1.
Not distributed and not persisted — calibration lives in this
instance. Share one instance per bulkhead (create it, pass
.estimator into tokenBudget, and subscribe .observe to the
same bulkhead's release events).
A self-calibrating wrapper around
createModelAwareTokenEstimator(v3.8).Character-ratio estimation is ±15% at best and drifts with content mix (code vs prose vs CJK) and provider tokenizer changes. This estimator closes the loop: feed it actual usage from completed calls via
observe(), and it maintains a per-model EWMA ofactual input / estimated input, multiplying future input estimates by that factor (clamped, and only afterminSamplesobservations). Output reservations are never corrected —max_tokens/outputCapis a ceiling, not an estimate.The correction applies to the whole input estimate, including any
opaqueBlockTokenssurcharge andextraInputTokens— the observed ratio necessarily includes those components too, so scaling the total is the self-consistent choice (the fixed point is "corrected estimate ≈ actual"). If yourextraInputTokensvalues are exact, expect the factor to settle slightly differently than a pure-prose deployment; the clamps bound the damage either way.observe()always measures against the uncorrected base estimate, so feedback does not compound: an already-corrected estimator does not drag its own factor back toward 1.Not distributed and not persisted — calibration lives in this instance. Share one instance per bulkhead (create it, pass
.estimatorintotokenBudget, and subscribe.observeto the same bulkhead'sreleaseevents).