All notable changes to this project will be documented in this file.
The format loosely follows Keep a Changelog and adheres to Semantic Versioning.
Resource attribution for admitted work. Successful acquire results,
tokens, run contexts, and admission/release events now expose immutable
resources with exact borrowed-concurrency and borrowed-token attribution.
Borrowed local-slot abandonment. LLMToken and admitted run contexts add
idempotent abandonBorrowedConcurrency(). It returns only a concurrency slot
that came from shared class capacity; token holds remain charged until final
release. New global and per-class counters plus the
borrowedConcurrencyAbandon event expose the split lifecycle. The event
carries a cause of manual or deadline, and
llm.borrowedConcurrencyAbandonedByCause counts each separately, so
deliberate shedding is distinguishable from a lease configured too tight.
Wall-clock borrowed-slot deadline. run() accepts
borrowedConcurrencyDeadlineMs. The post-admission timer applies only to a
borrowed local slot. Expiry returns that slot, aborts the callback signal,
and rejects the caller with LLMBorrowedConcurrencyDeadlineError while the
underlying callback retains responsibility for token settlement.
drain() now waits for final LLM settlement even after local concurrency was
abandoned. Timed drain results report outstanding token-holding admissions
instead of incorrectly declaring the bulkhead drained when only the local
slot has returned.Existing methods and event names remain available. Exact object-equality consumers should account for new resource fields and counters. Callers that do not use borrowed-slot abandonment retain their previous lifecycle.
After abandonBorrowedConcurrency(), drain() without timeoutMs waits for
the matching token settlement even though the local slot is already free.
Manual acquire() callers must eventually call release() or an unbounded
drain remains pending; bounded drain reports the admission as outstanding.
This split lifecycle is new in 3.17 and prevents shutdown from reporting
false completion.
Listener and LLMBorrowedConcurrencyAbandonCause are new entry-point type
exports. Listener was previously reachable only through on() without
being nameable by consumers.
The runtime dependency remains async-bulkhead-ts@^1.0.1.
Admission-decision timing on lifecycle events. Instrumented admit and
admission-decision reject events now carry additive optional integer
nanosecond fields: decisionDurationNs and queueWaitNs. Timing uses
Node's monotonic process.hrtime.bigint() clock.
Decision time is separated from queue wait. queueWaitNs measures the
complete awaited bulkhead.acquire() span. decisionDurationNs excludes
that await and measures only synchronous admission work, so queue contention
cannot be misreported as local decision cost. Precheck rejects report
queueWaitNs === 0 exactly.
Admission timing coverage tests and clock-overhead microbenchmark. Tests
cover precheck rejection, immediate and queued admission, acquire timeout,
queue-limit rejection, postcheck rejection after a limits race, and both
observe-mode bypass shapes. The suite also records the cost of the four
process.hrtime.bigint() reads required on paths that cross the acquire
await; it is diagnostic rather than a timing-sensitive CI gate.
Observe-mode calls that are actually bypassed do not expose admission timing.
The advisory bypass path never enters admission. When a race causes
_acquire() to reject and observe mode converts that result to a bypass, the
existing reject event remains for backward compatibility but its timing
fields are absent.
Deduplication-follower rejections remain untimed because they are waits on an existing admission rather than new admission decisions.
admit / reject events.Documentation and package-metadata patch. No runtime code, public API, or dependency behavior changed.
rate-limiting npm keyword. async-bulkhead-llm provides
concurrency and in-flight token admission control; it does not implement a
conventional request/token rate limiter such as a fixed window, sliding
window, or token bucket. Removing the keyword avoids implying functionality
the library intentionally does not provide.Updated the security support matrix. SECURITY.md now identifies 3.15.3
as the only supported release, consistent with the policy that only the
latest release receives security fixes.
Corrected stale cost-ceiling language in the security policy. The token under-reservation threat now describes exceeding the intended in-flight token commitment bound rather than undermining a cumulative cost ceiling. This matches the token-budget semantics documented in 3.15.2.
Documentation release, with one development-only dependency update. No runtime code changed and the public API is identical to 3.15.1; no upgrade action is required.
README rewritten for first-time readers. The README is now a short
landing page (what the library does, install, quick start, token limits,
handling rejections, shutdown) instead of a 1,850-line reference. The
thirteen stacked "What's New in vX" sections have been removed from it;
per-release detail lives in this changelog, and the topical content moved
into guides/.
Long-form documentation moved to guides/. Eleven guides now carry the
material previously inlined in the README: how admission works, token budget
and estimation, streaming and reconciliation, admission classes, runtime
limits and reconfiguration, observe mode, deduplication, gateway
integration, stats and events, comparison, and migration. Content was moved
rather than rewritten.
Hand-maintained API reference retired. The ## API Reference section is
replaced by the generated TypeDoc reference, which is derived from the
source types and therefore cannot drift from the code.
TypeDoc documentation site. npm run docs generates a static site into
docs/ from src/index.ts, with the README as the landing page and the
guides/ files and this changelog rendered as site pages. Added
docs:watch for local iteration and docs:coverage to report symbols
missing TSDoc comments.
GitHub Pages publishing. .github/workflows/docs.yml builds the site on
every pull request and deploys it from main. Publishing requires the
repository Pages source to be set to "GitHub Actions". Generated output is
git-ignored and built in CI rather than committed.
npm run lint no longer fails after a docs build. eslint . linted
TypeDoc's generated browser assets under docs/, producing 367 errors on
any working tree where documentation had been generated. docs/** and
coverage/** are now ignored by ESLint.
Removed the cost-ceiling claim. Earlier documentation described
tokenBudget as enforcing a "cost ceiling" that stops requests "before you
pay for them." That was wrong. The budget is a ceiling on tokens reserved
concurrently: admission compares inFlightTokens + reserved against the
budget, and the entire reservation is released when the request completes.
It bounds in-flight commitment, not cumulative spend, and a given budget
places no limit on total tokens consumed over time. The documentation now
states this explicitly, consistent with the library's long-standing
no-cost-accounting non-goal. No behavior changed — only the description of
it.
Broken documentation links. The README's ./LICENSE and ./SECURITY.md
links resolved correctly on GitHub but not in the generated site; both now
use absolute repository URLs so they work from GitHub, npm, and the
documentation site.
nanoid (development-only). npm audit
reported this high-severity advisory -- custom generators can loop
indefinitely when size is zero -- against every nanoid before 3.3.18. The
lockfile now resolves 3.3.18, which carries the fix. nanoid reaches this
repository only as a development-time transitive dependency (vitest ->
vite -> postcss -> nanoid); it has never been a runtime dependency and
is not part of the published tarball, so consumers of async-bulkhead-llm
were never exposed. npm audit now reports no known vulnerabilities at any
severity.nanoid
bump is confined to development-only tooling. The published tarball
contents are unchanged apart from the rewritten README.md. guides/ and
the documentation tooling are development-only and are not published.README.md, LICENSE, and CHANGELOG.md so the npm registry package page
and downloaded package consistently contain the release documentation.Protected admission-class floors. Admission classes may reserve strict
local concurrency with protectedConcurrent and strict in-flight token
capacity with protectedInFlightTokens. The sum of configured floors must
fit inside the corresponding global limit, and a class floor may not exceed
its hard class ceiling.
Shared-capacity borrowing. Capacity remaining after all protected floors forms a shared pool. A class consumes its floor first and may then borrow from the shared remainder, subject to its class ceilings, global limits, and the priority-adjusted token budget.
Protection-aware rejection detail. Requests rejected to preserve another
class's floor report constraint: "admission_class_protection" and include
shared concurrency/token capacity plus protected and borrowed class usage.
Borrowing telemetry. stats().admissionClasses.shared exposes the shared
remainder. Per-class stats expose current protected and borrowed usage,
cumulative admissions using shared concurrency, and cumulative reservation
tokens placed in shared capacity.
Bounded admission classes. admissionClasses configures a fixed table of
policy-class IDs with optional hard maxConcurrent and
maxInFlightTokens ceilings. Calls select a class with admissionClass or
fall back to defaultClass; unknown IDs throw instead of creating
unbounded runtime state.
Hierarchical capacity decisions. Successful admission must fit both the enclosing global limits and the selected class limits. Class ceilings are fail-fast, use shrink-by-attrition under reconfiguration, and remain atomic with global concurrency and token accounting.
Class-aware progressive accounting. Reservations, progressive usage refunds, output overruns, and final release settlement update global and class token holds together.
Class-aware telemetry. Successful results, tokens, run contexts, usage
reports, lifecycle events, capacity details, and observe-mode bypasses carry
admissionClass when configured. Capacity details identify the binding
constraint as global or admission_class, and
stats().admissionClasses reports bounded per-class counters.
Class-safe deduplication. Admission class is an automatic deduplication partition when classes are configured, preventing one class from inheriting another class's admission decision or queue position. Class IDs are validated before the dedup follower fast path, and dedup/rejection events retain class attribution.
Atomic class-limit updates. LLMAdmissionLimits.admissionClasses updates
every construction-time class in the same versioned snapshot as global
concurrency, queue, token budget, and priority reserve. Runtime updates may
change numeric ceilings but cannot add or remove class keys.
admissionClasses is omitted.reportUsage() accepts an
optional remainingOutputTokens plus safetyMarginTokens declaration.
Gateways can release processed input and already-generated output capacity
during a live stream while retaining explicit future-output headroom.createLLMBulkhead() now accepts
maxConcurrent: 0, allowing gateways to start with no admission capacity and
remain closed until an external control plane installs a higher-revision
snapshot. Runtime zero-capacity behavior is unchanged: new work is rejected
with concurrency_limit, while later expansion begins admitting immediately.brace-expansion 5.0.8, resolving the reported high-severity audit
finding. This dependency is development-only; the runtime dependency graph is
unchanged.Published the finalized v3.11 contents under a new immutable package version after the v3.11.0 registry tarball produced an integrity mismatch. Consumers should install v3.11.1 rather than retrying the conflicting v3.11.0 artifact.
Updated the strict admit-capacity snapshot test to include the additive
limitRevision: 0 field introduced in v3.11.
Admission-linearized limit revisions. Successful acquire() results,
LLMToken, LLMRunContext, UsageReport, and admitted/bypassed lifecycle
events now carry an immutable limitRevision captured after both the
concurrency slot and token reservation are held.
Revisioned rejection snapshots. LLMRejectDetail.limitRevision and the
reject event identify the exact limit snapshot used for the capacity
decision, allowing gateways to report authoritative revisions without a
later limits() lookup.
admit listener can apply a newer limit snapshot before the
run() callback starts. The callback, usage events, and release event now
retain the original admission revision instead of being relabeled with the
newer configuration.Atomic, versioned admission-limit reconfiguration.
bulkhead.applyLimits(snapshot) replaces maxConcurrent, maxQueue,
tokenBudget.budget, and tokenBudget.highPriorityReserve as one complete
snapshot. Updates require a strictly increasing non-negative safe-integer
revision; equal or lower revisions return
{ applied: false, reason: "stale_revision" } without mutation.
Runtime concurrency and queue changes. Lower ceilings use
shrink-by-attrition: in-flight work and already accepted waiters are not
cancelled. Raising concurrency pumps accepted waiters immediately. A
runtime maxConcurrent: 0 acts as a fail-fast kill switch for new work.
Limit inspection and telemetry. bulkhead.limits() and
stats().limits expose the currently applied frozen snapshot.
initialRevision seeds the constructor state, and the new reconfigure
event contains the previous and current snapshots.
setBudget(tokens) remains backward compatible but now delegates to a
complete local reconfiguration and advances the revision by one. Distributed
control-plane integrations should use applyLimits() exclusively so one
authority owns the revision stream.
The internal concurrency gate now supports dynamic maxConcurrent and
maxQueue while preserving the public stats and rejection semantics used by
the LLM bulkhead.
First-class observe mode for run(). Pass { mode: "observe" } to
execute work without holding concurrency or token capacity when admission
would reject for budget_limit, concurrency_limit, queue_limit, or
timeout. The default remains { mode: "enforce" }. shadowReasons can
narrow the bypassable capacity reasons; shutdown, caller cancellation, and
unsafe deduplication fan-out remain hard failures and cannot be shadowed.
Admitted-versus-bypassed run context. LLMRunContext.admission is
"admitted" or "bypassed". Bypassed executions receive a stable
shadow-... identifier, the exact evaluated reservation, and optional
bypassReason / bypassDetail. reportUsage() remains available and
returns normal UsageReport snapshots without altering capacity accounting.
Observe telemetry. stats().observe reports bypass counts, race
bypasses, reasons, and final usage totals. New bypass, bypassUsage, and
bypassRelease events expose the same lifecycle without misrepresenting
bypassed work as admitted or released capacity.
raceBypassed. The
same resolved reservation is reused across both checks.Bounded drain: drain({ timeoutMs }). The no-argument form keeps its
Promise<void> contract unchanged. With a deadline, the promise always
resolves (never rejects) with an LLMDrainResult —
{ drained: true, inFlight: 0, pending: 0 } when everything completed in
time, or { drained: false, inFlight, pending } with the outstanding
counts at the moment the deadline elapsed. The deadline never cancels or
interrupts in-flight work and leaves accounting untouched: work that
finishes later still releases normally. Intended for shutdown paths that
must log what they are abandoning and proceed, rather than parking
forever behind one stuck upstream stream. timeoutMs must be a
non-negative integer; 0 is an immediate "is it drained right now?"
snapshot.
wouldAdmit(request, { detail: true }). Opt in to receive the same
LLMRejectDetail capacity snapshot that real rejections carry — on
every outcome, including admit: true, where it describes the capacity
the request would be admitted against (requested is the reservation
this request needs). Routing layers choosing between pools want the
numbers, not just the boolean. Omitted by default, so the result shape
and cost are unchanged for existing callers (results still deep-equal
{ admit: true } etc.).
estimate() results round-trip as the reservation override. The
per-call override now accepts LLMReservationOverride
(TokenEstimate & { reserved?: number }), so the frozen object returned
by bulkhead.estimate() can be passed back verbatim — no need to strip
reserved first. When reserved is present it is validated as a
consistency check: it must equal input + maxOutput, otherwise
admission throws. This catches hand-built overrides whose cached
reserved drifted from edited parts. Plain { input, maxOutput }
overrides (the 3.7 shape) are unchanged.
createAdaptiveTokenEstimator() — self-calibrating estimation. A
wrapper around createModelAwareTokenEstimator that closes the feedback
loop between estimates and reality. Feed it actual usage from completed
calls via observe(request, usage) (typically wired to the bulkhead's
release event); it maintains a per-model EWMA of
actual input / estimated input and multiplies future input
estimates by that factor — clamped to
[minCorrection, maxCorrection] (default [0.5, 2]) and applied only
after minSamples observations (default 5). Output reservations are
never corrected: max_tokens / outputCap is a ceiling, not an
estimate. Observations always measure against the uncorrected base
estimate, so the loop does not compound through its own corrections.
Tracked models are bounded (maxModels, default 64, oldest-inserted
evicted); corrections() exposes the calibration state for stats
endpoints and reset(model?) clears it. Calibration is in-memory and
per-instance — share one instance per bulkhead.
wouldAdmit() validates before the shutdown check. The request (and
any reservation override) is now validated before the closed fast
path, matching acquire()'s ordering — an invalid request (e.g. a
negative max_tokens) now throws even when the bulkhead is closed,
instead of returning { admit: false, reason: "shutdown" }. Valid
requests against a closed bulkhead behave exactly as before.src/index.ts (~2,400 lines) is now a barrel over types.ts,
errors.ts, profiles.ts, validation.ts, estimators.ts,
adaptive.ts, dedup.ts, and bulkhead.ts. No public API change:
the package entry point re-exports the identical surface (verified
against the pre-split build), and the exports map still exposes
only the entry point — deep imports of the internal modules remain
unsupported. dist/ now contains one file per module for ESM, CJS
(.cjs, with local requires rewritten), and declarations.drain() without arguments, wouldAdmit() without detail, and
{ input, maxOutput } reservation overrides behave exactly as in
3.7.0. The only observable difference is the validation-ordering change
above, which affects invalid requests on closed bulkheads only.LLMRequest.system — optional system prompt (string or content-block
array), counted by both built-in estimators exactly like message content.
Previously, callers had to fold the system prompt into a synthetic message
for it to participate in estimation, which also distorted events, logs,
and deduplication keys.
LLMRequest.extraInputTokens — a first-class channel for input tokens
the character-based estimators cannot see (tool schemas kept outside
messages, provider-priced media, etc.). Built-in estimators add the
value verbatim; custom estimators may honor or ignore it. Must be a
non-negative integer. Because it is an ordinary request field, it
participates in the default deduplication key — requests differing only
here are never conflated. This replaces the pattern of smuggling
out-of-band token costs through wrapper estimators or hidden properties.
opaqueBlockTokens on createModelAwareTokenEstimator — a
configurable input-token surcharge for opaque (non-text) content blocks:
either a flat number per block, or { default?, byType? } keyed by
block.type. Applies to blocks in messages[].content and system
arrays; validated at estimator creation. Without it, opaque blocks
contribute 0 input tokens (unchanged default) — an image-heavy request
estimates as nearly free, which is the wrong direction for admission
control. Malformed text blocks (type: "text" without a string text)
are treated as opaque, erring toward over-reservation.
Per-call reservation override — reservation?: TokenEstimate on
acquire() / run() options and on wouldAdmit() options. When
provided with tokenBudget configured, admission reserves
input + maxOutput from the override verbatim and skips the estimator
for that call. Intended for gateways that already compute a more accurate
estimate from the full provider request than any character-ratio
estimator could. Validated as non-negative integers; ignored when
tokenBudget is not configured; not reflected by estimate(), which
always previews the estimator path.
system / extraInputTokens produce
different default deduplication keys than hand-rolled 3.6.0 projections
of the same prompt — as intended, since those fields now distinguish
requests.Exact reservation preview via bulkhead.estimate(request). The new
method runs the same estimator and validation path used by
acquire(), run(), and wouldAdmit(), returning
{ input, maxOutput, reserved } without acquiring capacity. It returns
null when token-budget admission is disabled. This gives gateways and
external capacity coordinators one authoritative reservation calculation
instead of forcing them to duplicate estimator logic.
Stable admission IDs. Every successful admission now receives a
process-unique UUID exposed on the successful acquire() result,
LLMToken, LLMRunContext, and the admit / release lifecycle events.
Admission events also expose the resolved priority, and release events now
include the pre-release held-token count plus the final usage-event
sequence. These fields let gateways correlate HTTP requests, traces,
distributed leases, usage updates, and release records without maintaining
fragile side maps.
Ordered usage lifecycle events. Effective cumulative
reportUsage() updates now emit an event containing the admission ID,
priority, monotonically increasing per-admission sequence, previous/current
hold, hold delta, cumulative usage, output-cap state, and over-reservation
status. Duplicate or stale reports that do not increase either cumulative
usage field are suppressed. Sequence numbers allow external coordinators to
reject duplicate or out-of-order updates safely. The returned UsageReport
now also includes admissionId and the current sequence, so a gateway can
await an external absolute-hold update before forwarding more streamed data.
LLMEventMap["admit"] and LLMEventMap["release"] include additional
correlation and accounting fields. Existing listener behavior is unchanged;
the additions are backward-compatible for consumers that read only the
previous fields.
request.max_tokens validation is centralized in the shared reservation
path, so estimate(), wouldAdmit(), acquire(), and run() now validate
it consistently even when a custom estimator ignores that field.
ReadableStream, Node Readable, async iterables, Response
bodies — the shared object can only be consumed once: whichever caller
read first won, and the rest received a locked or drained stream, with
no error at the bulkhead. Followers whose shared result is detected as
single-consumer now reject with
LLMBulkheadRejectedError("unshareable_result") (new reject reason,
counted in stats().llm.rejectedByReason and emitted via the reject
event, without capacity detail — like other dedup-wait rejections).
The leader is never affected and always receives the original
result. Detection is deliberately shallow: only the result value itself
is inspected, so a stream nested inside a wrapper object is still
shared by reference as before.deduplication.shareResult fan-out hook. Called once per follower
with the leader's resolved result; its return value is what that
follower receives. This is the seam for making streaming dedup
work rather than merely fail loudly — e.g.
shareResult: (r) => (r as Response).clone() for fetch responses, or
a tee/replay of your provider stream. When provided, the hook runs for
every follower delivery (safe results included) and the
single-consumer detection is bypassed — the hook owns fan-out policy.
A throwing hook rejects that follower with the thrown error as-is;
leader and other followers are unaffected.
Per-call dedup: false on run() options. Opts a single call out
of deduplication entirely — it neither joins an existing in-flight
call nor registers as joinable. Intended for streaming routes on a
bulkhead where dedup is otherwise useful, replacing the previous
workaround of encoding exemptions in a bulkhead-wide keyFn.
dedup: true cannot enable deduplication when it is disabled at the
bulkhead level and is treated as omitted.
Note: This release is identical in content to what was intended to be published as
3.4.0. That version was published to npm and then unpublished, and npm's registry policy prevents re-publishing a version number once it has been unpublished. This release republishes the same changes as3.4.1.
Default deduplication key now covers the entire request. The old key
serialized only {messages, max_tokens, model}, silently conflating
requests that differed in any other field — identical messages with
temperature: 0 vs temperature: 1 shared one call, and the second
caller received a response generated under the first caller's parameters.
The default key is now a key-order-stable serialization of the whole
request object, so any own enumerable property difference prevents
conflation. Tradeoff: volatile per-request fields (request IDs,
timestamps) now also defeat deduplication — supply a custom keyFn that
omits them if your requests carry such fields. Missing a dedup
opportunity is cheap; serving a wrong-parameters response is a
correctness bug, so the default errs entirely toward non-conflation.
Dedup hit rates may drop for callers whose requests carry extra
fields; admission behavior is otherwise unchanged.
Dedup keys are SHA-256 hashed before storage. The in-flight map
previously held full serialized prompt text as its keys — unbounded
per-entry key memory, and prompt content resident for the lifetime of
the entry. Keys are now hashed (scope + \0 + raw key), bounding key
size and keeping prompt text out of the map. keyFn semantics are
unchanged: it still returns a plain string, and "" still opts out.
Bulkhead-level timeoutMs no longer applies to dedup followers.
timeoutMs is documented as a queue-wait timeout, but it was also
being applied to a follower's wait on an already-running shared call.
Under the batch profile (30s default), any LLM call slower than the
timeout caused every follower to reject with "timeout" while the
leader succeeded — defeating deduplication exactly when calls are long.
Followers are now capped only by their own signal and an explicitly
passed per-call timeoutMs. Callers who relied on the bulkhead
default bounding follower waits should pass timeoutMs per call.
reportUsage() after release() now reports held: 0. The token's
internal hold counter was not zeroed at release, so post-release
snapshots reported the stale pre-release hold while
stats().tokenBudget.inFlightTokens correctly showed the tokens
returned. Budget accounting was always correct; only the snapshot lied.
reserved continues to report the historical pre-admission reservation.
dedupScope option on run(). Requests deduplicate only within the
same scope; different scopes never share an in-flight call even with
identical keys. Intended for multi-tenant gateways: the default key has
no tenant dimension, so without a scope (or a tenant-aware keyFn),
two tenants sending byte-identical requests would share one response.
Default: "" (single global scope — prior behavior).
Documented explicitly: deduplication applies to run() only;
acquire() never deduplicates. (Existing behavior, previously
undocumented.)
effectiveBudget("normal") now clamps to 0 instead of going negative.
Construction validates 0 <= highPriorityReserve <= budget (a startup check
that catches config typos), but setBudget() does not re-run that
validation — a runtime budget update (e.g. driven by a lease-renewal
ledger) is trusted as-is, since rejecting the ledger's grant would be
incorrect. This means currentBudget can legitimately drop below
highPriorityReserve after setBudget(). Previously, the normal-priority
ceiling (currentBudget - highPriorityReserve) could go negative in that
state, which was surfaced as a negative effectiveBudget/available in
stats() and rejection detail. It is now Math.max(0, currentBudget - highPriorityReserve).
"budget_limit" while priority: "high" requests
are still checked against the full (shrunk) currentBudget and can
keep admitting whatever capacity remains. This is the intended degraded
behavior — highPriorityReserve exists specifically to protect
interactive traffic when capacity is scarce, and capacity is never
scarcer than when the grant itself falls below the reserve.stats().tokenBudget, rejection
detail.tokenBudget) from negative to 0.highPriorityReserve <= budget) is
unchanged.tokenBudget.budget: 0 is now accepted at construction. Previously,
createLLMBulkhead({ tokenBudget: { budget: 0 } }) threw because budget
was validated as a positive integer. A budget of 0 is a legitimate state
— e.g. a lease ledger reporting pool exhaustion for the current cycle — and
the bulkhead now represents it as "reject all budget-gated admissions"
rather than an invalid configuration. budget is now validated as a
non-negative integer, matching the existing validation already used by
setBudget(). Admission behavior is unaffected for any budget > 0; a
budget: 0 bulkhead rejects every admission that needs > 0 tokens with
"budget_limit", and admits requests whose estimator produces a 0-token
reservation, consistent with effectiveBudget()/tryReserveTokens()
semantics already in place for runtime-lowered budgets.bulkhead.setBudget(tokens) — mutate the token budget ceiling at
runtime. All admission math (acquire/run, wouldAdmit, rejection
detail, stats()) reads the ceiling dynamically, so a call to
setBudget() propagates immediately with no other behavioral changes.
inFlightTokens is legal — shrink by attrition.
No in-flight work is revoked or cancelled. New admissions reject with
"budget_limit" until enough in-flight work releases to bring
inFlightTokens back under the new ceiling. This is consistent with
the library's existing overrun tolerance (inFlightTokens can already
exceed budget via reportUsage() overrun) and is pinned with a
dedicated test.tokenBudget was never configured at construction — an
explicit error beats a silent no-op.tokens as a non-negative integer (0 is valid and
fully closes admission).tokenBudget.budget behavior is unchanged for
bulkheads that never call setBudget().Gateway-readiness release. All changes are additive (semver-minor).
LLMToken.reportUsage(usage) / run-context ctx.reportUsage(usage) —
mid-flight cumulative usage reporting for streaming workloads.
expands it (overrun), which can push inFlightTokensabovebudget` and correctly blocks new admissions
until the overrunning request releases.release() without explicit usage falls back to the last reported
usage for the final refund.UsageReport snapshot (reserved, held, consumed,
outputCap, outputRemaining, overReservation).tokenBudget.highPriorityReserve plus per-call
priority: "high" | "normal" on acquire()/run(). Normal-priority
admission is checked against budget - highPriorityReserve; high-priority
against the full budget.acquire() results, reject events, and
LLMBulkheadRejectedError now carry an optional detail: LLMRejectDetail
capacity snapshot (slots, queue, priority-adjusted budget numbers).wouldAdmit(request, { priority }) — advisory, non-reserving dry-run
for routing decisions. Documented as racy.tokenBudget.totalOverrun and tokenBudget.highPriorityReserve.UsageReport, ``LLMRunContext, LLMPriority, LLMRejectDetail, LLMAcquireOptions`.run() callbacks now receive an optional second argument
(LLMRunContext). Existing single-argument callbacks are unaffected.reportUsage() was used: refundedTokens
on the release event reflects the refund at release (against the
current hold); early refunds from reportUsage() are already included
in stats().tokenBudget.totalRefunded as they occur. When
reportUsage() is never called, behavior is byte-identical to 3.1.x.`
tokenBudget.budget, tokenBudget.outputCap, request.max_tokens, estimator output, and reported TokenUsage are now validated as finite non-negative integer token counts, with tokenBudget.budget required to be positive. Invalid usage passed to release() still releases capacity before surfacing the validation error.AbortSignal and timeoutMs while leaving the shared provider call running for other waiters.sourceMappingURL trailers and source map file fields from .js to .cjs.test:coverage script so vitest run --coverage works from a clean install.npm pack --dry-run.3.x line and removed the placeholder security email.model, matching the implementation and README.max_tokens, invalid estimator output, invalid usage reporting, one-estimation-per-acquire behavior, deduped abort/timeout behavior, and packaged ESM/CJS smoke checks.createModelAwareTokenEstimator({ defaultModel }) typing. The estimator factory now accepts an options object as the first argument when it includes defaultModel, matching external call-site expectations and avoiding the previous TypeScript error where defaultModel was interpreted as a numeric ratio override.createModelAwareTokenEstimator(overrides, opts) calls continue to work.stats().tokenBudget.totalReserved — cumulative tokens reserved at admission across all successful admissions. Monotonically increasing. Useful as the numerator companion to inFlightTokens/available for rate and saturation analysis over time.stats().tokenBudget.totalConsumed — cumulative actual tokens consumed (usage.input + usage.output), summed across releases that reported TokenUsage via token.release(usage) or run({ getUsage }). Releases without usage contribute 0 — totalConsumed is meaningful only when getUsage is wired up consistently. Not clamped: over-consumption (actual > reserved) is reported as-is.
When getUsage is wired consistently and no over-consumption occurs, the invariant totalReserved == totalConsumed + totalRefunded holds after all in-flight requests settle.release event JSDoc to document the per-request consumption math (usage ? usage.input + usage.output : null) and the null vs 0 distinction for unreported usage. Pure docstring change — no API or behavior change.totalRefunded field — no breaking changes.tokenBudget is configured (alongside totalRefunded); both are absent when tokenBudget is omitted.totalRefunded / totalReserved to surface overly generous max_tokens settings).release event remains the right primitive — it preserves the input/output split from TokenUsage, which totalConsumed collapses into a single sum. Token estimation in this library is approximate and intentionally suited to load-shedding, not finance-grade reconciliation.LLMStats shape changed. bulkhead.stats() no longer returns base Stats fields at the top level.
Base bulkhead stats now live under stats().bulkhead, and LLM-layer counters now live under
stats().llm.
Code that previously accessed:
stats().inFlightstats().pendingstats().maxConcurrentstats().maxQueuestats().closedmust now read:
stats().bulkhead.inFlightstats().bulkhead.pendingstats().bulkhead.maxConcurrentstats().bulkhead.maxQueuestats().bulkhead.closedstats().llm block with LLM-layer request counters:
admittedreleasedrejectedrejectedByReasonrun() callback signal type now derives from AcquireOptions["signal"]
instead of referring to the global AbortSignal type directly.AbortController globals.async-bulkhead-ts to ^0.4.1.From v2 → v3, update stats access only.
Before:
const s = bulkhead.stats();
s.inFlight;
s.pending;
After:
const s = bulkhead.stats();
s.bulkhead.inFlight;
s.bulkhead.pending;
LLM-layer counters are now separate:
const s = bulkhead.stats();
s.llm.admitted;
s.llm.rejected;
s.llm.rejectedByReason.budget_limit;
LLMMessage.content is now string | ContentBlock[]. Plain strings remain valid — no changes required for text-only callers. Code that assumed content is always a string (e.g. m.content.length) must be updated to use extractTextLength() or handle both shapes.max_tokens and model in addition to message content. Requests with identical messages but different max_tokens are no longer treated as duplicates (they were in v1). This is a behavioral change with no API signature change.LLMToken replaces Token: the token returned by acquire() now accepts optional TokenUsage at release time: token.release(usage?). Callers that call release() with no arguments are unaffected.LLMStats.tokenBudget shape: added totalRefunded field to the token budget stats block.TokenUsage is passed to token.release(usage) or extracted via getUsage in run(), the bulkhead returns the difference between the pre-admission reservation and actual consumption to the budget immediately. This dramatically improves budget utilization — requests that use fewer output tokens than max_tokens no longer hold phantom capacity.run() getUsage option. run(request, fn, { getUsage }) accepts a function that extracts TokenUsage from the result of fn. The refund is applied automatically on successful completion.LLMRequest.model is now an optional field. When present, the model-aware estimator uses it for ratio lookup instead of the bulkhead-level defaultModel. Supports A/B testing, canary deployments, and mixed-model routing.LLMMessage.content accepts ContentBlock[] with typed TextContentBlock and OpaqueContentBlock variants. Built-in estimators extract text from text blocks and ignore non-text blocks. extractTextLength() is exported as a utility.deduplication now accepts DeduplicationOptions with a keyFn property. The default key function includes messages, max_tokens, and model. Return an empty string from keyFn to opt a specific request out of deduplication.bulkhead.on(event, listener) subscribes to lifecycle events: 'admit', 'reject', 'release', 'dedup'. Returns an unsubscribe function. Listeners are called synchronously; exceptions are silently caught.bulkhead.close() stops admission permanently. bulkhead.drain() returns a promise that resolves when all in-flight work completes. Compose as close() → drain() for clean shutdown. Both are forwarded from async-bulkhead-ts.claude-haiku-4, claude-sonnet-4-5, claude-opus-4-5, claude-haiku-4-5, gpt-4.1, o4-mini, gemini-2.5.LLMStats.tokenBudget.totalRefunded — cumulative tokens returned to the budget via the refund mechanism.createModelAwareTokenEstimator now checks request.model before defaultModel for ratio lookup.max_tokens and model (see Breaking Changes).filter + sort, avoiding an allocation per call.LLMBulkhead return type now includes close, drain, and on methods.From v1 → v2, most callers need zero changes. The common path — createLLMBulkhead(opts) → bulkhead.run(request, fn) — is fully backward-compatible for text-only requests.
Changes required only if you:
content directly on LLMMessage: replace m.content.length with extractTextLength(m.content) or guard with typeof m.content === 'string'.max_tokens as identical: pass a custom keyFn that omits max_tokens: deduplication: { keyFn: (r) => JSON.stringify(r.messages) }.tokenBudget stats structurally: the new totalRefunded field is always present when tokenBudget is configured.release() with no argument behaves identically to v1.getUsage is intentionally separated from fn to avoid coupling the LLM call signature to the bulkhead. The caller extracts usage from whatever their provider returns.close() and drain() are thin forwards to async-bulkhead-ts. The LLM layer adds no new shutdown semantics — it inherits the base library's guarantees.estimator.package.json (homepage, repository, bugs) to point to the canonical repository1.0.0 and 1.0.1async-bulkhead-ts to ^0.3.0async-bulkhead-llmcreateLLMBulkhead(options) — fail-fast admission control for LLM workloads, wrapping async-bulkhead-tsmodel required at construction time — one bulkhead per model is the enforced deployment patternprofile option — 'interactive' (fail-fast, default) and 'batch' (bounded queue, 30s timeout) presets; escape hatch via plain LLMBulkheadPreset object; explicit options always override preset defaultstokenBudget — reserves input + maxOutput tokens pre-admission; fail-fast when the budget ceiling is hit, independent of concurrency headroom and profilenaiveTokenEstimator — flat 4.0 character-per-token ratio; zero configuration; suitable for load-sheddingcreateModelAwareTokenEstimator — per-model character ratios for known model families across Anthropic, OpenAI, and Google; longest-prefix match; exact caller overrides checked before prefix scan; onUnknownModel hook; falls back to 4.0 ratio for unknown modelsdeduplication: true — identical requests (keyed on JSON.stringify(messages)) share one in-flight LLM call; dedup hits tracked in stats()bulkhead.run(request, fn, options?) — primary API; acquire + release handled automatically; throws LLMBulkheadRejectedError on rejectionbulkhead.acquire(request, options?) — manual acquire / release for advanced control flow; returns typed result objectLLMBulkheadRejectedError — typed error with code: 'LLM_BULKHEAD_REJECTED' and reason: LLMRejectReasonLLMRejectReason — extends base RejectReason from async-bulkhead-ts with 'budget_limit'TokenUsage type — exported as a forward-looking type for v2 refund support; not acted on in v1bulkhead.stats() — returns LLMStats extending the base Stats type with optional tokenBudget and deduplication blocks; optional blocks are absent when the feature is disabledPROFILES exported — named presets available for direct inspection and compositiontimeoutMs cancellation — threaded through to fn via run(); waiting-only timeout semantics inherited from async-bulkhead-tsmaxQueue: 0 unless overridden via profile or explicit optionmodel is a required constructor argument and does not appear on LLMRequest; multi-model routing is documented as a README recipeTokenUsage type is exported now to allow call sites to be written correctly ahead of v2content must be a plain string; estimators ignore non-string content; documented on LLMRequestJSON.stringify(messages) in v1 — requests with identical messages but different max_tokens are treated as duplicates; key design improvements deferred to v2tryAcquire() from the base library is not exposed — synchronous non-blocking admission is not meaningful for LLM workloads where token budget requires the full request objectasync-bulkhead-ts