OptionalkeyCustom function to derive a deduplication key from a request. Requests with the same key that arrive while a matching call is already in-flight share that call.
Default (v3.4): a key-order-stable serialization of the entire
request object. Any own enumerable property difference —
temperature, tools, system, etc., not just messages /
max_tokens / model — prevents conflation. The tradeoff: volatile
per-request fields (request IDs, timestamps) also defeat
deduplication. If your requests carry such fields, supply a keyFn
that omits them. Non-serializable values (functions, symbols) are
dropped by JSON serialization and do not distinguish requests.
Keys are SHA-256 hashed before being stored, so the in-flight map never retains prompt text and per-entry key memory is bounded.
Deduplication applies to run() only — acquire() never
deduplicates.
Multi-tenant callers: the default key has no tenant dimension.
Two tenants sending byte-identical requests would share one
response. Pass dedupScope (per-call, on run() options) or bake
the tenant into a custom keyFn to prevent cross-tenant sharing.
When bounded admission classes are configured, the selected class is
automatically included as a separate deduplication partition;
dedupScope is still required to isolate tenants within one class.
Return an empty string to opt a specific request out of deduplication.
OptionalshareFan-out hook for delivering a shared in-flight result to deduplication followers (v3.5).
Called once per follower with the leader's resolved result; the
return value is what that follower receives. The leader always
receives the original result — shareResult is never called for
the leader.
Why this exists: without it, followers receive the same object
the leader does. For plain JSON results that is fine (and remains
the default), but single-consumer values — ReadableStream,
Node Readable, async iterables, Response bodies — can only be
consumed once. Whichever caller reads first wins; the rest get a
locked or drained stream. The library cannot tee arbitrary stream
types on your behalf, so it provides this seam instead:
// fetch Response: hand each follower an independent clone.
// (clone() must be called before any consumer reads the body —
// guaranteed here, since fan-out happens at resolution time.)
deduplication: { shareResult: (r) => (r as Response).clone() }
When shareResult is provided it is called for every follower
delivery, including safe (non-stream) results — it is a general
fan-out policy, usable e.g. for defensive deep-cloning.
If shareResult throws, that follower's run() rejects with the
thrown error as-is; the leader and other followers are unaffected.
When shareResult is not provided and the shared result is
detected as single-consumer, followers are rejected with
LLMBulkheadRejectedError("unshareable_result") rather than being
silently handed a stream they cannot read. Detection is shallow:
it inspects the result value itself, not nested properties — a
stream buried inside { stream: ... } is not detected and will be
shared by reference as before.
async-bulkhead-llm — public API surface.
This entry point re-exports everything the package supports; the implementation lives in focused modules:
types.ts— request/result/options/stats/event typeserrors.ts—LLMBulkheadRejectedErrorprofiles.ts—PROFILESpresetsestimators.ts— naive + model-aware estimators,extractTextLengthadaptive.ts—createAdaptiveTokenEstimator(v3.8)dedup.ts— deduplication internals (keying, share safety)validation.ts— internal numeric/estimate/usage guardsbulkhead.ts—createLLMBulkhead(admission, budget, events)Deep-importing the internal modules is not supported; the package
exportsmap exposes only this entry point.