async-bulkhead-llm - v3.17.0

async-bulkhead-llm

Decide whether an LLM call should run — before it runs.

What it does

You wrap your LLM calls in a bulkhead. Before each call runs, the bulkhead decides whether it should run — based on how many calls are already in flight and how many tokens are already committed. If the answer is no, you get a fast, typed rejection instead of a request that piles onto an already saturated provider.

That's it. It's one decision, made in-process, in microseconds, before any bytes leave your service.

await bulkhead.run(request, () => callYourProvider(request));
//     ^ throws LLMBulkheadRejectedError instead of making things worse

Should you use it?

Probably yes if: you call an LLM provider from a service that takes concurrent traffic, and you've either already watched a burst turn into a 429 storm or a saturation cascade — or you'd rather not.

Probably not if: you're building a CLI, a notebook, or anything where one request happens at a time. There's nothing to protect.

Not a replacement for: LangChain (orchestration), your provider SDK (the actual calls), or a retry library (what happens after a failure). This sits underneath all of them and answers a different question. See How it compares.

Install

npm install async-bulkhead-llm

Node.js 20+. ESM and CommonJS. TypeScript types included. One dependency (async-bulkhead-ts).

Quick start

import { createLLMBulkhead, LLMBulkheadRejectedError } from 'async-bulkhead-llm';

const bulkhead = createLLMBulkhead({
  model:         'claude-sonnet-4',
  maxConcurrent: 10,
});

const request = {
  messages:   [{ role: 'user', content: 'Summarise this document...' }],
  max_tokens: 1024,
};

try {
  const result = await bulkhead.run(request, async () => {
    return callYourLLMProvider(request);
  });
} catch (err) {
  if (err instanceof LLMBulkheadRejectedError) {
    // Shed on purpose. err.reason tells you which limit was hit.
    return respond503(err.reason);
  }
  throw err;
}

Ten calls run at once. The eleventh is rejected immediately with reason: 'concurrency_limit' — it does not wait, and it does not reach your provider. That is the whole idea: fail fast beats fail slow.

If you'd rather queue than shed, add maxQueue: 20. If you want queued callers to give up after a while, add timeoutMs: 5_000. Both are off by default, deliberately.

Limit tokens, not just requests

maxConcurrent counts requests. It cannot tell a 500-token request from a 100,000-token one — ten of each look identical to it. tokenBudget adds the second dimension:

const bulkhead = createLLMBulkhead({
  model:         'claude-sonnet-4',
  maxConcurrent: 10,
  tokenBudget:   { budget: 200_000 },
});

Each request now also reserves estimated input + max_tokens before it runs. When the budget is committed, further requests are rejected with budget_limit.

This is an in-flight ceiling, not a spend cap. The budget limits how many tokens may be reserved at the same time; every reservation is returned when its request completes. A 200,000-token budget does not mean you will spend 200,000 tokens — it means no more than that much work is ever committed at once. It is a load-shedding control, not a billing control.

Reservations are estimates, so give the budget back when you know the truth:

await bulkhead.run(request, () => callLLM(request), {
  getUsage: (response) => ({
    input:  response.usage.input_tokens,
    output: response.usage.output_tokens,
  }),
});

The unused portion returns to the in-flight budget immediately. Most calls use far fewer output tokens than max_tokens, so this frees real admission capacity.

→ Token budget and estimation

Handling rejections

Every rejection is typed and carries a capacity snapshot, so you can return something useful instead of a generic 500:

const r = await bulkhead.acquire(request);

if (!r.ok) {
  // r.reason: 'concurrency_limit' | 'queue_limit' | 'budget_limit'
  //         | 'timeout' | 'aborted' | 'shutdown'
  // r.detail: slots, queue depth, priority-adjusted budget numbers
  return respond429(r.reason, r.detail);
}

No Retry-After is fabricated — a fail-fast bulkhead has no honest ETA. If you need one, your gateway knows more than the bulkhead does.

Watching it work

const s = bulkhead.stats();
s.bulkhead.inFlight;              // slots in use
s.llm.rejectedByReason;           // why you're shedding
s.tokenBudget?.available;         // budget headroom

bulkhead.on('reject', ({ reason, detail }) => metrics.increment(reason));

admit and admission-decision reject events also expose optional decisionDurationNs and queueWaitNs fields. Decision time excludes the underlying concurrency wait, so local admission cost and queue contention stay separate.

Admissions also expose exact resource attribution through resources. For class policies that lend shared concurrency, run() can enforce a wall-clock lease on the borrowed local slot:

await bulkhead.run(request, callProvider, {
  admissionClass: 'background',
  borrowedConcurrencyDeadlineMs: 30_000,
});

Expiry returns that local concurrency slot, aborts the callback signal, and rejects with LLMBorrowedConcurrencyDeadlineError. It deliberately retains the token hold until the callback settles. This is a local restoration bound, not proof that a provider account, accelerator queue, or remote generation was reclaimed. Keep an unlent floor or a real provider-side partition for upstream capacity that client cancellation cannot authoritatively release.

→ Stats and events

Shutting down cleanly

// In your SIGTERM handler:
bulkhead.close();                             // stop admitting
await bulkhead.drain({ timeoutMs: 10_000 });  // wait, but not forever

drain() with a deadline always resolves and reports what remains outstanding; it does not abandon or release that work. One stuck upstream stream therefore cannot hold the shutdown decision open indefinitely.

Without timeoutMs, drain() waits for final token settlement as well as base concurrency. In particular, returning a borrowed slot with abandonBorrowedConcurrency() does not complete the admission: a manual acquire() caller must still call release(), or the unbounded drain remains pending. This preserves the distinction between restoring a local slot and settling the resource accounting for the work behind it.

Documentation

Guide	What's in it
How admission works	The decision path, `run()` vs `acquire()`, profiles, cancellation, shutdown
Token budget and estimation	Reservations, refunds, estimators, adaptive calibration, priority reserve
Streaming and reconciliation	`reportUsage()`, progressive holds, ordered usage events
Admission classes	Per-class floors and ceilings, shared capacity, borrowing
Runtime limits and reconfiguration	`applyLimits()`, revisions, fail-closed startup, kill switch
Observe mode	Measure a policy before you enforce it
Deduplication	In-flight dedup, tenant scoping, streaming results
Gateway integration	`wouldAdmit()`, admission IDs, external leases, control planes
Stats and events	Every counter and every event payload
How it compares	Against LangChain, p-limit, cockatiel, and raw SDKs
Migration	Upgrade notes for every release since v2

The full API reference is generated from the source types on every push to main, so it is never out of date with the code. Release-by-release detail lives in the changelog.

What it deliberately doesn't do

No retries. Compose a retry library around the bulkhead, not inside it.
No provider SDK. Bring your own client.
No distributed coordination. Bulkheads are per-process. Partition the budget per replica, or coordinate above this layer.
No cost accounting or spend caps. Token estimation is for load-shedding, not billing. Nothing here tracks cumulative spend or stops you reaching a dollar figure.

Each of those is a deliberate boundary, not a missing feature. A bulkhead that also retried would defeat its own purpose.

Compatibility

Node.js: 20+
Module formats: ESM and CommonJS
Types: shipped, generated from source
Public API: only the package entry point. Deep imports into src/ internals are not supported.

Contributing and security

Issues and pull requests are welcome at github.com/janbalangue/async-bulkhead-llm. For security reports, see the security policy.