async-bulkhead-llm - v3.17.0
    Preparing search index...

    Type Alias LLMBulkheadOptions

    async-bulkhead-llm — public API surface.

    This entry point re-exports everything the package supports; the implementation lives in focused modules:

    • types.ts — request/result/options/stats/event types
    • errors.ts — LLMBulkheadRejectedError
    • profiles.ts — PROFILES presets
    • estimators.ts — naive + model-aware estimators, extractTextLength
    • adaptive.ts — createAdaptiveTokenEstimator (v3.8)
    • dedup.ts — deduplication internals (keying, share safety)
    • validation.ts — internal numeric/estimate/usage guards
    • bulkhead.ts — createLLMBulkhead (admission, budget, events)

    Deep-importing the internal modules is not supported; the package exports map exposes only this entry point.

    type LLMBulkheadOptions = {
        admissionClasses?: LLMAdmissionClassesOptions;
        deduplication?: boolean | DeduplicationOptions;
        initialRevision?: number;
        maxConcurrent: number;
        maxQueue?: number;
        model: string;
        profile?: "interactive" | "batch" | LLMBulkheadPreset;
        timeoutMs?: number;
        tokenBudget?: TokenBudgetOptions;
    }
    Index
    admissionClasses? deduplication? initialRevision? maxConcurrent maxQueue? model profile? timeoutMs? tokenBudget?
    admissionClasses?: LLMAdmissionClassesOptions

    Optional bounded policy-class table for tenant/application isolation.

    The library enforces protected floors plus hard class ceilings. Capacity above a class floor is accounted as borrowed from the shared remainder. Identity mapping, weights, and distributed lending policy remain gateway / control-plane responsibilities. Class keys are fixed at construction to keep state and metric cardinality bounded; applyLimits() may change only their numeric limits.

    deduplication?: boolean | DeduplicationOptions

    Enable in-flight request deduplication.

    • true — use the default key function
    • DeduplicationOptions — customize the key function
    • false / omitted — disabled
    initialRevision?: number

    Initial revision for the admission-limit snapshot. Default: 0. Subsequent applyLimits() calls must use a strictly higher revision.

    maxConcurrent: number

    Maximum number of requests in-flight simultaneously. Set to 0 to start fail-closed until a higher-revision limit snapshot enables work.

    maxQueue?: number

    Maximum number of requests waiting for admission. Default: 0 (fail-fast). Prefer setting via profile.

    model: string

    Model identifier. Used by the default estimator for ratio lookup.

    One bulkhead per model is the strongly recommended deployment pattern. When routing multiple models through one bulkhead, set model on individual LLMRequest objects for accurate estimation.

    profile?: "interactive" | "batch" | LLMBulkheadPreset

    Opinionated defaults for common deployment patterns. Explicit options always override profile defaults.

    timeoutMs?: number

    Waiting timeout in milliseconds. Applies to queued requests only. Has no effect when maxQueue is 0.

    tokenBudget?: TokenBudgetOptions

    Token budget enforcement. Admission fails fast when the in-flight token ceiling is reached, regardless of concurrency headroom or profile. Omit to disable token-aware admission.

    Settings
    Member Visibility

    On This Page
    Properties