AnyInfer¶

Application-owned hybrid inference, from hosted APIs to a supervised local model.

Quickstart Installation Downloads

Typed events

A generation is an ordered stream of typed events; non-streaming is just the drained stream.

Routing & fallback

Deterministic retries, failure-specific fallback chains, and health gating — fully traceable after the fact.

Local inference

Hardware detection, tuning, verified downloads, and supervised llama-server — one target string.

Structured output

A schema is a contract: strongest native mechanism, always client-side validated, optional bounded repair.

Context budgets

Estimate fit and cost before dispatch, then reduce approved context with an explicit record of what was omitted.

Observability

Typed in-process telemetry, payload-free by default, with a lazy OpenTelemetry bridge.

SyncAsync

import anyinfer as ai

client = ai.Client([ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY")])
result = client.generate(prompt, target="anthropic:claude-sonnet-4-5")
print(result.text)

import anyinfer as ai

async with ai.AsyncClient([ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY")]) as client:
    result = await client.generate(prompt, target="anthropic:claude-sonnet-4-5")
    print(result.text)

Install¶

pip install anyinfer

The core depends on only httpx2 and jsonschema. Provider SDKs, the sidecar, and the demo app are optional extras. Local inference is part of the core. See installation and extras.

What AnyInfer gives you¶

Applications that talk to more than one model provider accumulate the same layer every time: per-provider request shaping, SSE parsing, retry logic, token accounting, and a tangle of if engine == ... branches that leaks into config screens and error handling. AnyInfer is that layer, extracted and made rigorous:

One primitive. A GenerationRequest becomes a typed event stream. Non-streaming is the drained stream. It is not an OpenAI-API clone — the OpenAI dialect is one edge format among several.
Adapters only translate. Retry, fallback, health gating, schema validation, repair, TTFT measurement, usage normalization, telemetry, and redaction live in the core, once.
Structured output is a contract. A request carrying a schema always returns a client-side-validated result, using the strongest mechanism the provider offers (grammar → json_schema → json_mode → prompt), with an opt-in bounded repair loop.
Context engineering is connected to dispatch. Preflight budgets, cost ranges, deterministic reduction, and hierarchical distillation all use the target's actual capability data and report uncertainty or omission instead of hiding it.
Capabilities carry provenance. Every context window, price, and feature flag records whether it was catalogued, discovered, probed, or defaulted. Nothing is guessed silently.
Local inference is first-class. Hardware detection, backend selection, llama-server supervision and tuning, verified GGUF downloads, and hardware→tier recommendation.

One engine, four kinds of target¶

Environment	Examples	What AnyInfer owns
Hosted provider	OpenAI, Anthropic, Gemini, Bedrock	Native protocol translation and capability discovery
Router or hub	OpenRouter, compatible gateways	Targeting, normalized events, and shared routing policy
Existing local service	Ollama, LM Studio, vLLM	Native or compatible client behavior; the service keeps process ownership
Managed local runtime	`llama.cpp`	Runtime and model acquisition, hardware fit, tuning, supervision, and loopback lifecycle

→ See the compatibility inventory — dedicated protocol adapters and declarative presets, from frontier APIs to local engines.

See the provider guides and the conformance matrix for exactly what each supports.

Next steps¶

Deciding whether you need this layer?

Start with when to use AnyInfer. It names the cases where a provider client, organization gateway, or dedicated local server is the better tool.
Integrating into an app?

The Python SDK guide covers lifecycle, generation, streaming, and errors. The Quickstart is the five-minute route.
Want existing OpenAI clients to use the same route?

Run the sidecar — an OpenAI-compatible loopback service, available as a Python extra or a standalone download. Anything that can point at an OpenAI base URL can use the providers, routes, and local models you configured.
Working from a shell?

anyinfer run sends one prompt through the same routing and structured-output path and streams the answer to stdout, then exits — no server to keep running, no Python to write.
Just want to see it?

The pack-in demo app runs fully offline against in-process fakes — streaming, retry and fallback, structured output, and live telemetry, no credentials required. Grab a standalone build from Downloads.
Reading code first?

Start with the examples — small, complete programs that run verbatim in CI — then the SDK reference.
Configuring more than one path?

Use one shared configuration file for the Python SDK, command-line tool, and sidecar.

Pre-1.0 and under active development. Python 3.11+; Windows, macOS, and Linux are all first-class. MIT licensed. Sources, design documents, and the issue tracker live at github.com/anthturner/AnyInfer.