Announcing Distillery: An open-source AI gateway that preserves provider differences

Most AI applications begin with one model and one API key. Then reality sets in.

A team adds Anthropic for one workload, Gemini for another, Bedrock for an AWS deployment, or Azure OpenAI for a customer with specific infrastructure requirements. Before long, credentials live in several places, usage is hard to compare, and every application has its own partial implementation of routing, quotas, and observability.

We built Distillery to give that traffic one operational boundary.

Distillery is an open-source, multi-provider AI gateway. It sits between your applications and model providers, where it centralizes credentials, routes requests, enforces policy, measures usage, and optionally captures structured interaction data.

We spend most of our time at Tonic.ai on the data side of AI development: generating it, de-identifying it, and getting it into engineers' hands safely. A gateway is where a lot of that data actually moves. Building Distillery gave us a place to apply what we already know about handling sensitive data in motion.

Today, we are releasing Distillery 0.1.0 under the MIT License.

How Distillery handles provider differences

There is an appealing idea behind many AI gateways: make every provider look the same. That helps in narrow cases, but broadly, it breaks down quickly. Providers have different endpoints, authentication schemes, streaming events, tool formats, and model-specific capabilities. Flattening all of that into one abstraction usually costs you the features that made a provider worth using in the first place.

Distillery takes a different approach: it supports OpenAI-compatible clients and preserves native provider APIs alongside them. The initial release supports OpenAI, Anthropic, Gemini AI Studio, Azure OpenAI, Vertex AI, and Amazon Bedrock, including OpenAI Chat Completions and Responses, Anthropic Messages, Gemini and Vertex generateContent, and Bedrock Converse and InvokeModel.

The gateway understands enough about each supported provider to select credentials, construct the right destination, adapt authentication, and normalize useful metadata. Unmatched native operations pass through untouched, and normalization stays limited to the request shapes Distillery explicitly understands. You keep the provider-specific capabilities you chose each provider for, and you still get one place to manage credentials and policy.

Put operational policy at the boundary

Once traffic crosses a shared gateway, controls that were previously scattered through application code live in one place.

Distillery provides proxy-issued keys, tenant-aware credential mapping, quotas, kill switches, usage and cost metrics, and an admin UI. It can turn supported calls into provider-neutral interaction records containing timing, usage, model, request, and response information.

Those interaction records are worth dwelling on. If you want to evaluate how an agent actually behaved, attribute spend to a team, or explain to an auditor what your application sent to a third-party model six months ago, the gateway is the only component that sees all of it. Capturing that traffic in one consistent shape means you can answer those questions without instrumenting every application separately.

Version 0.1.0 focuses on a narrower promise than a full traffic manager: reliably operate and observe traffic whose provider destination is known. A model-response cache and a generic load balancer are both valuable, and both require product decisions about semantic equivalence, failure behavior, and model selection that we would rather make carefully than ship early.

Privacy controls should be explicit

An AI gateway sees some of the most sensitive data in your application. Prompts carry customer records, support tickets, clinical notes, and internal documents. The moment you start capturing interaction records for evaluation or audit, you have created a new store of sensitive data in a new place, with its own retention, access, and residency questions attached.

We do not think capture or redaction should be implied by a checkbox and a broad claim.

Distillery keeps raw capture off by default. You can opt into local JSONL or HTTP capture, with bounded queues and size limits. Failed HTTP deliveries can be retained in a durable spool and replayed explicitly rather than disappearing silently.

Optional redaction is powered by Tonic Textual and is also off by default. Textual detects sensitive entities using proprietary NER models, then either redacts them or replaces them with realistic synthetic values. That second option matters at a gateway. A log full of blacked-out spans tells you very little about how your application behaved, while a log where names, dates, and account numbers have been swapped for consistent synthetic stand-ins is still usable for evaluation and analytics. Teams already run Textual as an LLM privacy proxy in front of model providers, and Distillery makes that same control available at the gateway.

Redaction in Distillery operates only on documented provider dialects. Unsupported redaction paths fail closed by default, and the documentation is direct about the paths it does not cover. Treat it as a targeted privacy control at the boundary rather than a universal DLP layer.

Where Distillery 0.1.0 fits today

Distillery 0.1.0 is built for a single instance or a small internal deployment. Metrics, quotas, and interaction indexing use local SQLite stores, so they are not coordinated across replicas. High availability and globally shared quota enforcement will require a shared storage adapter in a future release.

The container runs as a non-root user, includes a health check, and ships with a loopback-bound Compose configuration. A public deployment still needs the controls you would expect around any sensitive gateway: authentication, rate limiting, encrypted storage, upstream restrictions, and network policy.

We would rather make those boundaries clear now than call an early release "enterprise-ready" and let you discover them later.

Try Distillery

You can run Distillery locally with Python 3.12 or newer and uv:

git clone https://github.com/TonicAI/distillery.git
cd distillery
uv sync
DISTILLER_HOST=127.0.0.1 uv run distillery

That starts the conservative default configuration: local-only, no proxy-key authentication, and no raw capture. From there, the project documentation walks through provider credentials, tenant routing, quotas, indexing, capture, and optional redaction.

Distillery is available now at github.com/TonicAI/distillery. We would love feedback from teams operating real applications across more than one AI provider, especially on the provider surfaces and operational controls that should come next.