Keep your AI project running when free APIs hit their limits.

pip install ai-switch

The distribution is named ai-switch — PyPI rejected llmswitch as too similar to an unrelated existing project. The import name is unchanged.

Set a key for any provider you want in the rotation — LLMSwitch only uses the ones it finds:

export GROQ_API_KEY="..."
export OPENROUTER_API_KEY="..."
export NVIDIA_API_KEY="..."

Then write the call once:

from llmswitch import AIClient

client = AIClient()
print(client.chat("Write a Python web scraper"))

Your code never names a provider. That is the whole idea.

Providers without a key are skipped silently at startup.

Ollama as a local last resort

Ollama needs no API key and never rate-limits, which makes it a natural final fallback once every cloud free tier is spent. It is opt-in: set OLLAMA_HOST to enable it, so anyone without a local server does not pay for a failed attempt on every request.

export OLLAMA_HOST="http://localhost:11434/v1"

To pick the set and the order yourself:

from llmswitch import AIClient
from llmswitch.providers import GroqProvider, NVIDIAProvider

client = AIClient(providers=[GroqProvider(), NVIDIAProvider()])

A single chat() call walks the provider list until one answers:

Two failure kinds drive this. RateLimitError means you have been throttled or have burned the quota. ProviderUnavailableError means the provider is down, unreachable, or rejected the request. Anything else propagates untouched — a bug in your payload should not silently cost you three API calls.

If every provider fails, LLMSwitch raises ProviderError naming each failure, so you can see what actually went wrong.

Falling back on every request is not enough. Without memory, a rate-limited Groq is called again on the next request, and the one after that — burning a wasted round-trip every time before failing over. LLMSwitch rests a failed provider and moves it to the back of the queue:

client.chat("first")   # groq 429 -> falls back to openrouter
client.chat("second")  # groq skipped entirely; openrouter serves it

Inspect the live picture at any time with client.status():

How long a provider rests

• If the provider sent a Retry-After header, that wins — it knows better than any guess.
• Otherwise rate limits back off exponentially from 60s, doubling per consecutive failure, capped at 15 minutes.
• Transient outages start at 10s and cap at 5 minutes, since they usually clear fast.
• One success resets the streak, so a brief blip does not poison a provider for the rest of the session.

The same model is named three different things by three providers. Ask for the alias and LLMSwitch resolves it per provider:

client.chat("hello", model="fast-model")        # resolved per provider
client.chat("hello", model="llama-3.1-8b-instant") # raw names pass through

with AIClient() as client:
    for token in client.stream("Write a haiku about failover"):
        print(token, end="", flush=True)

What happens when a stream breaks

Fallback is transparent only up to the first token. Once tokens are in your hands they cannot be un-yielded, so a later failure raises StreamInterrupted carrying the partial text — rather than silently restarting the answer on another provider and making your reader watch it begin again mid-sentence.

except StreamInterrupted as exc:
    print(exc.partial)   # everything received before the break
    print(exc.provider)  # which provider dropped it

Async

AsyncAIClient shares the sync client's routing core, so fallback, cooldowns and retry behave identically — only the transport differs.

async with AsyncAIClient() as client:
    print(await client.chat("hello"))
    async for token in client.stream("count to five"):
        print(token, end="")

Connections

Each provider holds a pooled connection, reused across calls. Close the client when done — the context manager above is the tidy path; forgetting it still works, just cleaned up later.

Retries

A timeout usually means one unlucky request rather than a dead provider, so the same provider gets one retry before failover. A 429 is never retried — the provider has explicitly said stop.

AIClient(providers=None, health=None)

Both arguments are injectable, which is what makes the router testable without network access.

Exceptions

Any provider speaking the OpenAI chat-completions shape needs three lines:

from llmswitch.providers.base import OpenAICompatibleProvider

class MyProvider(OpenAICompatibleProvider):
    name = "myprovider"
    api_base = "https://api.example.com/v1"
    api_key_env = "MYPROVIDER_API_KEY"

The base class handles auth headers, the User-Agent, Retry-After parsing, and mapping status codes onto the two error types. For a provider with its own wire format (Gemini, for instance), subclass Provider and implement chat() directly — just raise the same two exceptions so the router can do its job.