Model routing with Eevo
With the rapid development of frontier large language models, an increasing share of real-world tasks don't require frontier intelligence, and can be handled considerably more cost-efficiently with smaller, more efficient models.
Selecting the best model for a given problem requires analyzing various attributes of the task, usually including:
- Difficulty
- Category (coding, writing, image generation, etc.)
- Stakes (consequences of getting it wrong or right)
- Tool requirements (what kind of tools the model needs)
- Sensitivity (whether the task involves handling PII, etc.)
In addition, for organisations that use AI subscriptions, one may wish to consider factors unrelated to the task, like "how much Claude subscription quota does this employee have left".
Using decision models to analyze a task's attributes (difficulty, category, stakes, tool requirements, etc.) gives organisations an opportunity to create smart routing policies that take both the attributes and other organisational context into account, flexibly routing to models from various providers, including in-house inference for sensitive tasks.
Example
Below is an example snippet for detecting a task's attributes for routing purposes:
import os
import requests
def call_eevo(state, questions):
response = requests.post(
"https://eevo.io/api/v1/systemone",
headers={"Authorization": f"Bearer {os.environ['EEVO_API_KEY']}"},
json={"model": "eevo-1", "state": state, "questions": questions},
)
response.raise_for_status()
return response.json()["answers"]
ROUTING_QUESTIONS = {
"category": {
"type": "choice",
"instructions": "What kind of work does answering the last user message take?",
"criteria": {
"coding": "Writing, explaining, testing, debugging or changing software",
"writing": "Writing or editing prose: emails, documents, posts",
"math": "Mathematical reasoning, proofs, or quantitative problems",
"knowledge": "Explaining facts or answering questions that need knowledge",
"summarisation": "Condensing or restructuring given material, keeping what matters",
"design": "Creating or evaluating a design, an architecture or a structured solution",
"image_generation": "Creating or editing an image",
"agentic": "Planning and carrying out several steps or actions on its own",
"general": "Conversation, or anything that fits none of the others",
},
},
"difficulty": {
"type": "score",
"instructions": "How capable a model does it take to answer the last user message reliably?",
"criteria": [
"Trivial: any small model answers it reliably",
"Easy: a small model answers it reliably with a little care",
"Moderate: needs a mid-size model",
"Hard: needs one of the strongest models",
"Frontier: even the strongest models are likely to struggle",
],
},
"stakes": {
"type": "score",
"instructions": "What would a wrong or misleading answer cost, given how it will be used?",
"criteria": [
"Negligible: little or no consequence",
"Low: minor inconvenience or rework",
"Medium: material financial, operational, legal, health or reputational harm is plausible",
"High: serious harm, major losses or disruption is plausible",
],
},
"needs_tools": {
"type": "noul",
"instructions": "Answering it needs tools: running code, searching, or calling another system.",
},
"needs_long_context": {
"type": "noul",
"instructions": "Answering it needs reasoning across a long document, a large codebase or many files at once.",
},
"personal_data": {
"type": "noul",
"instructions": "The conversation contains personal data about an identifiable person, such as contact details, an ID number, or health or financial records.",
},
}
MESSAGES = [
{
"role": "user",
"content": "A customer, Jonas Weber (jonas.weber@example.com), says he was charged twice for his annual plan. Here is the payments export for his account.",
},
{
"role": "user",
"content": "Can you check the export, work out whether the second charge was a duplicate, and draft a reply to him with the refund amount?",
},
]
answers = call_eevo({"conversation": MESSAGES}, ROUTING_QUESTIONS)
print(answers)Below is an example response:
{
"category": {
"type": "choice",
"choice": "writing",
"confidence": 0.6966,
"probabilities": {
"coding": 0.0066,
"writing": 0.6966,
"math": 0.036,
"knowledge": 0.0105,
"summarisation": 0.0233,
"design": 0.0053,
"image_generation": 0.0021,
"agentic": 0.1753,
"general": 0.0443
}
},
"difficulty": {
"type": "score",
"score": 2.2038,
"confidence": 0.3433,
"legend": {
"0": "Trivial: any small model answers it reliably",
"1": "Easy: a small model answers it reliably with a little care",
"2": "Moderate: needs a mid-size model",
"3": "Hard: needs one of the strongest models",
"4": "Frontier: even the strongest models are likely to struggle"
},
"probabilities": {
"0": 0.0727,
"1": 0.1784,
"2": 0.3433,
"3": 0.2837,
"4": 0.122
}
},
"stakes": {
"type": "score",
"score": 2.0319,
"confidence": 0.7206,
"legend": {
"0": "Negligible: little or no consequence",
"1": "Low: minor inconvenience or rework",
"2": "Medium: material financial, operational, legal, health or reputational harm is plausible",
"3": "High: serious harm, major losses or disruption is plausible"
},
"probabilities": {
"0": 0.014,
"1": 0.1028,
"2": 0.7206,
"3": 0.1626
}
},
"needs_tools": {
"type": "noul",
"noul": 0.1295
},
"needs_long_context": {
"type": "noul",
"noul": 0.094
},
"personal_data": {
"type": "noul",
"noul": 0.9388
}
}The probabilities associated with each attribute can be used in the smart routing policy, combined with further organisational context.
Below is an example of a chat message handler that implements a routing policy:
MODELS = {
"small": "claude-haiku-4-5",
"medium": "claude-sonnet-5-5",
"large": "claude-opus-5-5", # through the employee's Claude subscription
"large_api": "gpt-5.5", # when their Claude quota is used up
"private": "llama-4-maverick-internal", # in-house inference, for personal data
}
def at_least(answer, level):
"""The probability that a score answer is at `level` or above."""
return sum(p for index, p in answer["probabilities"].items() if int(index) >= level)
def apply_policy(answers, user):
"""The tier a request goes to: the organisation's rules first, then the task's attributes."""
if answers["personal_data"]["noul"] >= 0.2:
return "private"
needs_large = (
at_least(answers["stakes"], 3) >= 0.3 # high stakes
or answers["needs_long_context"]["noul"] >= 0.5
or at_least(answers["difficulty"], 3) >= 0.3 # hard or frontier
)
if needs_large:
# Organisational context: the employee's remaining Claude subscription quota.
return "large" if subscriptions.claude_quota_left(user) > 0 else "large_api"
if at_least(answers["difficulty"], 2) >= 0.3 or answers["needs_tools"]["noul"] >= 0.5:
return "medium"
return "small"
@app.post("/chat")
def chat(request: ChatRequest):
answers = call_eevo({"conversation": request.messages[-6:]}, ROUTING_QUESTIONS)
tier = apply_policy(answers, request.user)
model = MODELS[tier]
log.info("route", extra={"tier": tier, "model": model, "answers": answers})
reply = call_model(model, request.messages)
return {"model": model, "reply": reply}In the example response above, personal_data is 0.94, so this policy sends the conversation to the in-house model, whatever its difficulty.
Route through Opper
Opper's LLM gateway serves models from many providers behind one OpenAI-compatible API, with fallbacks when a model or provider fails. You can route through it in two ways: keep the policy in your own code and send each request to the model it picks, or build the policy as an Opper dynamic route with Eevo as its classifier.
With your own policy code
The routing policy stays in your code: the same call_eevo and apply_policy as in the handler above. Only the last step changes. Instead of calling one model for the tier, the handler sends the request to Opper with the tier's model and its fallbacks, which Opper tries in order:
import os
from openai import OpenAI
opper = OpenAI(base_url="https://api.opper.ai/v3/compat", api_key=os.environ["OPPER_API_KEY"])
# Where each tier of your policy goes through Opper: a model, then its fallbacks.
OPPER_MODELS = {
"small": ["openai/gpt-5-mini", "deepseek-v4.1-flash"],
"medium": ["anthropic/claude-sonnet-4-6", "aws/claude-sonnet-5-5"],
"large": ["openai/gpt-5.5", "gemini/gemini-3.1-pro-preview"],
"large_api": ["openai/gpt-5.5", "gemini/gemini-3.1-pro-preview"],
"private": ["vertexai/claude-sonnet-5-5-eu"], # an EU region, approved for personal data
}
@app.post("/chat")
def chat(request: ChatRequest):
answers = call_eevo({"conversation": request.messages[-6:]}, ROUTING_QUESTIONS) # Eevo assesses the task
tier = apply_policy(answers, request.user) # your policy picks the tier
primary, *fallbacks = OPPER_MODELS[tier] # Opper serves it
reply = opper.chat.completions.create(model=primary, messages=request.messages, extra_body={"models": fallbacks})
log.info("route", extra={"tier": tier, "answers": answers})
return {"tier": tier, "reply": reply}- Keep a fallback inside the tier's rules. The private tier lists only models approved for personal data, so a failure never sends the data elsewhere.
- Set Opper's model access rules as well, such as EU-only or zero-retention providers for the whole organisation. Opper rejects a request that breaks them at the gateway, so they hold even where a policy rule misses.
- Log which model answered, not only the one you asked for. Opper names it in the
X-Opper-Served-Modelresponse header. - Opper's fallback list (
models) works on its Chat Completions endpoint. For other endpoints, see Opper's documentation.
In an Opper dynamic route
An Opper dynamic route is the policy built in Opper itself, as a graph under Models › Routes. A Classifier node sorts each request into one of your labelled cases and takes that case's branch. Choose Eevo as the classifier's model.
Label the cases by what the request is, not by the model it should go to, so the route's branches stay your policy. For example:
| Case | Branch |
|---|---|
| Contains personal data, whatever the task | Models approved for personal data only |
| High stakes: a wrong answer could cause serious harm or loss | Your strongest model |
| Hard: needs one of the strongest models to answer reliably | Your strongest model |
| Routine: a small model answers it reliably | A small, fast model |
| Anything else | A mid-size model |
Then call the route by its name, through the same Opper client:
reply = opper.chat.completions.create(model="dynamic/support", messages=messages)Opper keeps drafts and published versions of a route, lets you simulate it before you publish, and names the route version and the model that answered in each response.
A classifier answers one question: which case a request falls in. Write the cases for the organisation's rules so they claim a request whatever else it is, for example "Contains personal data, whatever the task". When your rules depend on several answers at once, such as personal data and difficulty together, keep the policy in your own code instead.