Skip to content
IEEE white paper · HSI 2026 · Jul 17, 2026

Adaptive Inference Orchestration for Enterprise AI Assistants: Tiered Routing and Conditional Execution

18th IEEE International Conference on Human System Interaction · Sarajevo, presented remotely
In one minute
01The problem

Enterprise AI assistants run many agents, and background agents wake every 15–60 minutes. Sending every call to the largest model is slow and expensive, and most of those checks find nothing new.

02The idea

Check cheaply for anything new before any model runs. When there is work, route it to the smallest model tier that can do it well.

03The result

Over three weeks in production, 78.5% of checks stopped at the gate with zero tokens spent. With routing, a projected 90.9% total cost reduction, with task completion above 95%.

Results

What the deployment showed

0.0%fewer unnecessary inference cycles, from conditional gatingmeasured

2,988 of 3,805 scheduled checks found nothing new and stopped before any model call, over three weeks of production.

0.0%total cost reduction with tiered routingprojected

Gates plus routing, projected from the observed task mix (30% light, 40% medium, 31% heavy) and published model-class ratios. The live system ran every triggered task on the largest model.

>0%task completion accuracy maintainedmeasured

81 of 100 scheduled runs completed; 15 of the 19 misses were infrastructure errors such as expired tokens and rate limits. Excluding those, 95.3%. No run took an incorrect action.

0 weeksin a live enterprise deploymentmeasured

Running continuously in production, connected to Slack, email and calendar, and retrieval systems.

How it works

Spend the expensive model only where it matters

Routing classifierreads structure, history and agent defaults, then picks a model tier · runs onFast· adds <100 ms
Background agentsFastBalanced
Every 15–60 minutes: messages, email, calendar
Gate 1, no model: anything new since the last watermark? If not, stop. No tokens spent.Gate 2, fast model: is the new data relevant? Only then does the agent run.78.5% of cycles skipped
Primary conversational agentBalancedSmart
Talks with the user, synchronously, low latencySpawns task agents for long work and picks up their results when they finish.
On-demand task agentsSmart
Research, analysis, document generation; asynchronousLong work that can wait, so it gets the largest model without slowing the conversation; results return to the primary agent.
Shared integration layer

Messaging · email and calendar · knowledge graph and search · files · tasks, each behind permission-bounded tools; a shared workspace passes state between agents without re-reading the same data.

Fast≤8B params · <1 s · 1× costBalanced30–70B · 2–5 s · 5–10×Smart>100B · 5–15 s · 20–50×
Figure 1A fast classifier routes each task to the smallest model tier that can do it; background agents pass two gates before any model runs.
Explore the numbers

Three weeks of checks, and what each idea saves

78.5%of checks stopped at the gate
Something new arrived: the agent ran (817) Nothing new: stopped before any model call, zero tokens (2,988)Each dot is five scheduled checks, in time order · measured over three weeks

Observed task mix: 30% light, 40% medium, 31% heavy. With mostly heavy work (60%), the paper estimates routing alone would save about 35%; the gates save the same either way.

Gates stop checks with nothing newMeasured
21.5%of the original inference cost remains · 78.5% saved
Abstract

The proliferation of Large Language Model (LLM)-powered personal assistants in enterprise environments introduces significant challenges in balancing response quality, latency, and inference cost, particularly in agentic workflows where multiple collaborating agents trigger cascading model calls. This paper presents an adaptive inference orchestration archit…

Read the full abstract
Index terms · Inference optimization · Multi-agent systems · Model routing · Enterprise AI · Agentic AI
Details
Conference
HSI 2026, Sarajevo, presented remotely
Session
Hybrid session 6: Machine Learning
Presented
Jul 17, 2026
Paper ID
HSI26-000077
Cite

G. R. Ramadurai, “Adaptive Inference Orchestration for Enterprise AI Assistants: Tiered Routing and Conditional Execution,” in Proc. 18th IEEE International Conference on Human System Interaction (HSI 2026), Sarajevo, 2026.

Full paper

Awaiting publication in the IEEE proceedings. It will be listed on my Google Scholar profile once indexed.

Google Scholar profile ↗