Adaptive Inference Orchestration for Enterprise AI Assistants: Tiered Routing and Conditional Execution
18th IEEE International Conference on Human System Interaction · Sarajevo, presented remotelyEnterprise AI assistants run many agents, and background agents wake every 15–60 minutes. Sending every call to the largest model is slow and expensive, and most of those checks find nothing new.
Check cheaply for anything new before any model runs. When there is work, route it to the smallest model tier that can do it well.
Over three weeks in production, 78.5% of checks stopped at the gate with zero tokens spent. With routing, a projected 90.9% total cost reduction, with task completion above 95%.
What the deployment showed
2,988 of 3,805 scheduled checks found nothing new and stopped before any model call, over three weeks of production.
Gates plus routing, projected from the observed task mix (30% light, 40% medium, 31% heavy) and published model-class ratios. The live system ran every triggered task on the largest model.
81 of 100 scheduled runs completed; 15 of the 19 misses were infrastructure errors such as expired tokens and rate limits. Excluding those, 95.3%. No run took an incorrect action.
Running continuously in production, connected to Slack, email and calendar, and retrieval systems.
Spend the expensive model only where it matters
Messaging · email and calendar · knowledge graph and search · files · tasks, each behind permission-bounded tools; a shared workspace passes state between agents without re-reading the same data.
Three weeks of checks, and what each idea saves
Observed task mix: 30% light, 40% medium, 31% heavy. With mostly heavy work (60%), the paper estimates routing alone would save about 35%; the gates save the same either way.
The proliferation of Large Language Model (LLM)-powered personal assistants in enterprise environments introduces significant challenges in balancing response quality, latency, and inference cost, particularly in agentic workflows where multiple collaborating agents trigger cascading model calls. This paper presents an adaptive inference orchestration archit… ecture for enterprise AI assistants that employs three key optimization strategies: (1) tiered model routing, which dynamically selects model complexity based on task classification; (2) trigger-based conditional execution, which uses lightweight deterministic gates to prevent unnecessary LLM invocations when no actionable context exists; and (3) hierarchical agent decomposition, which distributes workloads across specialized sub-agents with appropriate model tiers rather than routing all tasks through a single high-cost model. We evaluate the architecture in a real-world enterprise deployment over three weeks, integrating with Slack messaging, Microsoft Outlook email and calendar, and knowledge retrieval systems. Results demonstrate a 78.5% reduction in unnecessary inference cycles through conditional gating and a projected 90.9% total cost reduction when combined with tiered routing, while maintaining task completion accuracy above 95%. Our findings suggest that inference-aware orchestration design is essential for sustainable deployment of multi-agent AI systems in interactive enterprise contexts.
Read the full abstract
- Conference
- HSI 2026, Sarajevo, presented remotely
- Session
- Hybrid session 6: Machine Learning
- Presented
- Jul 17, 2026
- Paper ID
- HSI26-000077
G. R. Ramadurai, “Adaptive Inference Orchestration for Enterprise AI Assistants: Tiered Routing and Conditional Execution,” in Proc. 18th IEEE International Conference on Human System Interaction (HSI 2026), Sarajevo, 2026.
Awaiting publication in the IEEE proceedings. It will be listed on my Google Scholar profile once indexed.
Google Scholar profile ↗