Orbit: Efficient Agentic Inference using Priority Scheduling
Large language models (LLMs) are increasingly deployed as autonomous agents that execute complex tasks through long sequences of reasoning steps and tool calls. Serving these agentic workloads at scale is a growing priority, yet existing LLM inference engines are mostly designed for single-pass, chat-style interactions. Agentic tasks, however, can fail, and under system congestion, policies such as first come first serve (FCFS) allocate GPU resources equally to all tasks regardless of their likelihood of success, letting failing tasks inflate queue waiting times for tasks that will complete correctly. We introduce Orbit, a trajectory-aware scheduler that addresses this by dynamically deprioritizing tasks predicted to fail, shifting resources toward likely-correct tasks. We present two scheduling algorithms. Orbit-step is a lightweight heuristic that assigns priority inversely proportional to the number of steps a task has taken, leveraging the empirical observation that longer-running tasks are more likely to fail. Orbit-judge uses an asynchronous LLM-as-a-judge that reads partial execution trajectories and predicts task correctness online, enabling finer-grained prioritization. We evaluate both variants on the GAIA benchmark using the Magentic-One benchmark and the GPT-OSS-120B model. Compared to vLLM's default FCFS policy, Orbit-step and Orbitjudge show 11.2% and 4.3% decrease in average end-to-end latency, respectively, for correct tasks. These results demonstrate that realtime trajectory observation is a practical and effective signal for resource allocation in agentic serving systems. CCS Concepts • Computing methodologies → Artificial intelligence.
Cite this item
Abuzakuk, S., Kermarrec, A.-M., Palak, Pires, R., Sharma, R., & de Vos, M. (2026). Orbit: Efficient Agentic Inference using Priority Scheduling. EuroMLSys ’26: Proceedings of the Sixth European Workshop on Machine Learning and Systems, 386–396. https://doi.org/10.1145/3805621.3807661
Click Copy to copy the citation, or select the text manually.
3805621.3807661.pdf
Main Document
Published version
openaccess
CC BY
993.46 KB
Adobe PDF
442523b5942540dc1744a95a980cd231