Helios

Overview

Built this to understand continuous batching properly. Supports priority-based scheduling, prefix caching, and has a terminal dashboard for monitoring request queues and GPU utilization in real time. Most inference servers are either closed-source or designed for Google-scale — this one is meant to be readable.

Technologies

• python
• cuda
• async io
• tui