llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover
Authors/Creators
Contributors
Project member (3):
Description
llmsays is an open-source Python library that reduces large language model (LLM) inference to a single function call. It introduces a two-stage hybrid routing mechanism that classifies an input prompt into one of four
complexity tiers — small, medium, large, and extra_large — using a lightweight sentence-transformer model,
and subsequently dispatches the query to the most appropriate provider-specific model. The library supports
five commercial inference providers (Groq, NVIDIA NIM, OpenRouter, Fireworks AI, and Baseten) with automatic latency-aware failover, eliminating manual provider management for researchers and practitioners alike.
An optional multiprocessing mode enables parallel provider queries, returning the first successful response for
latency-critical applications.
Files
llmsays.pdf
Additional details
Dates
- Created
-
2026-04-01
Software
- Repository URL
- https://github.com/abhirajadhikary06/llmsays
- Programming language
- Python
- Development Status
- Active