Published April 1, 2026 | Version 0.1.1

llmsays: One-Line LLM Inference with Automatic Prompt-Tier Routing and Multi-Provider Failover

  • 1. ROR icon Haldia Institute of Technology

Description

llmsays is an open-source Python library that reduces large language model (LLM) inference to a single function call. It introduces a two-stage hybrid routing mechanism that classifies an input prompt into one of four
complexity tiers — small, medium, large, and extra_large — using a lightweight sentence-transformer model,
and subsequently dispatches the query to the most appropriate provider-specific model. The library supports
five commercial inference providers (Groq, NVIDIA NIM, OpenRouter, Fireworks AI, and Baseten) with automatic latency-aware failover, eliminating manual provider management for researchers and practitioners alike.
An optional multiprocessing mode enables parallel provider queries, returning the first successful response for
latency-critical applications.

Files

llmsays.pdf

Files (380.2 kB)

Name Size
md5:dde6164eeb116768836221011958dbae
380.2 kB Preview Download

Additional details

Dates

Created
2026-04-01

Software

Repository URL
https://github.com/abhirajadhikary06/llmsays
Programming language
Python
Development Status
Active