Memory-Efficient Training of 1B-Parameter Models on 6 GB Consumer GPUs

We demonstrate that a 1.07-billion-parameter language model can be trained—not merely loaded for inference—on a single 6 GB consumer GPU (RTX 4050 laptop), consuming 4.05 GB of VRAM with 1.9 GB to spare at 1,111 tokens/sec. 

This is achieved by composing five published techniques: block-coordinate descent (BAdam), CPU weight offload, ternary weights (BitNet b1.58), tied input/output embeddings, and gradient checkpointing. No technique is novel; the contributions are:

1. A working integration resolving a previously unhandled conflict between tied embeddings and CPU offload that caused 13 subsystem tests to be silently skipped.
2. The empirical finding that cross-entropy loss computation—not model weights or activations—is the dominant memory consumer for small models at long context, with chunking recovering 1,436 MB at 7% step-time cost.
3. The observation that perplexity 4.13 can coincide with 0% task accuracy, with a data-recipe fix raising first-token accuracy from 2.2% to 59.8% on identical compute.
4. A published record of 11 failed experiments, 5 self-retracted conclusions, and the methodology rule that produced them.

The practical training ceiling on this hardware is a few hundred million parameters; the memory ceiling and training ceiling are different numbers. All figures are measured on the described hardware. Code is released under Apache 2.0.

Project Repository: https://gitlab.com/komalbarun/kramba-ai