Project VELOCITY: Overcoming the PCIe Bottleneck via Quad-Buffered Asynchronous Pipelining
Authors/Creators
Description
The VELOCITY architecture implements a Quad-Buffered Ring Pipeline that leverages the multi-channel nature of modern system RAM to eliminate "Stop-and-Go" latency during data ingestion. By implementing a 4-zone rotary buffer (Ingest $\rightarrow$ Tokenize $\rightarrow$ Pin/Feed $\rightarrow$ Writeback), the system ensures 100% saturation of the PCIe bus, effectively treating System RAM as a high-speed L4 cache for the GPU.
Phase 2: The "Infinite Model" Capability
To enable the training of models exceeding physical VRAM capacity, VELOCITY introduces a Layer-Wise Swapper ("The Teleporter") utilizing a sequential V-Cycle Architecture:
-
Downstream (Forward): Layers are streamed $0 \to N$ from System RAM to GPU. Activations are computed and offloaded back to System RAM.
-
Upstream (Backward): Layers are streamed $N \to 0$. Stored activations are retrieved, and gradients are computed and optimized in sequence.
Empirical Validation
We benchmarked the architecture on consumer hardware (NVIDIA RTX 4070 Ti, 128GB DDR5 RAM) using JAX with Pinned Memory optimization. The results confirm the effective removal of the I/O bottleneck:
-
Pipeline Saturation: The architecture sustained a synthetic data throughput of 37.5 Million tokens/second, proving that the Ring Buffer completely hides SSD and RAM latency.
-
Real-World Fine-Tuning (Simulation): In a simulation of a Llama-3 8B Decoder Layer (FP32 precision with Gradient Checkpointing), the system achieved a sustained training speed of $\approx$ 7,100 tokens/second.
-
Full 8B Backpropagation (Real Weights): We executed the full V-Cycle on the Meta Llama-3 8B model (32GB FP32 payload) on a 12GB GPU. The system successfully performed forward and backward passes for all 32 layers with an average step time of $\approx$ 101 seconds (Forward: 0.94s/layer, Backward: 1.95s/layer).
Implication: 70B Feasibility
Based on the linear scaling observed in the 8B benchmarks, we project the training time for a 70B Parameter Model (80 layers) on the same hardware to be $\approx$ 4 minutes per step. These results demonstrate that the bottleneck has been successfully shifted from I/O (Bandwidth) to Compute (FLOPs), confirming that full-parameter fine-tuning of State-of-the-Art models is feasible on consumer hardware without quantization.
Key Features
-
Ring Buffer Pipelining: Utilizes asynchronous Direct Memory Access (DMA) with Pinned Memory (Page-Locked RAM).
-
Quantization Multiplier: Integrates 4-bit (Int4) quantization logic for the transport layer to maximize bus data density.
-
Zero-Copy Loading: Eliminates redundant CPU memory moves during tensor creation.
Relation to Previous Work
This proposal extends the "Infinite Context" capabilities established in the UFCE Framework by addressing the specific constraints of model plasticity (training) rather than static retrieval.
Code Availability
The complete source code, JAX kernels, and reproduction scripts are available in the linked GitHub repository.
References:
[1] Killian, K. (2025). The UniField Coupling Equation (UFCE): A Universal Framework for Modeling Coupled Spatial-Temporal Dynamics. Zenodo. https://doi.org/10.5281/zenodo.18055873
Files
Project VELOCITY Overcoming the PCIe Bottleneck via Quad-Buffered Asynchronous Pipelining v2.pdf
Additional details
Related works
- Is supplement to
- Preprint: 10.5281/zenodo.18055873 (DOI)