Reverse Engineering Lab - Data Collection Platform

v0.2.0 - 2026-04

Description

Major expansion of the platform: reworked authentication, overhauled frontend-app and frontend-web, Raspberry Pi camera live streaming, observability, and a substantially hardened CI/CD pipeline.

Features

Authentication and Access Control

• Cookie- and refresh-token-based auth replacing session-only flow
• Custom OAuth router with frontend redirects (Google, GitHub)
• Login with username or email, superuser username support
• Rate limiting on login/register with dev/test bypass
• Last-login and IP tracking, email masking in logs
• YouTube OAuth association, toggle, and token cleanup on unlink

Backend API

• Full-text (tsvector) product search endpoint
• Pagination, order_by, created_at/updated_at filters on list routes
• Circularity properties on products; weight unit moved from kg to g
• Bounded recursive loading for category and product-tree endpoints
• Image processing pipeline: resize endpoint, product thumbnails, preview-thumbnail URLs with mtime cache-busting
• File cleanup service and script; file storage path bootstrapping
• User preferences field; product ownership and visibility controls
• Healthcheck endpoints
• Video patching and YouTube video ingestion via link

Raspberry Pi Camera Integration

• WebSocket pairing, management, and image capture UI
• Cross-worker WebSocket command relay
• LL-HLS proxy and telemetry endpoints
• YouTube live streaming integration
• Local access info retrieval and self-unpairing
• Background unpair notifications on camera delete
• Simplified rpi-cam plugin; dev mock script

Frontend App

• UI overhaul, ~1500 lint fixes, refactored test suite
• React Hook Form + Zod resolver across all auth forms
• Camera connection and capture hooks, streaming components
• Product detail, navigation, and new-product-reset fixes
• Cross-browser E2E tagging and expanded test coverage

Frontend Web

• Migrated from Expo to Astro
• Styling aligned with the app; newsletter and token-action forms
• Privacy policy page; E2E test suite

Observability and Operations

• OpenTelemetry integration with OTLP log/trace export and headers
• Migrated backend logging to Loguru
• Production and staging Docker Compose configurations
• Caddyfile hardening: CSP, static asset handling, SPA routing
• Manual Postgres backup script and rclone sync with stats

Email and Caching

• Migrated to fastapi-mail with MJML templates
• Redis-backed disposable-mail cache and refresh-token storage
• In-memory refresh-token fallback for Redis-less dev

Documentation

• Migrated from mkdocs-material to Astro
• Footer with copyright and social links; 404 fix

Developer Experience and CI/CD

• just task runner across the monorepo; migrated to pnpm
• Release-please replaces commitizen for version management
• Composite GitHub Actions: runtime setup, change detection, security change detection, Codecov upload
• Full-stack cross-browser E2E workflow; frontend-web E2E job
• Container security matrix, .trivyignore allowlist, gitleaks
• cspell spellcheck and pre-commit caching
• Devcontainers per service (backend, frontend-app, frontend-web, docs)
• Moved type checking from pyright to ty; Python 3.14
• SecretStr for secret env vars in core config

Testing

• Broad backend unit-test suite (auth, OAuth, relay, encryption, background data, organizations, newsletter preferences)
• Frontend-app and frontend-web test coverage expansion