Zing Forum

Reading

Squat-Coach: Edge AI Fitness Coach with Local Visual Inference for Real-Time Squat Guidance

Squat-Coach is an innovative edge AI project that combines MediaPipe pose estimation and the local Gemma model to deliver fully offline real-time squat guidance. Using a teacher-student training paradigm, it generates training data via Gemini and fine-tunes the local model, providing professional fitness feedback while protecting privacy.

端侧AI姿态估计MediaPipeGemma微调健身教练隐私保护教师学生训练计算机视觉生物力学本地推理
Published 2026-05-14 16:13Recent activity 2026-05-14 16:19Estimated read 5 min
Squat-Coach: Edge AI Fitness Coach with Local Visual Inference for Real-Time Squat Guidance
1

Section 01

【Introduction】Squat-Coach: Edge AI Enables Offline Real-Time Squat Guidance

Squat-Coach is an innovative edge AI project that combines MediaPipe pose estimation and the local Gemma model to deliver fully offline real-time squat guidance. Using a teacher-student training paradigm, it generates training data via Gemini to fine-tune the local model, providing professional fitness feedback while protecting privacy.

2

Section 02

Project Background & Vision: Addressing Pain Points in Professional Fitness Guidance

In the fitness field, correct posture is crucial for training effectiveness and safety, but professional coaches are expensive and cannot be available at all times. Squat-Coach aims to provide an always-accessible, fully private virtual fitness coach using edge AI technology. Its core vision is to create a coach agent with visual input, edge inference, real-time feedback, and intelligent decision-making capabilities—not just a simple overlay of AI text on pose data.

3

Section 03

System Architecture: Layered Design for Intelligent Guidance

Adopting a layered architecture:

  • Visual Perception Layer: MediaPipe Pose Lite (runs in browser via WASM/WebGL, 4MB model, extracts 3D key points)
  • Biomechanical Interpretation Layer: Calculates angles of knees/hips/torso/shins, identifies squat phases (standing/squatting/bottom/rising)
  • Intelligent Coach Layer: Fine-tuned Gemma model as the 'brain' to output guidance suggestions
  • Agent Decision Loop: Setup (check framing), Calibration (personalized thresholds), Real-Time Guidance, Inter-Set Summary
4

Section 04

Privacy First: Fully Offline Operation

The core selling point is that all processing is done locally—workout videos never leave the device, and it works even without an internet connection. Dependencies (MediaPipe WASM, JS packages, pose model) are downloaded once via the setup.sh script, with no network requests during runtime, ensuring privacy and reliability.

5

Section 05

Training Method: Teacher-Student Paradigm

Innovative training method:

  • Teacher Model: Cloud-based Gemini analyzes real squat videos + rules to generate annotated datasets
  • Student Model: Gemma is fine-tuned with the dataset to learn to generate feedback based on pose data
  • Data Quality: Emphasizes the importance of real data; recommends collecting standard side views and error actions (shallow squats/forward lean/knee valgus, etc.) to cover scenarios.
6

Section 06

Technical Details: Input, Output & Configuration

Technical implementation:

  • Input Features: 7 core features (phase/rep_count/knee_angle/hip_angle/torso_angle/shin_angle/hip_below_knee)
  • Output Specification: JSON format (say: guidance text; priority: priority level; ui: UI instructions; cooldown_s: cooldown time)
  • Runtime Configuration: Sampling parameters (temperature/top_p/top_k), repetition penalty, maximum length, random seed, etc.
7

Section 07

Application Value & Future Outlook

Application Value:

  • Provides users with private, low-cost professional guidance—only requires a regular computer + camera
  • Offers developers a reference solution for edge AI integration (large model + computer vision)

Project Status: Core functions (local pose detection/rep counting/angle calculation, etc.) are implemented; Gemma fine-tuning and dataset auditing are ongoing Future Directions: Support more exercise types, personalized training plans, integration with fitness devices.