When you perform a squat in Fiwa AI, your phone can detect whether your knee angle is off, your shoulders are asymmetric, or your spine has lost its neutral position — all within 50 milliseconds. Behind that real-time feedback is a sophisticated AI pipeline. Here's how it works, layer by layer.
Layer 1: Keypoint Detection
Fiwa AI's pose recognition is built on a deep learning model whose core task is locating 33 skeletal keypoints on the human body in each camera frame — including the nose, shoulders, elbows, wrists, hips, knees, and ankles.
The model takes each RGB video frame as input and outputs the (x, y) coordinates of each keypoint along with a confidence score. Points with confidence below a threshold are ignored or interpolated to handle occlusion and blur.
This step runs entirely on-device with no image data uploaded, and latency is kept under 30ms.
Layer 2: Angle Computation
With keypoint coordinates in hand, the system calculates joint angles in real time. For the knee joint:
- Take the coordinates of hip point H, knee point K, and ankle point A
- Compute vectors KH (knee to hip) and KA (knee to ankle)
- Apply the arc-cosine formula:
θ = arccos(KH · KA / |KH| × |KA|)
A standard squat knee angle falls between 90° and 100°. Fiwa AI dynamically adjusts joint thresholds based on the exercise you've selected.
Layer 3: Rep State Machine
Once angles are available, the system needs to determine "what counts as a completed rep." Fiwa AI uses a finite state machine to track movement phases:
- READY: Standing start position, knee angle > 160°
- DOWN: Descending, knee angle steadily decreasing
- BOTTOM: Lowest point reached, knee angle < 100° (squat)
- UP: Rising, knee angle steadily increasing
- COMPLETE: Returned to start — one valid rep counted
State transitions require both an angle threshold and a time window condition, filtering out tremor and false positives.
Layer 4: Form Scoring
After each rep, the system calculates a 0–100 score based on:
- Knee tracking consistency (deviation from toe direction)
- Torso forward-lean angle stability
- Left-right symmetry (angle delta between both sides)
- Movement tempo (descent-to-ascent speed ratio)
- Whether sufficient depth was achieved
Each dimension is weighted and summed. Scores below 80 trigger specific improvement cues.
Why No Video Upload?
The entire pipeline runs locally on your phone: camera → skeleton detection model → angle computation → state machine → scoring. No raw video or image data ever leaves your device. This protects your privacy and means Fiwa AI works fully offline.
What's Next
We're actively researching 3D pose estimation for Fiwa AI — using monocular depth inference to dramatically improve accuracy for lateral movements like side lunges and lateral raises. Stay tuned.