KinesteX

Camera Component

KinesteX Motion Recognition: Real-Time Engagement.


  • Interactive Tracking: Advanced motion recognition for immersive fitness experiences
  • Real-Time Feedback: Instantly track reps, spot mistakes, and calculate calories burned
  • Boost Motivation: Keep users engaged with detailed exercise feedback
  • Custom Integration: Adapt camera placement to fit your app's design

Important — what to pass for `currentExercise` and `exercises`: we recommend exercise IDs — they're stable, human-readable, and you already have them when listing exercises from the Content API. Set exerciseFetchType: "exercise_id" to use them. The Camera Component also accepts model IDs (the default, kept for backward compatibility — but they require an extra round-trip through the Content API to look up) and exercise titles (case-sensitive — convenient for prototyping, but title matching depends on locale-normalization and can mismatch similar exercises, so prefer IDs in production). See the Fetching Exercises section below.


Before showing the camera UI, wait for both model_warmedup and models_loaded events to fire. See the Preloading & Events section below.

Quick Start
Swift example:
1// Model IDs come from the Content API or admin dashboard.
2// See "Camera: Model IDs" subsection.
3@State var currentExercise = "3"
4
5kinestex.createCameraView(
6    exercises: ["3"], // preload every model ID you may switch to
7    currentExercise: $currentExercise,
8    user: nil,
9    isLoading: $isLoading,
10    onMessageReceived: { message in
11        switch message {
12        case .reps(let value):
13            reps = value["value"] as? Int ?? 0
14        case .mistake(let value):
15            mistake = value["value"] as? String ?? "--"
16        default:
17            break
18        }
19    }
20)

Model IDs

Recommendation: prefer exercise IDs — they're easier to get (you already have them when listing exercises from the Content API) and don't require an extra round-trip just to look up a numeric model ID. Set exerciseFetchType: "exercise_id" and pass exercise.id directly. See the Fetching Exercises section.

Model IDs are still supported (it's the default when exerciseFetchType is omitted) and useful when you already have one handy — e.g. from a stored WorkoutModel.sequence or the admin dashboard. They are numeric values like "3" for Squats or "394" for Jumping Jack.


Three ways to get a model ID:


1. Content API — call fetchExercises() (Swift / Kotlin), fetchContent(contentType: ContentType.exercise) (Flutter), or GET /api/v1/exercises (REST). Each ExerciseModel has a model_id field. See the Content API section for full details. (Note: this is the extra round-trip that exercise IDs let you skip.)

2. Admin dashboard — open the exercise in admin.kinestex.com; the model ID is shown at the top of the page header.

3. Workout sequences — when iterating WorkoutModel.sequence, each ExerciseModel entry exposes its own model_id.

Fetch a Model ID
Swift example:
1Task {
2    let result = await kinestex.fetchExercises(limit: 10)
3    if case .success(let response) = result,
4       let exercise = response.exercises.first {
5        // exercise.modelId is what the Camera Component expects
6        currentExercise = exercise.modelId
7    }
8}

Fetching Exercises (by ID or Title)

Besides model IDs, the Camera Component can fetch exercises by exercise ID (recommended) or exercise title. Set the exerciseFetchType parameter to choose the form:


exerciseFetchTypeMeaningWhat goes in exercises / currentExercise
"exercise_id"recommendedExercise IDs from the Content API — easiest to use, no extra round-tripe.g. "squats_v2"
"model_id" (default)Numeric model IDs — kept for backward compatibility; requires a Content API lookup to obtain"3", "394"
"exercise_title"Exercise titles (case-sensitive) — handy for prototyping, but title matching depends on locale-normalization and can mismatch similar exercises"Squats", "Jumping Jack"

Omitting exerciseFetchType keeps the default model-ID behavior — no migration needed for existing integrations.


Where to pass it:

  • React Native (SDK v1.3.1+): directly in postData, alongside exercises and currentExercise.
  • Swift, Kotlin, Flutter, HTML/JS, React (TypeScript): inside customParams / customParameters along with exercises and currentExercise.

Keep one form per session: use the same form for both exercises and currentExercise, and for any later switches.

Fetch by Exercise Title
Swift example:
1// exerciseFetchType goes inside customParams
2kinestex.createCameraView(
3    exercises: ["Squats", "Jumping Jack"],
4    currentExercise: $currentExercise, // e.g. "Squats"
5    customParams: [
6        "exerciseFetchType": "exercise_title" // "model_id" (default) | "exercise_id" | "exercise_title"
7    ]
8)

Preloading & Events

Two events fire as the component initializes. Wait for both before revealing the camera UI to the user.


EventMeaning
model_warmedupPose-tracking (MediaPipe) model is ready
models_loadedAll exercise models in exercises have downloaded

Pattern: mount the camera hidden (e.g. opacity: 0) with a loader on top. Reveal once both events have fired.

Wait for Both Events
Swift example:
1@State private var modelWarmedUp = false
2@State private var modelsLoaded = false
3var isReady: Bool { modelWarmedUp && modelsLoaded }
4
5kinestex.createCameraView(
6    exercises: ["3"],
7    currentExercise: $currentExercise,
8    user: nil,
9    isLoading: $isLoading,
10    onMessageReceived: { message in
11        if case .custom_type(let value) = message,
12           let type = value["type"] as? String {
13            if type == "model_warmedup" { modelWarmedUp = true }
14            if type == "models_loaded"  { modelsLoaded  = true }
15        }
16    }
17)
18.opacity(isReady ? 1 : 0)

Controls

Switching exercises in real time. Set currentExercise to any model ID from the exercises array — the camera component swaps tracking immediately.


Control commands. Send any of these strings as currentExercise to control the session:


CommandEffect
"Pause Exercise"Pauses motion tracking; rep counter freezes
"Pause Audio"Mutes voice feedback
"Resume Audio"Re-enables voice feedback
"Workout Overview"Triggers a summary snapshot for the current session
"Stop Camera"⚠️ Destructive — releases camera + models, fires stop_camera. Not recoverable without re-mounting the component (see warning below)

To resume tracking after a pause, set currentExercise back to a real model ID from exercises.


⚠️ `"Stop Camera"` is destructive — avoid it for normal flows. It tears down the camera, MediaPipe, and all loaded exercise models, then fires the stop_camera event. You cannot recover from it without unmounting and re-creating the entire camera component. Only send it when the user is permanently leaving the camera screen. For temporary pauses, use "Pause Exercise" instead.
Switch Exercise and Send Control Commands
Swift example:
1// Switch exercise (Swift uses two-way binding via @State)
2currentExercise = "394"
3
4// Pause / resume tracking
5currentExercise = "Pause Exercise"
6currentExercise = "3"

Customization

Pass any of these fields in customParams (Swift / Kotlin / Flutter) or directly in postData (HTML/JS, React Native, React TS) at initialization.


FieldTypeDescription
restSpeechesstring[]Audio phrases to preload (from ExerciseModel.rest_speech)
videoURLstringUse a video file instead of the live camera
landmarkColorstringPose overlay color in hex with # (default #14FF00)
showSilhouettebooleanShow "get into frame" silhouette (default true)
includeRealtimeAccuracyboolean(Beta) Stream live position-confidence alongside reps
includePoseDatastring[]Any of "angles", "poseLandmarks", "worldLandmarks". Performance impact — only enable for custom calculations

Event Reference

All events the Camera Component emits to the host app:


EventPayloadWhen
model_warmedup{ message }Pose model is ready
models_loaded{ message }All exercise models in exercises finished downloading
person_in_frame{ message }User entered the silhouette frame
successful_repeat{ exercise, value, accuracy }A rep was counted (value = total reps so far)
mistake{ value }Form mistake detected
correct_position_accuracy{ accuracy }(Beta) Live position confidence — only when includeRealtimeAccuracy: true
pose_landmarks{ poseLandmarks }Per-frame screen-space landmarks — only when includePoseData includes "poseLandmarks"
world_landmarks{ worldLandmarks }Per-frame world-space landmarks — only when includePoseData includes "worldLandmarks"
speech_fetch_complete{ successCount, failureCount }All restSpeeches finished loading
error_occurred{ message } or { data, error }Any error (model fetch fail, phrase fail, etc.)
warning{ data }Non-fatal config issue (e.g. no model IDs provided)
stop_camera{ message }Confirms the "Stop Camera" command finished

Pose Data

When includePoseData contains "poseLandmarks" or "worldLandmarks", the camera emits per-frame events with raw skeleton data. Only enable this if you're doing custom calculations — there is a performance cost.


Two coordinate spaces:


  • poseLandmarks — values 0–1, normalized to the camera frame.
  • worldLandmarks — meters, relative to the hips (best Z accuracy).

Each landmark has { x, y, z, visibility } (all 0–1).


Available landmarks (same names in both spaces): nose, leftEyeInner, leftEye, leftEyeOuter, rightEyeInner, rightEye, rightEyeOuter, leftEar, rightEar, mouthLeft, mouthRight, leftShoulder, rightShoulder, leftElbow, rightElbow, leftWrist, rightWrist, leftPinky, rightPinky, leftIndex, rightIndex, leftThumb, rightThumb, leftHip, rightHip, leftKnee, rightKnee, leftAnkle, rightAnkle, leftHeel, rightHeel, leftFootIndex, rightFootIndex.


Available angles (when "angles" is included — both 2D and 3D versions are emitted): leftKneeAngle, rightKneeAngle, leftHipAngle, rightHipAngle, leftShoulderAngle, rightShoulderAngle, leftElbowAngle, rightElbowAngle, leftWristAngle, rightWristAngle, leftAnkleAngle, rightAnkleAngle, leftArmpitAngle, rightArmpitAngle.

Complete Example

Minimal working example with Next / Previous buttons that cycle between exercises and a live rep counter in the UI.

Complete Implementation
Swift example:
1import SwiftUI
2import KinesteXAIKit
3
4struct CameraScreen: View {
5    let kinestex = KinesteXAIKit(
6        apiKey: "YOUR_API_KEY",
7        companyName: "YOUR_COMPANY_NAME",
8        userId: "YOUR_USER_ID"
9    )
10
11    // 3 = Squats, 394 = Jumping Jack
12    let exerciseIds = ["3", "394"]
13
14    @State private var index = 0
15    @State private var currentExercise = "3"
16    @State private var reps = 0
17    @State private var isLoading = false
18
19    var body: some View {
20        VStack(spacing: 16) {
21            Text("Reps: \(reps)")
22                .font(.title)
23                .padding(.top)
24
25            kinestex.createCameraView(
26                exercises: exerciseIds,
27                currentExercise: $currentExercise,
28                user: nil,
29                isLoading: $isLoading,
30                onMessageReceived: { message in
31                    if case .reps(let value) = message {
32                        reps = value["value"] as? Int ?? 0
33                    }
34                }
35            )
36
37            HStack(spacing: 24) {
38                Button("Previous") { switchTo(index - 1) }
39                Button("Next")     { switchTo(index + 1) }
40            }
41            .padding(.bottom)
42        }
43    }
44
45    private func switchTo(_ newIndex: Int) {
46        index = (newIndex + exerciseIds.count) % exerciseIds.count
47        currentExercise = exerciseIds[index]
48        reps = 0
49    }
50}