Camera Component
KinesteX Motion Recognition: Real-Time Engagement.
- Interactive Tracking: Advanced motion recognition for immersive fitness experiences
- Real-Time Feedback: Instantly track reps, spot mistakes, and calculate calories burned
- Boost Motivation: Keep users engaged with detailed exercise feedback
- Custom Integration: Adapt camera placement to fit your app's design
Important — what to pass for `currentExercise` and `exercises`: we recommend exercise IDs — they're stable, human-readable, and you already have them when listing exercises from the Content API. Set exerciseFetchType: "exercise_id" to use them. The Camera Component also accepts model IDs (the default, kept for backward compatibility — but they require an extra round-trip through the Content API to look up) and exercise titles (case-sensitive — convenient for prototyping, but title matching depends on locale-normalization and can mismatch similar exercises, so prefer IDs in production). See the Fetching Exercises section below.
Before showing the camera UI, wait for both model_warmedup and models_loaded events to fire. See the Preloading & Events section below.
1// Model IDs come from the Content API or admin dashboard.
2// See "Camera: Model IDs" subsection.
3@State var currentExercise = "3"
4
5kinestex.createCameraView(
6 exercises: ["3"], // preload every model ID you may switch to
7 currentExercise: $currentExercise,
8 user: nil,
9 isLoading: $isLoading,
10 onMessageReceived: { message in
11 switch message {
12 case .reps(let value):
13 reps = value["value"] as? Int ?? 0
14 case .mistake(let value):
15 mistake = value["value"] as? String ?? "--"
16 default:
17 break
18 }
19 }
20)Model IDs
exerciseFetchType: "exercise_id" and pass exercise.id directly. See the Fetching Exercises section.Model IDs are still supported (it's the default when exerciseFetchType is omitted) and useful when you already have one handy — e.g. from a stored WorkoutModel.sequence or the admin dashboard. They are numeric values like "3" for Squats or "394" for Jumping Jack.
Three ways to get a model ID:
1. Content API — call fetchExercises() (Swift / Kotlin), fetchContent(contentType: ContentType.exercise) (Flutter), or GET /api/v1/exercises (REST). Each ExerciseModel has a model_id field. See the Content API section for full details. (Note: this is the extra round-trip that exercise IDs let you skip.)
2. Admin dashboard — open the exercise in admin.kinestex.com; the model ID is shown at the top of the page header.
3. Workout sequences — when iterating WorkoutModel.sequence, each ExerciseModel entry exposes its own model_id.
1Task {
2 let result = await kinestex.fetchExercises(limit: 10)
3 if case .success(let response) = result,
4 let exercise = response.exercises.first {
5 // exercise.modelId is what the Camera Component expects
6 currentExercise = exercise.modelId
7 }
8}Fetching Exercises (by ID or Title)
Besides model IDs, the Camera Component can fetch exercises by exercise ID (recommended) or exercise title. Set the exerciseFetchType parameter to choose the form:
exerciseFetchType | Meaning | What goes in exercises / currentExercise |
"exercise_id" ✅ recommended | Exercise IDs from the Content API — easiest to use, no extra round-trip | e.g. "squats_v2" |
"model_id" (default) | Numeric model IDs — kept for backward compatibility; requires a Content API lookup to obtain | "3", "394" |
"exercise_title" | Exercise titles (case-sensitive) — handy for prototyping, but title matching depends on locale-normalization and can mismatch similar exercises | "Squats", "Jumping Jack" |
Omitting exerciseFetchType keeps the default model-ID behavior — no migration needed for existing integrations.
Where to pass it:
- React Native (SDK v1.3.1+): directly in
postData, alongsideexercisesandcurrentExercise. - Swift, Kotlin, Flutter, HTML/JS, React (TypeScript): inside
customParams/customParametersalong withexercisesandcurrentExercise.
Keep one form per session: use the same form for both exercises and currentExercise, and for any later switches.
1// exerciseFetchType goes inside customParams
2kinestex.createCameraView(
3 exercises: ["Squats", "Jumping Jack"],
4 currentExercise: $currentExercise, // e.g. "Squats"
5 customParams: [
6 "exerciseFetchType": "exercise_title" // "model_id" (default) | "exercise_id" | "exercise_title"
7 ]
8)Loading More Exercises at Runtime (React Native v1.3.1+)
Availability: this walkthrough uses the React Native SDK v1.3.1+ helper. HTML/JS hosts can post the same { workout_activity_action: "load_models", exercises: [...] } message to the iframe directly, read modelIds from the parsed models_loaded message, and switch by posting { currentExercise: "..." }. Other SDKs have no helper yet, so pass the full exercises list at initialization.
In React Native you can fetch and cache additional exercise models after the session has started, without re-mounting the camera. This is done by calling sendAction with the "load_models" action and an extras object containing the new identifiers.
Three-step flow:
1. Send the load_models action with the identifiers you want to add.
2. Wait for the models_loaded event — data.modelIds echoes the identifiers that resolved.
3. Switch the active exercise with changeExercise(...) (do not repeat exerciseFetchType).
If exercises is missing or empty, the SDK posts back { "type": "error_occurred", "severity": "warning", "message": "load_models: no exercises provided" }. If some identifiers fail, you get one error_occurred with severity: "critical" and a message listing them all ("Failed to fetch 2 model(s): a, b"). Identifiers that did resolve are still listed in modelIds. If every identifier fails, no models_loaded is sent.
1// Step 1 — fetch additional models at runtime
2sdkRef.current?.sendAction(
3 "workout_activity_action",
4 "load_models",
5 {
6 exercises: ["Jumping Jack", "Lunges"],
7 exerciseFetchType: "exercise_title", // match the form you used initially
8 }
9);
10
11// Step 2 — wait for models_loaded, then switch
12const handleMessage = (type: string, data: { [key: string]: any }) => {
13 if (type === "models_loaded" && data.modelIds?.includes("Jumping Jack")) {
14 // Step 3 — switch the active exercise. Do NOT repeat exerciseFetchType here.
15 sdkRef.current?.changeExercise("Jumping Jack");
16 }
17 if (type === "error_occurred") {
18 console.warn("KinesteX error:", data.message);
19 }
20};Preloading & Events
Two events fire as the component initializes. Wait for both before revealing the camera UI to the user.
| Event | Meaning |
model_warmedup | Pose-tracking (MediaPipe) model is ready |
models_loaded | All exercise models in exercises have downloaded |
Pattern: mount the camera hidden (e.g. opacity: 0) with a loader on top. Reveal once both events have fired.
1@State private var modelWarmedUp = false
2@State private var modelsLoaded = false
3var isReady: Bool { modelWarmedUp && modelsLoaded }
4
5kinestex.createCameraView(
6 exercises: ["3"],
7 currentExercise: $currentExercise,
8 user: nil,
9 isLoading: $isLoading,
10 onMessageReceived: { message in
11 if case .custom_type(let value) = message,
12 let type = value["type"] as? String {
13 if type == "model_warmedup" { modelWarmedUp = true }
14 if type == "models_loaded" { modelsLoaded = true }
15 }
16 }
17)
18.opacity(isReady ? 1 : 0)Controls
Switching exercises in real time. Set currentExercise to any model ID from the exercises array — the camera component swaps tracking immediately.
Control commands. Send any of these strings as currentExercise to control the session:
| Command | Effect |
"Pause Exercise" | Pauses motion tracking; rep counter freezes |
"Pause Audio" | Mutes voice feedback |
"Resume Audio" | Re-enables voice feedback |
"Workout Overview" | Triggers a summary snapshot for the current session |
"Stop Camera" | ⚠️ Destructive — releases camera + models, fires stop_camera. Not recoverable without re-mounting the component (see warning below) |
To resume tracking after a pause, set currentExercise back to a real model ID from exercises.
stop_camera event. You cannot recover from it without unmounting and re-creating the entire camera component. Only send it when the user is permanently leaving the camera screen. For temporary pauses, use "Pause Exercise" instead.1// Switch exercise (Swift uses two-way binding via @State)
2currentExercise = "394"
3
4// Pause / resume tracking
5currentExercise = "Pause Exercise"
6currentExercise = "3"Customization
Pass any of these fields in customParams (Swift / Kotlin / Flutter) or directly in postData (HTML/JS, React Native, React TS) at initialization.
| Field | Type | Description |
restSpeeches | string[] | Audio phrases to preload (from ExerciseModel.rest_speech) |
videoURL | string | Use a video file instead of the live camera — the full tracking pipeline (reps, accuracy) runs on the video. Ideal for testing without physically performing exercises; see Testing & Simulation |
landmarkColor | string | Pose overlay color in hex with # (default #14FF00) |
showSilhouette | boolean | Show "get into frame" silhouette (default true) |
includeRealtimeAccuracy | boolean | (Beta) Stream live position-confidence alongside reps |
includePoseData | string[] | Any of "angles", "poseLandmarks", "worldLandmarks". Performance impact — only enable for custom calculations |
Event Reference
All events the Camera Component emits to the host app:
| Event | Payload | When |
model_warmedup | { message } | Pose model is ready |
models_loaded | { message, modelIds? } | Exercise models finished downloading. modelIds (the identifiers that resolved, as you sent them) is present when the event answers a fetch, including load_models, and missing on the "All models loaded" notice |
person_in_frame | { message } | User entered the silhouette frame |
successful_repeat | { exercise, value, accuracy } | A rep was counted (value = total reps so far) |
mistake | { value } | Form mistake detected |
correct_position_accuracy | { accuracy } | (Beta) Live position confidence — only when includeRealtimeAccuracy: true |
pose_landmarks | { poseLandmarks } | Per-frame screen-space landmarks — only when includePoseData includes "poseLandmarks" |
world_landmarks | { worldLandmarks } | Per-frame world-space landmarks — only when includePoseData includes "worldLandmarks" |
speech_fetch_complete | { successCount, failureCount, modelIds? } | Without modelIds: the restSpeeches that needed downloading finished (counts are phrases). With modelIds: feedback audio for those models finished loading; successCount is the number of models and failureCount is always 0 |
exercise_change | { prevExerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories } | You switched currentExercise to a different exercise. Session totals for the exercise you left (timeSpent counts from the first time you switched to it) |
exercise_overview | data: [{ exerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories }] | Reply to the "Workout Overview" command, one entry per exercise used this session. Not sent if none was started |
workout_overview | data: { totalReps, totalTimeSpent, totalAverageAccuracy, totalCalories } | Reply to the "Workout Overview" command, session totals. Note the camelCase keys, unlike the workout player's workout_overview |
error_occurred | { severity, message } or { severity, data } | Any error (model fetch fail, phrase fail, camera access fail, etc.). severity is "critical" or "warning" |
warning | { data } or { message } | Non-fatal issue (e.g. no model IDs provided, or no current exercise set) |
stop_camera | { message } | Confirms the "Stop Camera" command finished |
Pose Data
When includePoseData contains "poseLandmarks" or "worldLandmarks", the camera emits per-frame events with raw skeleton data. Only enable this if you're doing custom calculations — there is a performance cost.
Two coordinate spaces:
poseLandmarks— values 0–1, normalized to the camera frame.worldLandmarks— meters, relative to the hips (best Z accuracy).
Each landmark has { x, y, z, visibility } (all 0–1).
Available landmarks (same names in both spaces): nose, leftEyeInner, leftEye, leftEyeOuter, rightEyeInner, rightEye, rightEyeOuter, leftEar, rightEar, mouthLeft, mouthRight, leftShoulder, rightShoulder, leftElbow, rightElbow, leftWrist, rightWrist, leftPinky, rightPinky, leftIndex, rightIndex, leftThumb, rightThumb, leftHip, rightHip, leftKnee, rightKnee, leftAnkle, rightAnkle, leftHeel, rightHeel, leftFootIndex, rightFootIndex.
Available angles (when "angles" is included — both 2D and 3D versions are emitted): leftKneeAngle, rightKneeAngle, leftHipAngle, rightHipAngle, leftShoulderAngle, rightShoulderAngle, leftElbowAngle, rightElbowAngle, leftWristAngle, rightWristAngle, leftAnkleAngle, rightAnkleAngle, leftArmpitAngle, rightArmpitAngle.
Complete Example
Minimal working example with Next / Previous buttons that cycle between exercises and a live rep counter in the UI.
1import SwiftUI
2import KinesteXAIKit
3
4struct CameraScreen: View {
5 let kinestex = KinesteXAIKit(
6 apiKey: "YOUR_API_KEY",
7 companyName: "YOUR_COMPANY_NAME",
8 userId: "YOUR_USER_ID"
9 )
10
11 // 3 = Squats, 394 = Jumping Jack
12 let exerciseIds = ["3", "394"]
13
14 @State private var index = 0
15 @State private var currentExercise = "3"
16 @State private var reps = 0
17 @State private var isLoading = false
18
19 var body: some View {
20 VStack(spacing: 16) {
21 Text("Reps: \(reps)")
22 .font(.title)
23 .padding(.top)
24
25 kinestex.createCameraView(
26 exercises: exerciseIds,
27 currentExercise: $currentExercise,
28 user: nil,
29 isLoading: $isLoading,
30 onMessageReceived: { message in
31 if case .reps(let value) = message {
32 reps = value["value"] as? Int ?? 0
33 }
34 }
35 )
36
37 HStack(spacing: 24) {
38 Button("Previous") { switchTo(index - 1) }
39 Button("Next") { switchTo(index + 1) }
40 }
41 .padding(.bottom)
42 }
43 }
44
45 private func switchTo(_ newIndex: Int) {
46 index = (newIndex + exerciseIds.count) % exerciseIds.count
47 currentExercise = exerciseIds[index]
48 reps = 0
49 }
50}