Camera Component
KinesteX Motion Recognition: Real-Time Engagement.
- Interactive Tracking: Advanced motion recognition for immersive fitness experiences
- Real-Time Feedback: Instantly track reps, spot mistakes, and calculate calories burned
- Boost Motivation: Keep users engaged with detailed exercise feedback
- Custom Integration: Adapt camera placement to fit your app's design
- Your Own Screens: Build prompts, overlays and voice-over on the real-time events the component reports (presence, start position, reps and holds, form feedback, inactivity)
Important: what to pass for `currentExercise` and `exercises`. We recommend exercise IDs: they're stable, and you already have them when listing exercises from the Content API. Set exerciseFetchType: "exercise_id" to use them. The Camera Component also accepts model IDs (the default, kept for backward compatibility, but they require an extra round-trip through the Content API to look up) and exercise titles (case-sensitive: convenient for prototyping, but title matching depends on locale normalization and can mismatch similar exercises, so prefer IDs in production). See the Fetching Exercises section below.
Before showing the camera UI, wait for both model_warmedup and models_loaded events to fire. See the Preloading & Events section below.
1// Model IDs come from the Content API or admin dashboard.
2// See "Camera: Model IDs" subsection.
3@State var currentExercise = "3"
4
5kinestex.createCameraView(
6 exercises: ["3"], // preload every model ID you may switch to
7 currentExercise: $currentExercise,
8 user: nil,
9 style: nil,
10 isLoading: $isLoading,
11 onMessageReceived: { message in
12 switch message {
13 case .custom_type(let value):
14 // A counted rep: "action" of kind "rep" (the older successful_repeat arrives as .reps)
15 if value["type"] as? String == "action", value["kind"] as? String == "rep" {
16 reps = value["value"] as? Int ?? 0
17 }
18 case .mistake(let value):
19 mistake = value["value"] as? String ?? "--"
20 default:
21 break
22 }
23 }
24)Model IDs
exerciseFetchType: "exercise_id" and pass exercise.id directly. See the Fetching Exercises section.Model IDs are still supported (it's the default when exerciseFetchType is omitted) and useful when you already have one handy — e.g. from a stored WorkoutModel.sequence or the admin dashboard. They are numeric values like "3" for Squats or "394" for Jumping Jack.
Three ways to get a model ID:
1. Content API — call fetchExercises() (Swift / Kotlin), fetchContent(contentType: ContentType.exercise) (Flutter), or GET /api/v1/exercises (REST). Each ExerciseModel has a model_id field. See the Content API section for full details. (Note: this is the extra round-trip that exercise IDs let you skip.)
2. Admin dashboard — open the exercise in admin.kinestex.com; the model ID is shown at the top of the page header.
3. Workout sequences — when iterating WorkoutModel.sequence, each ExerciseModel entry exposes its own model_id.
1Task {
2 let result = await kinestex.fetchExercises(limit: 10)
3 if case .success(let response) = result,
4 let exercise = response.exercises.first {
5 // exercise.modelId is what the Camera Component expects
6 currentExercise = exercise.modelId
7 }
8}Fetching Exercises (by ID or Title)
Besides model IDs, the Camera Component can fetch exercises by exercise ID (recommended) or exercise title. Set the exerciseFetchType parameter to choose the form:
exerciseFetchType | Meaning | What goes in exercises / currentExercise |
"exercise_id" ✅ recommended | Exercise IDs from the Content API: easiest to use, no extra round-trip | e.g. "squats_v2" |
"model_id" (default) | Numeric model IDs, kept for backward compatibility; requires a Content API lookup to obtain | "3", "394" |
"exercise_title" | Exercise titles (case-sensitive): handy for prototyping, but title matching depends on locale normalization and can mismatch similar exercises | "Squats", "Jumping Jack" |
Omitting exerciseFetchType keeps the default model-ID behavior, so existing integrations need no migration.
Where to pass it:
- React Native (SDK v1.3.1+): directly in
postData, alongsideexercisesandcurrentExercise. - Swift, Kotlin, Flutter: inside
customParams. - React (TypeScript): inside
customParameters. - HTML/JS: at the top level of the launch message, next to
exercisesandcurrentExercise(a nested object is not read).
Keep one form per session: use the same form for both exercises and currentExercise, and for any later switches.
1// exerciseFetchType goes inside customParams
2kinestex.createCameraView(
3 exercises: ["Squats", "Jumping Jack"],
4 currentExercise: $currentExercise, // e.g. "Squats"
5 user: nil,
6 style: nil,
7 isLoading: $isLoading,
8 customParams: [
9 "exerciseFetchType": "exercise_title" // "model_id" (default) | "exercise_id" | "exercise_title"
10 ],
11 onMessageReceived: handleMessage
12)Loading More Exercises at Runtime
Fetch and cache additional exercise models after the session has started, without re-mounting the camera. Send { workout_activity_action: "load_models", exercises: [...] }, optionally with exerciseFetchType (use the form you launched with). The identifiers that load are added to the list currentExercise can switch to.
The message carries a list, so it needs an SDK call that can send more than one key:
| SDK | How to send it |
| Swift | webViewState.sendMessage([...]), with a WebViewState passed to createCameraView(..., webViewState:) |
| Kotlin | KinesteXSDK.sendMessage(mapOf(...)) |
| React Native (v1.3.0+) | ref.sendAction("workout_activity_action", "load_models", { exercises: [...] }): the third argument (extras) adds the list |
| HTML/JS | iframe.contentWindow.postMessage({...}, srcURL) |
| Flutter, React (TypeScript) | Not available. Their sendAction(action, value) sends a single key and cannot carry the exercises list. Pass every exercise you may need in exercises at launch |
Three-step flow:
1. Send load_models with the identifiers you want to add.
2. Wait for the models_loaded event: its modelIds lists the identifiers that resolved, as you sent them.
3. Switch the active exercise with currentExercise (do not repeat exerciseFetchType). In React Native, use sendAction("currentExercise", id) here: changeExercise only accepts identifiers from the launch exercises list.
If exercises is missing or empty, the SDK posts back { "type": "error_occurred", "severity": "warning", "message": "load_models: no exercises provided" }. If some identifiers fail, you get one error_occurred with severity: "critical" and a message listing them all ("Failed to fetch 2 model(s): a, b"). Identifiers that did resolve are still listed in modelIds. If every identifier fails, no models_loaded is sent.
1// Pass the same WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4// Step 1: fetch additional models at runtime
5webViewState.sendMessage([
6 "workout_activity_action": "load_models",
7 "exercises": ["Jumping Jack", "Lunges"],
8 "exerciseFetchType": "exercise_title" // match the form you used initially
9])
10
11// Step 2: wait for models_loaded (it arrives as .custom_type), then switch
12func handleMessage(_ message: KinestexMessage) {
13 if case .custom_type(let value) = message,
14 value["type"] as? String == "models_loaded",
15 let ids = value["modelIds"] as? [String], ids.contains("Jumping Jack") {
16 // Step 3: switch the active exercise. Do NOT repeat exerciseFetchType here.
17 currentExercise = "Jumping Jack"
18 }
19}Exercise Setup
exercise_setup tells you how the user starts an exercise and where the camera goes, so you can show your own setup screen before tracking. It comes from the exercise's motion model, which the component already loaded, so it needs no request of its own.
When it fires:
- Once each time
currentExerciseswitches to an exercise. Resuming the same exercise after"Pause Exercise"sends nothing; the camera view opening again (for example after the camera selector) sends it again. - If that exercise's model is still loading, once the load settles (
tracked: falseif it failed). - At once, with
tracked: false, for an identifier that is not inexercisesand was not added withload_models. - In reply to
get_exercise_setup(below).
Payload:
| Field | Type | Description |
exercise | string | The identifier as you sent it |
tracked | boolean | false when KinesteX has no motion model for it; every field below is then null (or []) |
start_position | string or null | The position the user starts in, as set on the exercise's model (for example "PLANK" or "LYING_SUPINE"); null when not set |
camera_view | string or null | How the user should face the camera, such as "FRONT" or "SIDE"; null when not set. orientation_check compares the user against it |
alternate_camera_views | string[] | Other views that work too; [] when none |
camera_side | string or null | Where the camera goes, as set on the exercise's model; null when not set |
timestamp | number | Milliseconds since the Unix epoch |
1{ "type": "exercise_setup", "exercise": "16", "tracked": true, "start_position": "PLANK", "camera_view": "SIDE", "alternate_camera_views": [], "camera_side": null, "timestamp": 1793368803120 }
2{ "type": "exercise_setup", "exercise": "999", "tracked": false, "start_position": null, "camera_view": null, "alternate_camera_views": [], "camera_side": null, "timestamp": 1793368803120 }(The first line's values are for illustration; each model carries its own.)
Asking for it: `get_exercise_setup`. Send { workout_activity_action: "get_exercise_setup", exercises: ["3", "394"] } to get the setup of exercises you loaded without switching to them, for example to build a setup screen for the whole session up front. You get one exercise_setup per identifier you sent in exercises or with load_models (each identifier once); one still loading is answered when its load settles. Identifiers you never sent get no event, only one error_occurred warning naming them: "get_exercise_setup: <ids> not in exercises; add them with load_models first". An empty list gets the warning "get_exercise_setup: no exercises provided".
The request carries a list, so it needs Swift webViewState.sendMessage, Kotlin KinesteXSDK.sendMessage, React Native sendAction with extras, or a raw postMessage (HTML/JS). Flutter and React (TypeScript) cannot send it (their sendAction sends a single key); they still get exercise_setup on every exercise switch.
In workouts and custom workouts launched with includePoseData containing "events", exercise_setup is sent as each exercise starts, with exercise set to the workout's exercise ID.
1// Pass the same WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4// Ask for the setup of exercises you loaded
5webViewState.sendMessage([
6 "workout_activity_action": "get_exercise_setup",
7 "exercises": ["3", "394"]
8])
9
10// In onMessageReceived (exercise_setup arrives as .custom_type)
11if case .custom_type(let value) = message,
12 value["type"] as? String == "exercise_setup" {
13 let exercise = value["exercise"] as? String ?? ""
14 let tracked = value["tracked"] as? Bool ?? false
15 let cameraView = value["camera_view"] as? String // nil when not set
16 showSetupScreen(exercise: exercise, tracked: tracked, cameraView: cameraView)
17}Preloading & Events
Two events fire as the component initializes. Wait for both before revealing the camera UI to the user.
| Event | Meaning |
model_warmedup | Pose-tracking (MediaPipe) model is ready |
models_loaded | All exercise models in exercises have downloaded |
Pattern: mount the camera hidden (e.g. opacity: 0) with a loader on top. Reveal once both events have fired.
The two events can arrive in either order. If every exercise fails to load, models_loaded never arrives: you get error_occurred with severity: "critical" instead, so stop waiting on it. model_warmedup fires again if the view is re-entered (for example after the camera selector). Both also arrive when you launch with the camera off (pauseCamera: true, see Camera Off & Mute), so you can preload before the user needs the camera.
1@State private var modelWarmedUp = false
2@State private var modelsLoaded = false
3var isReady: Bool { modelWarmedUp && modelsLoaded }
4
5kinestex.createCameraView(
6 exercises: ["3"],
7 currentExercise: $currentExercise,
8 user: nil,
9 style: nil,
10 isLoading: $isLoading,
11 onMessageReceived: { message in
12 if case .custom_type(let value) = message,
13 let type = value["type"] as? String {
14 if type == "model_warmedup" { modelWarmedUp = true }
15 if type == "models_loaded" { modelsLoaded = true }
16 }
17 }
18)
19.opacity(isReady ? 1 : 0)Controls
Switching exercises in real time. Set currentExercise to any identifier from exercises (or one added later with load_models), in the form you picked with exerciseFetchType. Tracking switches at once, and you receive exercise_change for the exercise you left and exercise_setup for the new one (see Exercise Setup). Sending the value that is already active does nothing.
How each SDK sets `currentExercise`:
| SDK | Exercise identifiers | Control commands (below) |
| Swift | Set the currentExercise binding | Same |
| Kotlin | KinesteXSDK.updateCurrentExercise(id) | Same |
| React Native, React (TypeScript) | ref.changeExercise(id) (only identifiers listed in the launch exercises) or ref.sendAction("currentExercise", id) | ref.sendAction("currentExercise", "Pause Exercise"): changeExercise rejects anything not in the launch exercises |
| Flutter | KinesteXAIFramework.sendAction("currentExercise", id) | Same |
| HTML/JS | postMessage({ currentExercise: id }, srcURL) | Same |
Control commands. Send any of these strings as currentExercise:
| Command | Effect | Reply |
"Pause Exercise" | Stops rep counting and mistake detection. pose_landmarks / world_landmarks keep streaming. Send an exercise identifier to resume | none |
"Pause Audio" | Stops coach cues and rest speech; the rep sound still plays. feedback and mistake events are still sent. To silence everything, use mute_workout (Camera Off & Mute) | none |
"Resume Audio" | Turns coach cues back on | none |
"Workout Overview" | Reports the session so far. Tracking continues on the current exercise | exercise_overview (only once an exercise has started) and workout_overview |
"Stop Camera" | Destructive. Releases the camera and the pose model and clears the view (see warning below) | stop_camera |
A command acts only when the value changes, so "Workout Overview" sent twice in a row replies once. Send the exercise identifier between two overviews.
Restart controls. Restart without re-mounting by sending these as workout_activity_action, the channel also used by load_models, get_exercise_setup, pause_camera / resume_camera and mute_workout / unmute_workout. Each one acts every time it is sent, including twice in a row.
workout_activity_action | Effect | Reply |
"reset_progress" | Clears the current exercise's reps, accuracy, mistakes and time. Tracking continues from rep 1; a rep in progress is discarded | { type: "reset_progress", exerciseId, date } |
"reset_all_progress" | Clears every exercise tracked this session, then restarts the current one from rep 1 | { type: "reset_all_progress", exerciseId, date } (exerciseId is null if none started) |
"reset_check_frame" | Shows the positioning check again, also with showSilhouette: false. Reps pause until the user is back in position; a rep or hold in progress starts over | { type: "reset_check_frame", date }, then person_in_frame |
Send reset_progress (or reset_all_progress) together with reset_check_frame for a full restart with a positioning check.
After a reset, the rep value (action kind rep, successful_repeat) counts from 1 again, and exercise_overview / exercise_change cover only the time since the reset. reset_progress keeps the other exercises in exercise_overview; reset_all_progress removes them. After "Pause Exercise", reset_progress resets the exercise that was active before the pause.
Send restart controls once the camera view is open (after model_warmedup is safe). Otherwise you get { type: "error_occurred", severity: "warning", message: "<action>: camera is not active" }, also after "Stop Camera". reset_progress before any currentExercise was set replies message: "reset_progress: no active exercise".
stop_camera. Sending an exercise identifier as currentExercise afterwards reopens the camera and reloads the pose model (no re-mount, but slower than pause_camera / resume_camera). Send it only when the user is leaving the camera screen for good. For a temporary stop use "Pause Exercise"; to turn the camera off for a while use pause_camera (Camera Off & Mute); to start over use the restart controls.1// Switch exercise (Swift uses two-way binding via @State)
2currentExercise = "394"
3
4// Pause / resume tracking
5currentExercise = "Pause Exercise"
6currentExercise = "3"1// Pass a WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4webViewState.sendMessage(["workout_activity_action": "reset_progress"])Camera Off & Mute
Turn the device camera off and on, and silence all KinesteX audio, at runtime without re-mounting the view. These are workout_activity_action commands, like the restart controls. Nothing changes unless you send them.
workout_activity_action | Effect | Echo |
"pause_camera" | Releases the camera: the OS camera indicator goes off, no frame is captured or tracked, and the camera area is empty. The pose model stays loaded and warm, and the current exercise, rep counts and settings are kept | { type: "pause_camera", date } |
"resume_camera" | Reopens the same camera and keeps tracking the current exercise | { type: "resume_camera", date }, then camera_selected once the camera is open |
"mute_workout" | Silences every sound the component plays: the rep sound, coach cues and rest speech. Tracking and every event continue | { type: "mute_workout", date } |
"unmute_workout" | Turns audio back on. A "Pause Audio" you sent stays in effect | { type: "unmute_workout", date } |
Starting with the camera off or muted. Launch with pauseCamera: true to open with the camera off, as if pause_camera had been sent before it opened: the exercise models and the pose model still load and warm up, so model_warmedup still arrives. Launch with muteWorkout: true to start muted. Both accept true or "true", send no echo, apply to the Camera Component only, and are not saved: the next launch starts with the camera and sound on unless you pass them again.
When to send them. Any time after kinestex_loaded, including before the camera opens. A pause_camera sent while the view is still loading keeps the camera from ever turning on. A mute_workout sent before the camera view opens is applied, and echoed, when it opens.
Notes:
- Only
resume_cameraturns the camera back on. SwitchingcurrentExercise, the app returning from the background and a Retry tap all leave it off. - A
pause_camerawhile the camera is already off, or aresume_camerawhile it is on, changes nothing but still echoes. In the second case nocamera_selectedfollows, so do not wait for one. - After
resume_camera, rep counts continue from the lastvalue; a rep or hold that was half done starts over. If the user was not in position yet, the positioning check shows again. Sendreset_check_frameright afterresume_camerato make the user get back in position in any case. timeSpentinexercise_changeandexercise_overviewincludes time with the camera off.mute_workoutand"Pause Audio"are independent:"Pause Audio"stops coach cues and rest speech but keeps the rep sound. After"Pause Audio", a mute and an unmute bring back only the rep sound; coach cues return with"Resume Audio". The device volume and silent switch still apply on top.pause_cameraandresume_camerawork in the Camera Component only. Other integrations reply{ type: "error_occurred", severity: "warning", message: "pause_camera: only supported in the camera component" }(orresume_camera: ...). Workouts pause withpause_workout/resume_workout, which the Camera Component does not act on.
| Control | Camera | Use it for |
currentExercise: "Pause Exercise" | Stays on; the user still sees themselves | A short stop, such as a rest between sets |
pause_camera | Off and released; the pose model stays warm | The camera must not run for a while (the user opted out of motion tracking, an exercise you don't track) |
currentExercise: "Stop Camera" | Off; the pose model is released | The user leaves the camera screen for good |
1// Pass the same WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4kinestex.createCameraView(
5 exercises: ["3"],
6 currentExercise: $currentExercise,
7 user: nil,
8 style: nil,
9 isLoading: $isLoading,
10 customParams: ["pauseCamera": true, "muteWorkout": true],
11 onMessageReceived: { message in
12 if case .custom_type(let value) = message,
13 let type = value["type"] as? String {
14 if type == "resume_camera" { cameraOn = true }
15 if type == "pause_camera" { cameraOn = false }
16 }
17 },
18 webViewState: webViewState
19)
20
21// Later, when the user needs the camera
22webViewState.sendMessage(["workout_activity_action": "resume_camera"])
23// Sound back on
24webViewState.sendMessage(["workout_activity_action": "unmute_workout"])Customization
Pass any of these fields at initialization: in customParams (Swift, Kotlin, Flutter), in customParameters (React Native, React TS), or at the top level of the launch message (HTML/JS).
| Field | Type | Description |
restSpeeches | string[] | Audio phrases to preload (from ExerciseModel.rest_speech) |
videoURL | string | Use a video file instead of the live camera: the full tracking pipeline (reps, accuracy) runs on the video. Ideal for testing without physically performing exercises; see Testing & Simulation |
landmarkColor | string | Pose overlay color in hex with # (default #14FF00) |
showSilhouette | boolean | Show "get into frame" silhouette (default true). With false, tracking starts at once, and framing_hint and person_in_frame fire only during a check you open with reset_check_frame |
includeRealtimeAccuracy | boolean | (Beta) Stream live position-confidence alongside reps |
includePoseData | string[] | Any of "angles", "poseLandmarks", "worldLandmarks". Performance impact: only enable for custom calculations. "angles" only fills the angle fields inside the landmark events. ("events" is for workouts; the Camera Component always sends its real-time events) |
pauseCamera | boolean | Open with the camera off (default false). See Camera Off & Mute |
muteWorkout | boolean | Start with all KinesteX audio muted (default false). See Camera Off & Mute |
inactivityTimeout | number | Idle seconds before activity INACTIVE (default 15). See Real-Time Events |
includeFramingHints | boolean | Turns on framing_hint (default false) |
includeFormState | boolean | Turns on form_state (default false) |
enableGestureControl | boolean | Turns on gesture_detected (default false). KinesteX reports the gesture; pausing is up to you |
frameExitPause | boolean | No reps count while the user is out of view (presence LEFT) (default false) |
frameExitPrompt | boolean | With KinesteX audio on, plays the "Pause activated" prompt of KinesteX workouts on presence LEFT (default false) |
currentRestSpeech | string | Plays one of the restSpeeches now; "Stop audio" stops it. Needs restSpeeches at launch. Update it like currentExercise; the same value twice in a row plays once |
showFormGuidance | boolean | Shows the live depth bar for exercises that have depth guidance (default false in the Camera component) |
streamFormGuidance | boolean | Hides the bar and streams form_guidance_config / form_guidance_frame to your app instead |
includePoseBorders | boolean | Out-of-frame guard (default true): when elbows or knees leave the frame, the skeleton turns red, a "step back" cue plays and those frames are not tracked (no reps, no pose_landmarks) |
isDrawingPose | boolean | false hides the skeleton overlay; tracking is unchanged |
videoFit | string | "cover" (default) or "contain" |
shouldShowCameraSelector | boolean | Adds a switch-camera button to the positioning check. Opening it leaves the camera view: on return model_warmedup and person_in_frame fire again and successful_repeat.value restarts at 1 |
hideConnectTvButton, checkFrameText | boolean, string | Positioning-check options, see UI Controls |
Real-Time Events
The component reports what the user is doing, so you can build your own screens, prompts, overlays and voice-over on top of KinesteX motion tracking. The events reach the message handler you already pass; switch on type. State events fire on change, never on every frame, so each one maps straight to a UI state.
Try them live: the Camera Events Lab embeds the Camera Component with your company name and API key, plays sample videos (or uses your camera), sends every command with one click, and lists every event with its payload.
| Event | What it tells you | Key fields | Turn on with |
presence | The user left the picture (LEFT: nobody in view for 3 s) or came back (RETURNED: back in view for 1 s). Every LEFT gets its RETURNED | state | always on |
orientation_check | Whether the user faces the camera the way the exercise needs: the first reading for each exercise, then every change (held 0.6 s). Only for exercises whose camera_view is FRONT or SIDE | expected, detected (FRONT, SIDE or UNKNOWN), matches | always on |
framing_hint | While the positioning check is open: which body parts are out of view and what would fix it | missing_body_parts, hint (STEP_BACK, MOVE_TO_CENTER or null) | includeFramingHints: true |
start_position_reached | The user settled into the exercise's start position (again after every interruption and reset) | exercise | always on |
action kind rep | A rep was counted | value, accuracy, rep_duration_ms | always on |
action kind hold | A timed hold (plank, wall sit) started, broke or resumed | state (STARTED, BROKEN or RESUMED), held_ms | always on |
feedback | Every form cue KinesteX detects, whether its audio is on, paused or muted. announce: true is a cue to voice now; announce: false is a live re-detection between announcements | kind (mistake or tip), message, feedback_id, body_region, count, announce | always on |
mistake | What KinesteX's coach says, in the older shape | value, mistake_id, body_region, count, is_tip | always on |
form_state | Form quality changed | state (GOOD, ADJUST or OUT_OF_POSITION) | includeFormState: true |
activity | The user has been idle for inactivityTimeout seconds (INACTIVE), or moved again (ACTIVE). Every INACTIVE gets its ACTIVE | state, idle_seconds | always on; inactivityTimeout (default 15) |
gesture_detected | The user made the hands-free gesture (arms down by the sides, then forearms crossed in front of the chest). KinesteX only reports it; pausing is up to you | gesture (PAUSE or RESUME) | enableGestureControl: true |
Common fields. Every event has type and timestamp (milliseconds since the Unix epoch, from the device clock). The exercise events also carry exercise, the identifier as you sent it in currentExercise; framing_hint and gesture_detected do not.
When they run. presence, start_position_reached, action, form_state and activity follow the exercise being tracked: from the exercise you send in currentExercise until "Pause Exercise" or "Stop Camera" (after "Workout Overview" only action reps keep coming, until you send an exercise id), and not while the positioning check is open, the camera is off (pause_camera) or the app is in the background.
Handle one of each pair. action kind rep replaces successful_repeat: both are sent for every rep with the same fields and timestamp. feedback covers what mistake reports, and presence comes right after left_camera_frame / returned_camera_frame. Handle one of each pair, or every rep, cue or exit counts twice. New integrations: action, feedback and presence.
Launch options (details in Real-Time Motion Parameters):
| Option | Default | Effect |
inactivityTimeout | 15 | Idle seconds before activity INACTIVE |
includeFramingHints | false | Turns on framing_hint |
includeFormState | false | Turns on form_state |
enableGestureControl | false | Turns on gesture_detected |
frameExitPause | false | No reps count while the user is out of view (presence LEFT); a rep half done when they left starts over on RETURNED |
frameExitPrompt | false | With KinesteX audio on, plays the "Pause activated" prompt of KinesteX workouts on presence LEFT |
showSilhouette | true | With false, tracking starts at once, and framing_hint fires only during a check you open with reset_check_frame |
The boolean options also accept the string "true".
Forward compatibility. KinesteX adds to these events over time: new event types, new fields and new kind and state values. Ignore types, fields and values you don't know, and let your switch fall through to doing nothing, so a new KinesteX release never breaks your integration.
Field-by-field payloads and timings: Real-Time Motion Events. Workouts and custom workouts send the same events when launched with includePoseData containing "events" (see Custom Workout).
1@State private var currentExercise = "3"
2@State private var reps = 0
3@State private var isAway = false
4@State private var turnPrompt: String?
5@State private var cue: String?
6@State private var showHelp = false
7
8kinestex.createCameraView(
9 exercises: ["3"],
10 currentExercise: $currentExercise,
11 user: nil,
12 style: nil,
13 isLoading: $isLoading,
14 customParams: [
15 "includeFramingHints": true,
16 "includeFormState": true,
17 "inactivityTimeout": 20
18 ],
19 onMessageReceived: { message in
20 // These events arrive as .custom_type; the event name is in "type"
21 guard case .custom_type(let data) = message,
22 let type = data["type"] as? String else { return }
23 switch type {
24 case "presence":
25 isAway = data["state"] as? String == "LEFT"
26 case "orientation_check":
27 // UNKNOWN says nothing about the way the user faces: don't prompt on it
28 if data["detected"] as? String != "UNKNOWN" {
29 let matches = data["matches"] as? Bool ?? true
30 turnPrompt = matches ? nil : data["expected"] as? String
31 }
32 case "start_position_reached":
33 showGo()
34 case "action":
35 if data["kind"] as? String == "rep" {
36 reps = data["value"] as? Int ?? reps
37 }
38 case "feedback":
39 if data["announce"] as? Bool == true {
40 cue = data["message"] as? String
41 }
42 case "activity":
43 showHelp = data["state"] as? String == "INACTIVE"
44 default:
45 break // types, kinds and states added later: ignore
46 }
47 }
48)Event Reference
All events the Camera Component emits to the host app. Every event also carries timestamp (milliseconds since the Unix epoch, from the device clock). date fields use the format "dd mm yyyy hh:mm:ss".
| Event | Payload | When |
kinestex_launched | { data } | Your launch payload was verified |
model_warmedup | { message } | Pose model is ready |
models_loaded | { message, modelIds? } | Exercise models finished downloading. modelIds (the identifiers that resolved, as you sent them) is present when the event answers a fetch, including load_models, and missing on the "All models loaded" notice |
camera_selected | { id, label, isMirrorCamera } | The camera stream opened (at start, after the app returns from the background, after a camera switch, or after resume_camera) |
person_in_frame | { message } | The user passed the positioning check, at start and again after reset_check_frame. Not sent at start with showSilhouette: false |
exercise_setup | { exercise, tracked, start_position, camera_view, alternate_camera_views, camera_side } | On each switch to an exercise, and in reply to get_exercise_setup. See Exercise Setup |
action | { kind: "rep", exercise, value, accuracy, rep_duration_ms? } or { kind: "hold", exercise, state, held_ms } | A rep was counted, or a timed hold started, broke or resumed. The rep form replaces successful_repeat (same fields and timestamp, sent right after it): handle one of the two. See Real-Time Events |
successful_repeat | { exercise, value, accuracy, rep_duration_ms? } | A rep was counted. exercise is the active identifier as you sent it; value is that exercise's rep count this session (back to 1 after a reset); accuracy is the rep's peak position confidence; rep_duration_ms is the time since the previous rep, left out for the first rep of an exercise and the first rep after an interruption (back in view, camera back on, "Stop Camera", a reset, an exercise switch) or an idle stretch. Kept for existing integrations; action kind rep is its replacement |
feedback | { exercise, kind, message, feedback_id, body_region, count, announce } | Every form cue KinesteX detects, audio on or off. See Real-Time Events |
mistake | { exercise, value, mistake_id, body_region, count, is_tip } | What KinesteX's coach says (value is the cue text; is_tip marks a tip rather than a correction). With KinesteX's voice on, sent as the coach starts saying the cue; with audio paused or muted, when the cue is detected, once per cue per its repeat interval. It is not one event per frame. Handle feedback or mistake, not both |
presence | { exercise, state } | The user left the picture (LEFT) or came back (RETURNED) |
left_camera_frame / returned_camera_frame | { date, exercise } | Sent right before presence LEFT / RETURNED, in the shape KinesteX workouts use. Handle these or presence, not both |
orientation_check, framing_hint, start_position_reached, form_state, activity, gesture_detected | see Real-Time Events | User state changes (framing_hint, form_state and gesture_detected only with their launch options) |
correct_position_accuracy | { accuracy } | (Beta) Live position confidence, only with includeRealtimeAccuracy: true |
pose_landmarks | { poseLandmarks } | Per frame, only when includePoseData includes "poseLandmarks". See Pose Data |
world_landmarks | { worldLandmarks } | Per frame, only when includePoseData includes "worldLandmarks". See Pose Data |
form_guidance_config, form_guidance_frame, rep_depth_angle | see Camera & Frame Events | Depth guidance, with showFormGuidance or streamFormGuidance. id is the current exercise |
speech_fetch_complete | { successCount, failureCount, modelIds? } | Without modelIds: the restSpeeches that needed downloading finished (counts are phrases). With modelIds: feedback audio for those models finished loading; successCount is the number of models and failureCount is always 0 |
exercise_change | { prevExerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories } | You switched currentExercise to a different exercise. Totals for the exercise you left; timeSpent counts from the first time you switched to it (or from its last reset), time on other exercises included. mistakes is [{ mistake, count }] |
exercise_overview | data: [{ exerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories }] | Reply to "Workout Overview", one entry per exercise used this session. Not sent if none was started |
workout_overview | data: { totalReps, totalTimeSpent, totalAverageAccuracy, totalCalories } | Reply to "Workout Overview", session totals (all 0 if no exercise started). Note the camelCase keys, unlike the workout player's workout_overview |
reset_progress / reset_all_progress | { exerciseId, date } | Echo of the restart control (exerciseId can be null for reset_all_progress) |
reset_check_frame | { date } | Echo; person_in_frame follows when the user is back in position |
pause_camera / resume_camera | { date } | Echo of the command, every time it is sent. See Camera Off & Mute |
mute_workout / unmute_workout | { date } | Echo of the command, every time it is sent (a command sent before the camera view opens is echoed when it opens) |
error_occurred | { severity, message } or { severity, data } | Any error (model fetch, phrase fetch, camera access, a restart control with nothing to act on, a get_exercise_setup for identifiers you never loaded). severity is "critical" or "warning" |
warning | { data } or { message } | Non-fatal issue (e.g. no model IDs provided, or no current exercise set) |
stop_camera | { message } | Confirms the "Stop Camera" command finished |
The Camera component does not send exercise_completed, workout_progress or exit_kinestex.
Pose Data
When includePoseData contains "poseLandmarks" or "worldLandmarks", the camera emits one event per processed frame. Only enable this for custom calculations; there is a performance cost. No event is sent on frames with no person in view, or while the user is outside the frame with includePoseBorders on.
Payload:
1{
2 "type": "pose_landmarks",
3 "poseLandmarks": {
4 "coordinates": { "nose": { "x": 0.51, "y": 0.18, "z": -0.32, "visibility": 0.99 }, "...": {} },
5 "angles2D": { "leftKneeAngle": 172.4, "...": 0 },
6 "angles3D": { "leftKneeAngle": 168.9, "...": 0 }
7 },
8 "timestamp": 1793368803120
9}world_landmarks has the same shape under worldLandmarks. Only the first detected person is reported. timestamp (milliseconds since the Unix epoch) is on every frame. The Camera Component's frames carry no exercise field; match them to the exercise you set in currentExercise.
In workouts. Workouts, plans, challenges and custom workouts take the same includePoseData values. Their frames also carry exercise (the workout's exercise ID, null during the positioning check), are sent during the positioning check and the exercises but not on rests, and always include 2D angles. "events" adds the real-time events there; the Camera Component always sends them, so it needs no "events". See Custom Workout: Pose Data & Real-Time Events.
Two coordinate spaces:
poseLandmarks:xandynormalized 0-1 to the camera frame.xis flipped (1 - x) to match the mirrored preview (the camera is mirrored by default), so a point on the right of the screen has a largerx.zis relative depth, not 0-1.worldLandmarks: meters, origin between the hips, not flipped (best Z accuracy).
visibility is 0-1 in both. Landmark names use the person's own left and right.
Angles: "angles" adds no event of its own. It fills angles2D and angles3D inside the pose_landmarks / world_landmarks payloads (without it they are all 0, except angles2D, which enableGestureControl: true also fills), so pass it together with "poseLandmarks" or "worldLandmarks". Both keys are always present. Each payload's angles are computed from its own coordinates.
Available landmarks (same names in both spaces): nose, leftEyeInner, leftEye, leftEyeOuter, rightEyeInner, rightEye, rightEyeOuter, leftEar, rightEar, mouthLeft, mouthRight, leftShoulder, rightShoulder, leftElbow, rightElbow, leftWrist, rightWrist, leftPinky, rightPinky, leftIndex, rightIndex, leftThumb, rightThumb, leftHip, rightHip, leftKnee, rightKnee, leftAnkle, rightAnkle, leftHeel, rightHeel, leftFootIndex, rightFootIndex.
Available angles (filled when "angles" is included, in both angles2D and angles3D): leftKneeAngle, rightKneeAngle, leftHipAngle, rightHipAngle, leftShoulderAngle, rightShoulderAngle, leftElbowAngle, rightElbowAngle, leftWristAngle, rightWristAngle, leftAnkleAngle, rightAnkleAngle, leftArmpitAngle, rightArmpitAngle.
Complete Example
Minimal working example with Next / Previous buttons that cycle between exercises and a live rep counter in the UI.
1import SwiftUI
2import KinesteXAIKit
3
4struct CameraScreen: View {
5 let kinestex = KinesteXAIKit(
6 apiKey: "YOUR_API_KEY",
7 companyName: "YOUR_COMPANY_NAME",
8 userId: "YOUR_USER_ID"
9 )
10
11 // 3 = Squats, 394 = Jumping Jack
12 let exerciseIds = ["3", "394"]
13
14 @State private var index = 0
15 @State private var currentExercise = "3"
16 @State private var reps = 0
17 @State private var isLoading = false
18
19 var body: some View {
20 VStack(spacing: 16) {
21 Text("Reps: \(reps)")
22 .font(.title)
23 .padding(.top)
24
25 kinestex.createCameraView(
26 exercises: exerciseIds,
27 currentExercise: $currentExercise,
28 user: nil,
29 style: nil,
30 isLoading: $isLoading,
31 onMessageReceived: { message in
32 // A counted rep: "action" of kind "rep" (arrives as .custom_type)
33 if case .custom_type(let value) = message,
34 value["type"] as? String == "action",
35 value["kind"] as? String == "rep" {
36 reps = value["value"] as? Int ?? 0
37 }
38 }
39 )
40
41 HStack(spacing: 24) {
42 Button("Previous") { switchTo(index - 1) }
43 Button("Next") { switchTo(index + 1) }
44 }
45 .padding(.bottom)
46 }
47 }
48
49 private func switchTo(_ newIndex: Int) {
50 index = (newIndex + exerciseIds.count) % exerciseIds.count
51 currentExercise = exerciseIds[index]
52 reps = 0
53 }
54}