KinesteX

Camera Component

KinesteX Motion Recognition: Real-Time Engagement.


  • Interactive Tracking: Advanced motion recognition for immersive fitness experiences
  • Real-Time Feedback: Instantly track reps, spot mistakes, and calculate calories burned
  • Boost Motivation: Keep users engaged with detailed exercise feedback
  • Custom Integration: Adapt camera placement to fit your app's design
  • Your Own Screens: Build prompts, overlays and voice-over on the real-time events the component reports (presence, start position, reps and holds, form feedback, inactivity)

Important: what to pass for `currentExercise` and `exercises`. We recommend exercise IDs: they're stable, and you already have them when listing exercises from the Content API. Set exerciseFetchType: "exercise_id" to use them. The Camera Component also accepts model IDs (the default, kept for backward compatibility, but they require an extra round-trip through the Content API to look up) and exercise titles (case-sensitive: convenient for prototyping, but title matching depends on locale normalization and can mismatch similar exercises, so prefer IDs in production). See the Fetching Exercises section below.


Before showing the camera UI, wait for both model_warmedup and models_loaded events to fire. See the Preloading & Events section below.

Quick Start
Swift example:
1// Model IDs come from the Content API or admin dashboard.
2// See "Camera: Model IDs" subsection.
3@State var currentExercise = "3"
4
5kinestex.createCameraView(
6    exercises: ["3"], // preload every model ID you may switch to
7    currentExercise: $currentExercise,
8    user: nil,
9    style: nil,
10    isLoading: $isLoading,
11    onMessageReceived: { message in
12        switch message {
13        case .custom_type(let value):
14            // A counted rep: "action" of kind "rep" (the older successful_repeat arrives as .reps)
15            if value["type"] as? String == "action", value["kind"] as? String == "rep" {
16                reps = value["value"] as? Int ?? 0
17            }
18        case .mistake(let value):
19            mistake = value["value"] as? String ?? "--"
20        default:
21            break
22        }
23    }
24)

Model IDs

Recommendation: prefer exercise IDs — they're easier to get (you already have them when listing exercises from the Content API) and don't require an extra round-trip just to look up a numeric model ID. Set exerciseFetchType: "exercise_id" and pass exercise.id directly. See the Fetching Exercises section.

Model IDs are still supported (it's the default when exerciseFetchType is omitted) and useful when you already have one handy — e.g. from a stored WorkoutModel.sequence or the admin dashboard. They are numeric values like "3" for Squats or "394" for Jumping Jack.


Three ways to get a model ID:


1. Content API — call fetchExercises() (Swift / Kotlin), fetchContent(contentType: ContentType.exercise) (Flutter), or GET /api/v1/exercises (REST). Each ExerciseModel has a model_id field. See the Content API section for full details. (Note: this is the extra round-trip that exercise IDs let you skip.)

2. Admin dashboard — open the exercise in admin.kinestex.com; the model ID is shown at the top of the page header.

3. Workout sequences — when iterating WorkoutModel.sequence, each ExerciseModel entry exposes its own model_id.

Fetch a Model ID
Swift example:
1Task {
2    let result = await kinestex.fetchExercises(limit: 10)
3    if case .success(let response) = result,
4       let exercise = response.exercises.first {
5        // exercise.modelId is what the Camera Component expects
6        currentExercise = exercise.modelId
7    }
8}

Fetching Exercises (by ID or Title)

Besides model IDs, the Camera Component can fetch exercises by exercise ID (recommended) or exercise title. Set the exerciseFetchType parameter to choose the form:


exerciseFetchTypeMeaningWhat goes in exercises / currentExercise
"exercise_id" ✅ recommendedExercise IDs from the Content API: easiest to use, no extra round-tripe.g. "squats_v2"
"model_id" (default)Numeric model IDs, kept for backward compatibility; requires a Content API lookup to obtain"3", "394"
"exercise_title"Exercise titles (case-sensitive): handy for prototyping, but title matching depends on locale normalization and can mismatch similar exercises"Squats", "Jumping Jack"

Omitting exerciseFetchType keeps the default model-ID behavior, so existing integrations need no migration.


Where to pass it:

  • React Native (SDK v1.3.1+): directly in postData, alongside exercises and currentExercise.
  • Swift, Kotlin, Flutter: inside customParams.
  • React (TypeScript): inside customParameters.
  • HTML/JS: at the top level of the launch message, next to exercises and currentExercise (a nested object is not read).

Keep one form per session: use the same form for both exercises and currentExercise, and for any later switches.

Fetch by Exercise Title
Swift example:
1// exerciseFetchType goes inside customParams
2kinestex.createCameraView(
3    exercises: ["Squats", "Jumping Jack"],
4    currentExercise: $currentExercise, // e.g. "Squats"
5    user: nil,
6    style: nil,
7    isLoading: $isLoading,
8    customParams: [
9        "exerciseFetchType": "exercise_title" // "model_id" (default) | "exercise_id" | "exercise_title"
10    ],
11    onMessageReceived: handleMessage
12)

Loading More Exercises at Runtime

Fetch and cache additional exercise models after the session has started, without re-mounting the camera. Send { workout_activity_action: "load_models", exercises: [...] }, optionally with exerciseFetchType (use the form you launched with). The identifiers that load are added to the list currentExercise can switch to.


The message carries a list, so it needs an SDK call that can send more than one key:


SDKHow to send it
SwiftwebViewState.sendMessage([...]), with a WebViewState passed to createCameraView(..., webViewState:)
KotlinKinesteXSDK.sendMessage(mapOf(...))
React Native (v1.3.0+)ref.sendAction("workout_activity_action", "load_models", { exercises: [...] }): the third argument (extras) adds the list
HTML/JSiframe.contentWindow.postMessage({...}, srcURL)
Flutter, React (TypeScript)Not available. Their sendAction(action, value) sends a single key and cannot carry the exercises list. Pass every exercise you may need in exercises at launch

Three-step flow:


1. Send load_models with the identifiers you want to add.

2. Wait for the models_loaded event: its modelIds lists the identifiers that resolved, as you sent them.

3. Switch the active exercise with currentExercise (do not repeat exerciseFetchType). In React Native, use sendAction("currentExercise", id) here: changeExercise only accepts identifiers from the launch exercises list.


If exercises is missing or empty, the SDK posts back { "type": "error_occurred", "severity": "warning", "message": "load_models: no exercises provided" }. If some identifiers fail, you get one error_occurred with severity: "critical" and a message listing them all ("Failed to fetch 2 model(s): a, b"). Identifiers that did resolve are still listed in modelIds. If every identifier fails, no models_loaded is sent.

Load More Models, Then Switch
Swift example:
1// Pass the same WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4// Step 1: fetch additional models at runtime
5webViewState.sendMessage([
6    "workout_activity_action": "load_models",
7    "exercises": ["Jumping Jack", "Lunges"],
8    "exerciseFetchType": "exercise_title" // match the form you used initially
9])
10
11// Step 2: wait for models_loaded (it arrives as .custom_type), then switch
12func handleMessage(_ message: KinestexMessage) {
13    if case .custom_type(let value) = message,
14       value["type"] as? String == "models_loaded",
15       let ids = value["modelIds"] as? [String], ids.contains("Jumping Jack") {
16        // Step 3: switch the active exercise. Do NOT repeat exerciseFetchType here.
17        currentExercise = "Jumping Jack"
18    }
19}

Exercise Setup

exercise_setup tells you how the user starts an exercise and where the camera goes, so you can show your own setup screen before tracking. It comes from the exercise's motion model, which the component already loaded, so it needs no request of its own.


When it fires:

  • Once each time currentExercise switches to an exercise. Resuming the same exercise after "Pause Exercise" sends nothing; the camera view opening again (for example after the camera selector) sends it again.
  • If that exercise's model is still loading, once the load settles (tracked: false if it failed).
  • At once, with tracked: false, for an identifier that is not in exercises and was not added with load_models.
  • In reply to get_exercise_setup (below).

Payload:


FieldTypeDescription
exercisestringThe identifier as you sent it
trackedbooleanfalse when KinesteX has no motion model for it; every field below is then null (or [])
start_positionstring or nullThe position the user starts in, as set on the exercise's model (for example "PLANK" or "LYING_SUPINE"); null when not set
camera_viewstring or nullHow the user should face the camera, such as "FRONT" or "SIDE"; null when not set. orientation_check compares the user against it
alternate_camera_viewsstring[]Other views that work too; [] when none
camera_sidestring or nullWhere the camera goes, as set on the exercise's model; null when not set
timestampnumberMilliseconds since the Unix epoch

json
1{ "type": "exercise_setup", "exercise": "16", "tracked": true, "start_position": "PLANK", "camera_view": "SIDE", "alternate_camera_views": [], "camera_side": null, "timestamp": 1793368803120 }
2{ "type": "exercise_setup", "exercise": "999", "tracked": false, "start_position": null, "camera_view": null, "alternate_camera_views": [], "camera_side": null, "timestamp": 1793368803120 }

(The first line's values are for illustration; each model carries its own.)


Asking for it: `get_exercise_setup`. Send { workout_activity_action: "get_exercise_setup", exercises: ["3", "394"] } to get the setup of exercises you loaded without switching to them, for example to build a setup screen for the whole session up front. You get one exercise_setup per identifier you sent in exercises or with load_models (each identifier once); one still loading is answered when its load settles. Identifiers you never sent get no event, only one error_occurred warning naming them: "get_exercise_setup: <ids> not in exercises; add them with load_models first". An empty list gets the warning "get_exercise_setup: no exercises provided".


The request carries a list, so it needs Swift webViewState.sendMessage, Kotlin KinesteXSDK.sendMessage, React Native sendAction with extras, or a raw postMessage (HTML/JS). Flutter and React (TypeScript) cannot send it (their sendAction sends a single key); they still get exercise_setup on every exercise switch.


In workouts and custom workouts launched with includePoseData containing "events", exercise_setup is sent as each exercise starts, with exercise set to the workout's exercise ID.

Request and Handle exercise_setup
Swift example:
1// Pass the same WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4// Ask for the setup of exercises you loaded
5webViewState.sendMessage([
6    "workout_activity_action": "get_exercise_setup",
7    "exercises": ["3", "394"]
8])
9
10// In onMessageReceived (exercise_setup arrives as .custom_type)
11if case .custom_type(let value) = message,
12   value["type"] as? String == "exercise_setup" {
13    let exercise = value["exercise"] as? String ?? ""
14    let tracked = value["tracked"] as? Bool ?? false
15    let cameraView = value["camera_view"] as? String // nil when not set
16    showSetupScreen(exercise: exercise, tracked: tracked, cameraView: cameraView)
17}

Preloading & Events

Two events fire as the component initializes. Wait for both before revealing the camera UI to the user.


EventMeaning
model_warmedupPose-tracking (MediaPipe) model is ready
models_loadedAll exercise models in exercises have downloaded

Pattern: mount the camera hidden (e.g. opacity: 0) with a loader on top. Reveal once both events have fired.


The two events can arrive in either order. If every exercise fails to load, models_loaded never arrives: you get error_occurred with severity: "critical" instead, so stop waiting on it. model_warmedup fires again if the view is re-entered (for example after the camera selector). Both also arrive when you launch with the camera off (pauseCamera: true, see Camera Off & Mute), so you can preload before the user needs the camera.

Wait for Both Events
Swift example:
1@State private var modelWarmedUp = false
2@State private var modelsLoaded = false
3var isReady: Bool { modelWarmedUp && modelsLoaded }
4
5kinestex.createCameraView(
6    exercises: ["3"],
7    currentExercise: $currentExercise,
8    user: nil,
9    style: nil,
10    isLoading: $isLoading,
11    onMessageReceived: { message in
12        if case .custom_type(let value) = message,
13           let type = value["type"] as? String {
14            if type == "model_warmedup" { modelWarmedUp = true }
15            if type == "models_loaded"  { modelsLoaded  = true }
16        }
17    }
18)
19.opacity(isReady ? 1 : 0)

Controls

Switching exercises in real time. Set currentExercise to any identifier from exercises (or one added later with load_models), in the form you picked with exerciseFetchType. Tracking switches at once, and you receive exercise_change for the exercise you left and exercise_setup for the new one (see Exercise Setup). Sending the value that is already active does nothing.


How each SDK sets `currentExercise`:


SDKExercise identifiersControl commands (below)
SwiftSet the currentExercise bindingSame
KotlinKinesteXSDK.updateCurrentExercise(id)Same
React Native, React (TypeScript)ref.changeExercise(id) (only identifiers listed in the launch exercises) or ref.sendAction("currentExercise", id)ref.sendAction("currentExercise", "Pause Exercise"): changeExercise rejects anything not in the launch exercises
FlutterKinesteXAIFramework.sendAction("currentExercise", id)Same
HTML/JSpostMessage({ currentExercise: id }, srcURL)Same

Control commands. Send any of these strings as currentExercise:


CommandEffectReply
"Pause Exercise"Stops rep counting and mistake detection. pose_landmarks / world_landmarks keep streaming. Send an exercise identifier to resumenone
"Pause Audio"Stops coach cues and rest speech; the rep sound still plays. feedback and mistake events are still sent. To silence everything, use mute_workout (Camera Off & Mute)none
"Resume Audio"Turns coach cues back onnone
"Workout Overview"Reports the session so far. Tracking continues on the current exerciseexercise_overview (only once an exercise has started) and workout_overview
"Stop Camera"Destructive. Releases the camera and the pose model and clears the view (see warning below)stop_camera

A command acts only when the value changes, so "Workout Overview" sent twice in a row replies once. Send the exercise identifier between two overviews.


Restart controls. Restart without re-mounting by sending these as workout_activity_action, the channel also used by load_models, get_exercise_setup, pause_camera / resume_camera and mute_workout / unmute_workout. Each one acts every time it is sent, including twice in a row.


workout_activity_actionEffectReply
"reset_progress"Clears the current exercise's reps, accuracy, mistakes and time. Tracking continues from rep 1; a rep in progress is discarded{ type: "reset_progress", exerciseId, date }
"reset_all_progress"Clears every exercise tracked this session, then restarts the current one from rep 1{ type: "reset_all_progress", exerciseId, date } (exerciseId is null if none started)
"reset_check_frame"Shows the positioning check again, also with showSilhouette: false. Reps pause until the user is back in position; a rep or hold in progress starts over{ type: "reset_check_frame", date }, then person_in_frame

Send reset_progress (or reset_all_progress) together with reset_check_frame for a full restart with a positioning check.


After a reset, the rep value (action kind rep, successful_repeat) counts from 1 again, and exercise_overview / exercise_change cover only the time since the reset. reset_progress keeps the other exercises in exercise_overview; reset_all_progress removes them. After "Pause Exercise", reset_progress resets the exercise that was active before the pause.


Send restart controls once the camera view is open (after model_warmedup is safe). Otherwise you get { type: "error_occurred", severity: "warning", message: "<action>: camera is not active" }, also after "Stop Camera". reset_progress before any currentExercise was set replies message: "reset_progress: no active exercise".


`"Stop Camera"` is destructive. Avoid it for normal flows. It releases the camera and the pose model, then fires stop_camera. Sending an exercise identifier as currentExercise afterwards reopens the camera and reloads the pose model (no re-mount, but slower than pause_camera / resume_camera). Send it only when the user is leaving the camera screen for good. For a temporary stop use "Pause Exercise"; to turn the camera off for a while use pause_camera (Camera Off & Mute); to start over use the restart controls.
Switch Exercise and Send Control Commands
Swift example:
1// Switch exercise (Swift uses two-way binding via @State)
2currentExercise = "394"
3
4// Pause / resume tracking
5currentExercise = "Pause Exercise"
6currentExercise = "3"
Restart Controls
Restart the current exercise, the whole session, or the positioning check without re-mounting. Use "reset_all_progress" or "reset_check_frame" in place of "reset_progress" the same way.
Swift example:
1// Pass a WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4webViewState.sendMessage(["workout_activity_action": "reset_progress"])

Camera Off & Mute

Turn the device camera off and on, and silence all KinesteX audio, at runtime without re-mounting the view. These are workout_activity_action commands, like the restart controls. Nothing changes unless you send them.


workout_activity_actionEffectEcho
"pause_camera"Releases the camera: the OS camera indicator goes off, no frame is captured or tracked, and the camera area is empty. The pose model stays loaded and warm, and the current exercise, rep counts and settings are kept{ type: "pause_camera", date }
"resume_camera"Reopens the same camera and keeps tracking the current exercise{ type: "resume_camera", date }, then camera_selected once the camera is open
"mute_workout"Silences every sound the component plays: the rep sound, coach cues and rest speech. Tracking and every event continue{ type: "mute_workout", date }
"unmute_workout"Turns audio back on. A "Pause Audio" you sent stays in effect{ type: "unmute_workout", date }

Starting with the camera off or muted. Launch with pauseCamera: true to open with the camera off, as if pause_camera had been sent before it opened: the exercise models and the pose model still load and warm up, so model_warmedup still arrives. Launch with muteWorkout: true to start muted. Both accept true or "true", send no echo, apply to the Camera Component only, and are not saved: the next launch starts with the camera and sound on unless you pass them again.


When to send them. Any time after kinestex_loaded, including before the camera opens. A pause_camera sent while the view is still loading keeps the camera from ever turning on. A mute_workout sent before the camera view opens is applied, and echoed, when it opens.


Notes:

  • Only resume_camera turns the camera back on. Switching currentExercise, the app returning from the background and a Retry tap all leave it off.
  • A pause_camera while the camera is already off, or a resume_camera while it is on, changes nothing but still echoes. In the second case no camera_selected follows, so do not wait for one.
  • After resume_camera, rep counts continue from the last value; a rep or hold that was half done starts over. If the user was not in position yet, the positioning check shows again. Send reset_check_frame right after resume_camera to make the user get back in position in any case.
  • timeSpent in exercise_change and exercise_overview includes time with the camera off.
  • mute_workout and "Pause Audio" are independent: "Pause Audio" stops coach cues and rest speech but keeps the rep sound. After "Pause Audio", a mute and an unmute bring back only the rep sound; coach cues return with "Resume Audio". The device volume and silent switch still apply on top.
  • pause_camera and resume_camera work in the Camera Component only. Other integrations reply { type: "error_occurred", severity: "warning", message: "pause_camera: only supported in the camera component" } (or resume_camera: ...). Workouts pause with pause_workout / resume_workout, which the Camera Component does not act on.

ControlCameraUse it for
currentExercise: "Pause Exercise"Stays on; the user still sees themselvesA short stop, such as a rest between sets
pause_cameraOff and released; the pose model stays warmThe camera must not run for a while (the user opted out of motion tracking, an exercise you don't track)
currentExercise: "Stop Camera"Off; the pose model is releasedThe user leaves the camera screen for good
Start With the Camera Off and Muted, Then Turn It On
Swift example:
1// Pass the same WebViewState to createCameraView(..., webViewState: webViewState)
2@StateObject private var webViewState = WebViewState()
3
4kinestex.createCameraView(
5    exercises: ["3"],
6    currentExercise: $currentExercise,
7    user: nil,
8    style: nil,
9    isLoading: $isLoading,
10    customParams: ["pauseCamera": true, "muteWorkout": true],
11    onMessageReceived: { message in
12        if case .custom_type(let value) = message,
13           let type = value["type"] as? String {
14            if type == "resume_camera" { cameraOn = true }
15            if type == "pause_camera" { cameraOn = false }
16        }
17    },
18    webViewState: webViewState
19)
20
21// Later, when the user needs the camera
22webViewState.sendMessage(["workout_activity_action": "resume_camera"])
23// Sound back on
24webViewState.sendMessage(["workout_activity_action": "unmute_workout"])

Customization

Pass any of these fields at initialization: in customParams (Swift, Kotlin, Flutter), in customParameters (React Native, React TS), or at the top level of the launch message (HTML/JS).


FieldTypeDescription
restSpeechesstring[]Audio phrases to preload (from ExerciseModel.rest_speech)
videoURLstringUse a video file instead of the live camera: the full tracking pipeline (reps, accuracy) runs on the video. Ideal for testing without physically performing exercises; see Testing & Simulation
landmarkColorstringPose overlay color in hex with # (default #14FF00)
showSilhouettebooleanShow "get into frame" silhouette (default true). With false, tracking starts at once, and framing_hint and person_in_frame fire only during a check you open with reset_check_frame
includeRealtimeAccuracyboolean(Beta) Stream live position-confidence alongside reps
includePoseDatastring[]Any of "angles", "poseLandmarks", "worldLandmarks". Performance impact: only enable for custom calculations. "angles" only fills the angle fields inside the landmark events. ("events" is for workouts; the Camera Component always sends its real-time events)
pauseCamerabooleanOpen with the camera off (default false). See Camera Off & Mute
muteWorkoutbooleanStart with all KinesteX audio muted (default false). See Camera Off & Mute
inactivityTimeoutnumberIdle seconds before activity INACTIVE (default 15). See Real-Time Events
includeFramingHintsbooleanTurns on framing_hint (default false)
includeFormStatebooleanTurns on form_state (default false)
enableGestureControlbooleanTurns on gesture_detected (default false). KinesteX reports the gesture; pausing is up to you
frameExitPausebooleanNo reps count while the user is out of view (presence LEFT) (default false)
frameExitPromptbooleanWith KinesteX audio on, plays the "Pause activated" prompt of KinesteX workouts on presence LEFT (default false)
currentRestSpeechstringPlays one of the restSpeeches now; "Stop audio" stops it. Needs restSpeeches at launch. Update it like currentExercise; the same value twice in a row plays once
showFormGuidancebooleanShows the live depth bar for exercises that have depth guidance (default false in the Camera component)
streamFormGuidancebooleanHides the bar and streams form_guidance_config / form_guidance_frame to your app instead
includePoseBordersbooleanOut-of-frame guard (default true): when elbows or knees leave the frame, the skeleton turns red, a "step back" cue plays and those frames are not tracked (no reps, no pose_landmarks)
isDrawingPosebooleanfalse hides the skeleton overlay; tracking is unchanged
videoFitstring"cover" (default) or "contain"
shouldShowCameraSelectorbooleanAdds a switch-camera button to the positioning check. Opening it leaves the camera view: on return model_warmedup and person_in_frame fire again and successful_repeat.value restarts at 1
hideConnectTvButton, checkFrameTextboolean, stringPositioning-check options, see UI Controls

Real-Time Events

The component reports what the user is doing, so you can build your own screens, prompts, overlays and voice-over on top of KinesteX motion tracking. The events reach the message handler you already pass; switch on type. State events fire on change, never on every frame, so each one maps straight to a UI state.


Try them live: the Camera Events Lab embeds the Camera Component with your company name and API key, plays sample videos (or uses your camera), sends every command with one click, and lists every event with its payload.


EventWhat it tells youKey fieldsTurn on with
presenceThe user left the picture (LEFT: nobody in view for 3 s) or came back (RETURNED: back in view for 1 s). Every LEFT gets its RETURNEDstatealways on
orientation_checkWhether the user faces the camera the way the exercise needs: the first reading for each exercise, then every change (held 0.6 s). Only for exercises whose camera_view is FRONT or SIDEexpected, detected (FRONT, SIDE or UNKNOWN), matchesalways on
framing_hintWhile the positioning check is open: which body parts are out of view and what would fix itmissing_body_parts, hint (STEP_BACK, MOVE_TO_CENTER or null)includeFramingHints: true
start_position_reachedThe user settled into the exercise's start position (again after every interruption and reset)exercisealways on
action kind repA rep was countedvalue, accuracy, rep_duration_msalways on
action kind holdA timed hold (plank, wall sit) started, broke or resumedstate (STARTED, BROKEN or RESUMED), held_msalways on
feedbackEvery form cue KinesteX detects, whether its audio is on, paused or muted. announce: true is a cue to voice now; announce: false is a live re-detection between announcementskind (mistake or tip), message, feedback_id, body_region, count, announcealways on
mistakeWhat KinesteX's coach says, in the older shapevalue, mistake_id, body_region, count, is_tipalways on
form_stateForm quality changedstate (GOOD, ADJUST or OUT_OF_POSITION)includeFormState: true
activityThe user has been idle for inactivityTimeout seconds (INACTIVE), or moved again (ACTIVE). Every INACTIVE gets its ACTIVEstate, idle_secondsalways on; inactivityTimeout (default 15)
gesture_detectedThe user made the hands-free gesture (arms down by the sides, then forearms crossed in front of the chest). KinesteX only reports it; pausing is up to yougesture (PAUSE or RESUME)enableGestureControl: true

Common fields. Every event has type and timestamp (milliseconds since the Unix epoch, from the device clock). The exercise events also carry exercise, the identifier as you sent it in currentExercise; framing_hint and gesture_detected do not.


When they run. presence, start_position_reached, action, form_state and activity follow the exercise being tracked: from the exercise you send in currentExercise until "Pause Exercise" or "Stop Camera" (after "Workout Overview" only action reps keep coming, until you send an exercise id), and not while the positioning check is open, the camera is off (pause_camera) or the app is in the background.


Handle one of each pair. action kind rep replaces successful_repeat: both are sent for every rep with the same fields and timestamp. feedback covers what mistake reports, and presence comes right after left_camera_frame / returned_camera_frame. Handle one of each pair, or every rep, cue or exit counts twice. New integrations: action, feedback and presence.


Launch options (details in Real-Time Motion Parameters):


OptionDefaultEffect
inactivityTimeout15Idle seconds before activity INACTIVE
includeFramingHintsfalseTurns on framing_hint
includeFormStatefalseTurns on form_state
enableGestureControlfalseTurns on gesture_detected
frameExitPausefalseNo reps count while the user is out of view (presence LEFT); a rep half done when they left starts over on RETURNED
frameExitPromptfalseWith KinesteX audio on, plays the "Pause activated" prompt of KinesteX workouts on presence LEFT
showSilhouettetrueWith false, tracking starts at once, and framing_hint fires only during a check you open with reset_check_frame

The boolean options also accept the string "true".


Forward compatibility. KinesteX adds to these events over time: new event types, new fields and new kind and state values. Ignore types, fields and values you don't know, and let your switch fall through to doing nothing, so a new KinesteX release never breaks your integration.


Field-by-field payloads and timings: Real-Time Motion Events. Workouts and custom workouts send the same events when launched with includePoseData containing "events" (see Custom Workout).

Handle Real-Time Events
Swift example:
1@State private var currentExercise = "3"
2@State private var reps = 0
3@State private var isAway = false
4@State private var turnPrompt: String?
5@State private var cue: String?
6@State private var showHelp = false
7
8kinestex.createCameraView(
9    exercises: ["3"],
10    currentExercise: $currentExercise,
11    user: nil,
12    style: nil,
13    isLoading: $isLoading,
14    customParams: [
15        "includeFramingHints": true,
16        "includeFormState": true,
17        "inactivityTimeout": 20
18    ],
19    onMessageReceived: { message in
20        // These events arrive as .custom_type; the event name is in "type"
21        guard case .custom_type(let data) = message,
22              let type = data["type"] as? String else { return }
23        switch type {
24        case "presence":
25            isAway = data["state"] as? String == "LEFT"
26        case "orientation_check":
27            // UNKNOWN says nothing about the way the user faces: don't prompt on it
28            if data["detected"] as? String != "UNKNOWN" {
29                let matches = data["matches"] as? Bool ?? true
30                turnPrompt = matches ? nil : data["expected"] as? String
31            }
32        case "start_position_reached":
33            showGo()
34        case "action":
35            if data["kind"] as? String == "rep" {
36                reps = data["value"] as? Int ?? reps
37            }
38        case "feedback":
39            if data["announce"] as? Bool == true {
40                cue = data["message"] as? String
41            }
42        case "activity":
43            showHelp = data["state"] as? String == "INACTIVE"
44        default:
45            break // types, kinds and states added later: ignore
46        }
47    }
48)

Event Reference

All events the Camera Component emits to the host app. Every event also carries timestamp (milliseconds since the Unix epoch, from the device clock). date fields use the format "dd mm yyyy hh:mm:ss".


EventPayloadWhen
kinestex_launched{ data }Your launch payload was verified
model_warmedup{ message }Pose model is ready
models_loaded{ message, modelIds? }Exercise models finished downloading. modelIds (the identifiers that resolved, as you sent them) is present when the event answers a fetch, including load_models, and missing on the "All models loaded" notice
camera_selected{ id, label, isMirrorCamera }The camera stream opened (at start, after the app returns from the background, after a camera switch, or after resume_camera)
person_in_frame{ message }The user passed the positioning check, at start and again after reset_check_frame. Not sent at start with showSilhouette: false
exercise_setup{ exercise, tracked, start_position, camera_view, alternate_camera_views, camera_side }On each switch to an exercise, and in reply to get_exercise_setup. See Exercise Setup
action{ kind: "rep", exercise, value, accuracy, rep_duration_ms? } or { kind: "hold", exercise, state, held_ms }A rep was counted, or a timed hold started, broke or resumed. The rep form replaces successful_repeat (same fields and timestamp, sent right after it): handle one of the two. See Real-Time Events
successful_repeat{ exercise, value, accuracy, rep_duration_ms? }A rep was counted. exercise is the active identifier as you sent it; value is that exercise's rep count this session (back to 1 after a reset); accuracy is the rep's peak position confidence; rep_duration_ms is the time since the previous rep, left out for the first rep of an exercise and the first rep after an interruption (back in view, camera back on, "Stop Camera", a reset, an exercise switch) or an idle stretch. Kept for existing integrations; action kind rep is its replacement
feedback{ exercise, kind, message, feedback_id, body_region, count, announce }Every form cue KinesteX detects, audio on or off. See Real-Time Events
mistake{ exercise, value, mistake_id, body_region, count, is_tip }What KinesteX's coach says (value is the cue text; is_tip marks a tip rather than a correction). With KinesteX's voice on, sent as the coach starts saying the cue; with audio paused or muted, when the cue is detected, once per cue per its repeat interval. It is not one event per frame. Handle feedback or mistake, not both
presence{ exercise, state }The user left the picture (LEFT) or came back (RETURNED)
left_camera_frame / returned_camera_frame{ date, exercise }Sent right before presence LEFT / RETURNED, in the shape KinesteX workouts use. Handle these or presence, not both
orientation_check, framing_hint, start_position_reached, form_state, activity, gesture_detectedsee Real-Time EventsUser state changes (framing_hint, form_state and gesture_detected only with their launch options)
correct_position_accuracy{ accuracy }(Beta) Live position confidence, only with includeRealtimeAccuracy: true
pose_landmarks{ poseLandmarks }Per frame, only when includePoseData includes "poseLandmarks". See Pose Data
world_landmarks{ worldLandmarks }Per frame, only when includePoseData includes "worldLandmarks". See Pose Data
form_guidance_config, form_guidance_frame, rep_depth_anglesee Camera & Frame EventsDepth guidance, with showFormGuidance or streamFormGuidance. id is the current exercise
speech_fetch_complete{ successCount, failureCount, modelIds? }Without modelIds: the restSpeeches that needed downloading finished (counts are phrases). With modelIds: feedback audio for those models finished loading; successCount is the number of models and failureCount is always 0
exercise_change{ prevExerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories }You switched currentExercise to a different exercise. Totals for the exercise you left; timeSpent counts from the first time you switched to it (or from its last reset), time on other exercises included. mistakes is [{ mistake, count }]
exercise_overviewdata: [{ exerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories }]Reply to "Workout Overview", one entry per exercise used this session. Not sent if none was started
workout_overviewdata: { totalReps, totalTimeSpent, totalAverageAccuracy, totalCalories }Reply to "Workout Overview", session totals (all 0 if no exercise started). Note the camelCase keys, unlike the workout player's workout_overview
reset_progress / reset_all_progress{ exerciseId, date }Echo of the restart control (exerciseId can be null for reset_all_progress)
reset_check_frame{ date }Echo; person_in_frame follows when the user is back in position
pause_camera / resume_camera{ date }Echo of the command, every time it is sent. See Camera Off & Mute
mute_workout / unmute_workout{ date }Echo of the command, every time it is sent (a command sent before the camera view opens is echoed when it opens)
error_occurred{ severity, message } or { severity, data }Any error (model fetch, phrase fetch, camera access, a restart control with nothing to act on, a get_exercise_setup for identifiers you never loaded). severity is "critical" or "warning"
warning{ data } or { message }Non-fatal issue (e.g. no model IDs provided, or no current exercise set)
stop_camera{ message }Confirms the "Stop Camera" command finished

The Camera component does not send exercise_completed, workout_progress or exit_kinestex.

Pose Data

When includePoseData contains "poseLandmarks" or "worldLandmarks", the camera emits one event per processed frame. Only enable this for custom calculations; there is a performance cost. No event is sent on frames with no person in view, or while the user is outside the frame with includePoseBorders on.


Payload:


json
1{
2  "type": "pose_landmarks",
3  "poseLandmarks": {
4    "coordinates": { "nose": { "x": 0.51, "y": 0.18, "z": -0.32, "visibility": 0.99 }, "...": {} },
5    "angles2D": { "leftKneeAngle": 172.4, "...": 0 },
6    "angles3D": { "leftKneeAngle": 168.9, "...": 0 }
7  },
8  "timestamp": 1793368803120
9}

world_landmarks has the same shape under worldLandmarks. Only the first detected person is reported. timestamp (milliseconds since the Unix epoch) is on every frame. The Camera Component's frames carry no exercise field; match them to the exercise you set in currentExercise.


In workouts. Workouts, plans, challenges and custom workouts take the same includePoseData values. Their frames also carry exercise (the workout's exercise ID, null during the positioning check), are sent during the positioning check and the exercises but not on rests, and always include 2D angles. "events" adds the real-time events there; the Camera Component always sends them, so it needs no "events". See Custom Workout: Pose Data & Real-Time Events.


Two coordinate spaces:


  • poseLandmarks: x and y normalized 0-1 to the camera frame. x is flipped (1 - x) to match the mirrored preview (the camera is mirrored by default), so a point on the right of the screen has a larger x. z is relative depth, not 0-1.
  • worldLandmarks: meters, origin between the hips, not flipped (best Z accuracy).

visibility is 0-1 in both. Landmark names use the person's own left and right.


Angles: "angles" adds no event of its own. It fills angles2D and angles3D inside the pose_landmarks / world_landmarks payloads (without it they are all 0, except angles2D, which enableGestureControl: true also fills), so pass it together with "poseLandmarks" or "worldLandmarks". Both keys are always present. Each payload's angles are computed from its own coordinates.


Available landmarks (same names in both spaces): nose, leftEyeInner, leftEye, leftEyeOuter, rightEyeInner, rightEye, rightEyeOuter, leftEar, rightEar, mouthLeft, mouthRight, leftShoulder, rightShoulder, leftElbow, rightElbow, leftWrist, rightWrist, leftPinky, rightPinky, leftIndex, rightIndex, leftThumb, rightThumb, leftHip, rightHip, leftKnee, rightKnee, leftAnkle, rightAnkle, leftHeel, rightHeel, leftFootIndex, rightFootIndex.


Available angles (filled when "angles" is included, in both angles2D and angles3D): leftKneeAngle, rightKneeAngle, leftHipAngle, rightHipAngle, leftShoulderAngle, rightShoulderAngle, leftElbowAngle, rightElbowAngle, leftWristAngle, rightWristAngle, leftAnkleAngle, rightAnkleAngle, leftArmpitAngle, rightArmpitAngle.

Complete Example

Minimal working example with Next / Previous buttons that cycle between exercises and a live rep counter in the UI.

Complete Implementation
Swift example:
1import SwiftUI
2import KinesteXAIKit
3
4struct CameraScreen: View {
5    let kinestex = KinesteXAIKit(
6        apiKey: "YOUR_API_KEY",
7        companyName: "YOUR_COMPANY_NAME",
8        userId: "YOUR_USER_ID"
9    )
10
11    // 3 = Squats, 394 = Jumping Jack
12    let exerciseIds = ["3", "394"]
13
14    @State private var index = 0
15    @State private var currentExercise = "3"
16    @State private var reps = 0
17    @State private var isLoading = false
18
19    var body: some View {
20        VStack(spacing: 16) {
21            Text("Reps: \(reps)")
22                .font(.title)
23                .padding(.top)
24
25            kinestex.createCameraView(
26                exercises: exerciseIds,
27                currentExercise: $currentExercise,
28                user: nil,
29                style: nil,
30                isLoading: $isLoading,
31                onMessageReceived: { message in
32                    // A counted rep: "action" of kind "rep" (arrives as .custom_type)
33                    if case .custom_type(let value) = message,
34                       value["type"] as? String == "action",
35                       value["kind"] as? String == "rep" {
36                        reps = value["value"] as? Int ?? 0
37                    }
38                }
39            )
40
41            HStack(spacing: 24) {
42                Button("Previous") { switchTo(index - 1) }
43                Button("Next")     { switchTo(index + 1) }
44            }
45            .padding(.bottom)
46        }
47    }
48
49    private func switchTo(_ newIndex: Int) {
50        index = (newIndex + exerciseIds.count) % exerciseIds.count
51        currentExercise = exerciseIds[index]
52        reps = 0
53    }
54}