KinesteX

Camera & Frame Events

Events about the camera, the positioning check, the user's place in the frame and the Camera component's commands. The real-time events for custom UIs (presence, start position, reps, holds, form, feedback, idle time) are on Real-Time Motion Events.


Positioning and frame


EventData FieldsDescription
person_in_framemessage: string ("Person stepped into frame")The user passed the positioning check and tracking starts (workout flows and the Camera component, also after reset_check_frame)
check_frame_skippedmessage: string ("User skipped check frame")User tapped Skip on the workout positioning screen
left_camera_framedate: string ("DD MM YYYY HH:MM:SS"), exercise?: stringThe user left the picture. Workout flows: the workout pauses and plays "Pause activated..." (no exercise field). Camera component: sent with exercise, straight before presence LEFT
returned_camera_framedate: string, exercise?: stringThe user came back. Workout flows: only after a pause caused by leaving the frame (not one the user or the hands gesture made); the workout resumes. Camera component: sent with exercise, straight before presence RETURNED

In the Camera component, left_camera_frame / returned_camera_frame and presence report the same moments: handle one of the two. In workouts they come from the workout's own pause logic, while presence (with includePoseData "events") is measured separately, so the two can fire at different moments, or one without the other.


Camera selection


EventData FieldsDescription
camera_selectedid: string, label: string, isMirrorCamera?: booleanThe camera stream opened (start, return from background, after a switch, after resume_camera), in workout flows, the Camera component, assessments and games. Also sent when the user picks a camera in the selector, without isMirrorCamera
camera_selector_openedmessage: Array<{ deviceId: string, label: string }>Camera selector opened, with the available cameras
camera_selector_closedid: string or null, isMirrorCamera: booleanCamera selector closed; the camera and mirror setting now in use
open_camera_settings-User tapped "Go to Settings" in the camera help screen. Only shown with shouldShowOpenCameraSettings: true. Open the device camera settings from your app

Camera component: commands and results


EventData FieldsDescription
exercise_changeprevExerciseId: string, reps: number, timeSpent: number, averageAccuracy: number, mistakes: Array<{ mistake, count }>, accuracyReps: number[], calories: numberYou switched currentExercise to another exercise: the totals of the one before. Top-level fields, no data. timeSpent in seconds since that exercise started
exercise_overviewdata: Array<{ exerciseId, reps, timeSpent, averageAccuracy, mistakes, accuracyReps, calories }>After currentExercise: "Workout Overview": one entry per exercise tracked
workout_overviewdata: { totalReps, totalTimeSpent, totalAverageAccuracy, totalCalories }Right after that exercise_overview. When nothing was tracked yet, only this event is sent, with all four at 0
stop_cameramessage: string ("Camera stopped")After currentExercise: "Stop Camera"
reset_progressexerciseId: string, date: stringEcho of reset_progress: the current exercise starts over
reset_all_progressexerciseId: string or null, date: stringEcho of reset_all_progress: every exercise starts over
reset_check_framedate: stringEcho of reset_check_frame. person_in_frame follows when the user is back in position
pause_camera / resume_cameradate: stringEcho, every time (also when sent before the camera page opened). A resume_camera that turns the camera back on is followed by camera_selected. Other integrations answer with an error_occurred warning
mute_workout / unmute_workoutdate: stringEcho. A command sent before the camera page opens is echoed when it opens, and two identical ones sent before that give one echo

exercise_setup (on each exercise switch and in reply to get_exercise_setup) is on Real-Time Motion Events; models_loaded and model_warmedup on Application Lifecycle. "Pause Exercise", "Pause Audio" and "Resume Audio" send no event of their own. pause_workout, resume_workout, mute_speech and unmute_speech are ignored by the Camera component (no echo, no error). The command errors ("<command>: camera is not active", "reset_progress: no active exercise") are on Error & Status Events.


Depth guidance and accuracy


EventData FieldsDescription
form_guidance_configid, source, combine, smoothing?, angles[]Depth-guidance setup for the current exercise. Only with streamFormGuidance: true
form_guidance_frameid, fill, target, tol, neutral, zone, anglesLive depth reading, sent when it changes. Only with streamFormGuidance: true
rep_depth_angleid, rep, combine, angles[]Deepest joint angle reached in each rep, whenever depth guidance is active: by default in workout flows (showFormGuidance defaults to true everywhere except the Camera component), for exercises whose model has depth guidance. In the Camera component, with showFormGuidance or streamFormGuidance
correct_position_accuracyaccuracy: numberEvery frame the motion model matches a position: how close the user is to the correct one (0 when the matched position is not a correct one). Workout flows and the Camera component, only with includeRealtimeAccuracy: true

Depth guidance stream. id is the exercise id (the workout exercise's id, or currentExercise in the Camera component). fill, target and neutral are 0-1 positions on the depth bar (0 = standing, 1 = deepest). tol is the half-width of the "perfect" band on the same scale (target plus or minus tol). zone is "neutral", "approaching", "perfect" or "too_deep". angles in a frame maps each joint id to its measured angle in degrees. Each entry of angles[] in the other two events describes one tracked joint: id, joint (knee, hip, elbow, shoulder, ankle), side, source ("2d" or "3d"), topAngle, neutralAngle, targetAngle, tolerance, bottomAngle and an optional weight. rep_depth_angle adds deepestAngle. See showFormGuidance and streamFormGuidance in Camera & Pose Detection.


Raw pose data (`includePoseData`)


Off by default. Pass includePoseData as a list (or a comma-separated string) with the values you need. It works in the Camera component and in workout flows (workouts, plans, personalized plans, custom workouts, challenges).


ValueEventDescription
"poseLandmarks"pose_landmarksposeLandmarks: the 33 MediaPipe landmarks as named keys (nose, leftShoulder, rightKnee, ...) under coordinates, each { x, y, z, visibility }, normalized to the picture (x flipped when the camera is mirrored, which is the default)
"worldLandmarks"world_landmarksworldLandmarks: the same object in metres, centred between the hips
"angles"inside bothFills angles3D (joint angles in 3D)
"events"see Real-Time Motion EventsWorkout flows only: the Camera component's real-time events (the Camera component always sends them)

Both landmark objects always carry angles2D and angles3D (leftKneeAngle, rightKneeAngle, hips, shoulders, elbows, wrists, ankles and armpits, left and right); values are 0 when not computed. angles3D is computed with "angles". angles2D is computed in workouts, and in the Camera component with "angles" or enableGestureControl. Events are sent for every frame with a person in it (up to 30 a second), so ask only for what you process. In workouts each event also has exercise: the exercise id the frame belongs to (the exercise_id of its exercise_completed), null during the positioning check; frames are sent during the positioning check and the exercises, not on rest screens. The Camera component's events have no exercise field.


json
1{ "type": "pose_landmarks", "exercise": "101", "poseLandmarks": { "coordinates": { "nose": { "x": 0.51, "y": 0.18, "z": -0.3, "visibility": 0.99 } }, "angles2D": { "leftKneeAngle": 172 }, "angles3D": { "leftKneeAngle": 0 } }, "timestamp": 1793368803120 }