Integration Options/Camera Component

Camera Component

KinesteX Motion Recognition: Real-Time Engagement.


  • Interactive Tracking: Advanced motion recognition for immersive fitness experiences
  • Real-Time Feedback: Instantly track reps, spot mistakes, and calculate calories burned
  • Boost Motivation: Keep users engaged with detailed exercise feedback
  • Custom Integration: Adapt camera placement to fit your app's design
Camera Integration
1@State var currentExercise = "Squats" // State variable for current exercise
2
3kinestex.createCameraView(
4    exercises: arrayAllExercises, // string array with exercise modelIDs or names
5    currentExercise: $currentExercise, // current exercise (starting)
6    user: nil,
7    isLoading: $isLoading,
8    onMessageReceived: { message in
9        switch message {
10        case .reps(let value):
11            reps = value["value"] as? Int ?? 0
12        case .mistake(let value):
13            mistake = value["value"] as? String ?? "--"
14        case .custom_type(let value):
15            guard let receivedType = value["type"] else { return }
16            if let typeString = receivedType as? String {
17                switch typeString {
18                case "models_loaded":
19                    print("All models loaded")
20                case "person_in_frame":
21                    withAnimation { personInFrame = true }
22                default:
23                    break
24                }
25            }
26        default:
27            break
28        }
29    }
30)
31
32// Update exercise in real-time:
33currentExercise = "Lunges"
34
35// Pause motion tracking:
36currentExercise = "Pause Exercise"

Configuration

Getting Exercise Data: Exercise model IDs (used in the exercises array and currentExercise), exercise names, and restSpeeches values can all be fetched from our Content API. Use the exercises endpoint to retrieve available exercises with their IDs and rest speech audio references.


Customization Options (customParams):

FieldDescription
restSpeeches[String] with rest_speech values from ExerciseModel API. Pass to fetch audios for use throughout the session
videoURLIf specified, will use video instead of camera
landmarkColorColor of pose connections and landmarks. Hex format with #. Example: "#14FF00"
showSilhouetteWhether to show silhouette prompting user to get into frame
includeRealtimeAccuracy(Beta) If true, creates stream returning accuracy prediction for correct rep value
includePoseDatastring[]. Can contain: ["angles", "poseLandmarks", "worldLandmarks"]. Will return 2D and 3D angles. **IMPACTS PERFORMANCE**

Available Message Types:

EventDescription
error_occurredIncludes data field with error message
warningWarning if exercise IDs models are not provided
successful_repeatIndicates success rep. Includes: exercise, value (total reps), accuracy (confidence)
person_in_frameIndicates person is in frame
speech_fetch_completeIncludes successCount and failureCount for loaded restSpeeches phrases
correct_position_accuracy(Beta) Includes accuracy field for confidence in correct exercise position
pose_landmarksIf specified in includePoseData. Includes coordinates, angles2D, angles3D
world_landmarksIf specified in includePoseData. Includes coordinates, angles2D, angles3D

Control Options:

  • Pause tracking: Set currentExercise to "Pause Exercise"
  • Resume tracking: Set currentExercise to any exercise from the exercises array
  • Update rest speech: Use currentRestSpeech to play audio from restSpeeches array

Complete Example

Full implementation with state management, loading indicators, and stats display. This example demonstrates a complete app with exercise tracking, loading states, and results display.

Complete Implementation
1import SwiftUI
2import KinesteXAIKit
3
4// Main app state management
5class ExerciseData: ObservableObject {
6    @Published var reps = 0
7    @Published var mistake = "--"
8    @Published var maxAccuracy = 0.0
9    @Published var currentAccuracy = 0.0
10
11    func reset() {
12        reps = 0
13        mistake = "--"
14        maxAccuracy = 0.0
15        currentAccuracy = 0.0
16    }
17}
18
19// CameraViewModel manages the state related to the camera component's lifecycle
20class CameraViewModel: ObservableObject {
21    @Published var modelWarmedUp = false
22    @Published var modelsLoaded = false
23    @Published var isReady = false
24    @Published var isLoading = false // Bound to KinesteXAIKit's camera view
25    @Published var shouldShowCamera = false
26
27    private var hasProcessedReady = false // Prevents multiple ready state triggers
28
29    // Processes custom messages from the KinesteX camera view
30    func processMessage(type: String) {
31        DispatchQueue.main.async { [weak self] in
32            guard let self = self else { return }
33
34            switch type {
35            case "model_warmedup": // Custom message indicating model warm-up completion
36                self.modelWarmedUp = true
37                self.checkAndSetReady()
38            case "models_loaded": // Custom message indicating all models are loaded
39                self.modelsLoaded = true
40                self.checkAndSetReady()
41            default:
42                break
43            }
44        }
45    }
46
47    // Checks if all conditions are met to set the camera view as ready
48    private func checkAndSetReady() {
49        guard !hasProcessedReady, modelWarmedUp, modelsLoaded else { return }
50
51        hasProcessedReady = true
52        DispatchQueue.main.asyncAfter(deadline: .now() + 0.3) {
53            withAnimation(.easeIn(duration: 0.5)) {
54                self.isReady = true
55            }
56        }
57    }
58
59    // Resets the ViewModel's state
60    func reset() {
61        DispatchQueue.main.async {
62            self.modelWarmedUp = false
63            self.modelsLoaded = false
64            self.isReady = false
65            self.isLoading = false
66            self.shouldShowCamera = false
67            self.hasProcessedReady = false
68        }
69    }
70
71    // Prepares and triggers the display of the camera component
72    func initializeCamera() {
73        reset()
74        DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
75            self.shouldShowCamera = true
76        }
77    }
78}
79
80// Main app structure
81struct ContentView: View {
82    @StateObject private var exerciseData = ExerciseData()
83    @StateObject private var cameraViewModel = CameraViewModel()
84    @State private var currentScreen: Screen = .start
85
86    // Required for createCameraView binding; set to the desired initial exercise ID
87    @State private var currentExerciseID: String = "CnOcLpBo5RAyznE0z3jt"
88    // List of exercise IDs to be available in the camera view
89    private let exerciseListForCamera: [String] = ["CnOcLpBo5RAyznE0z3jt"]
90
91    enum Screen {
92        case start
93        case camera
94        case results
95    }
96
97    // Initialize KinesteXAIKit with your credentials
98    private let kinesteXKit = KinesteXAIKit(
99        apiKey: "YOUR_API_KEY",
100        companyName: "YOUR_COMPANY_NAME",
101        userId: "YOUR_USER_ID"
102    )
103
104    var body: some View {
105        ZStack {
106            switch currentScreen {
107            case .start:
108                StartPage {
109                    currentScreen = .camera
110                    cameraViewModel.initializeCamera()
111                }
112            case .camera:
113                VStack(spacing: 10) {
114                    HStack {
115                        Spacer()
116                        Button(action: {
117                            cameraViewModel.reset()
118                            DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
119                                currentScreen = .results
120                            }
121                        }) {
122                            HStack {
123                                Image(systemName: "xmark")
124                                Text("Finish")
125                            }
126                            .padding(10)
127                            .background(Color.black.opacity(0.6))
128                            .foregroundColor(.white)
129                            .cornerRadius(8)
130                        }
131                    }
132                    .padding(.horizontal)
133                    .padding(.top, 5)
134
135                    ZStack {
136                        if cameraViewModel.shouldShowCamera {
137                            cameraView
138                                .opacity(cameraViewModel.isReady ? 1.0 : 0.0)
139                                .animation(.easeInOut(duration: 0.5), value: cameraViewModel.isReady)
140                        }
141
142                        if !cameraViewModel.isReady && cameraViewModel.shouldShowCamera {
143                            loadingOverlay
144                        }
145                    }
146                    .frame(
147                        width: UIScreen.main.bounds.width * 0.9,
148                        height: UIScreen.main.bounds.height * 0.55
149                    )
150                    .cornerRadius(15)
151                    .shadow(radius: 5)
152
153                    statsView
154                        .padding(.top, 5)
155
156                    Spacer()
157                }
158                .padding(.top)
159
160            case .results:
161                ResultsPage(exerciseData: exerciseData) {
162                    exerciseData.reset()
163                    cameraViewModel.reset()
164                    currentScreen = .start
165                }
166            }
167        }
168        .animation(.easeInOut, value: currentScreen)
169    }
170
171    private var cameraView: some View {
172        kinesteXKit.createCameraView(
173            exercises: exerciseListForCamera,
174            currentExercise: $currentExerciseID,
175            user: nil,
176            isLoading: $cameraViewModel.isLoading,
177            customParams: [
178                "includeRealtimeAccuracy": true,
179            ],
180            onMessageReceived: { message in
181                switch message {
182                case .reps(let value):
183                    DispatchQueue.main.async {
184                        exerciseData.reps = value["value"] as? Int ?? exerciseData.reps
185                        exerciseData.maxAccuracy = value["accuracy"] as? Double ?? exerciseData.maxAccuracy
186                    }
187                case .mistake(let value):
188                    DispatchQueue.main.async {
189                        exerciseData.mistake = value["value"] as? String ?? "--"
190                    }
191                case .custom_type(let value):
192                    guard let received_type = value["type"] as? String else { return }
193                    cameraViewModel.processMessage(type: received_type)
194
195                    if received_type == "correct_position_accuracy" {
196                        DispatchQueue.main.async {
197                            exerciseData.currentAccuracy = value["accuracy"] as? Double ?? 0.0
198                        }
199                    }
200                case .exit_kinestex(_):
201                    cameraViewModel.reset()
202                    currentScreen = .start
203                default:
204                    break
205                }
206            }
207        )
208    }
209
210    private var loadingOverlay: some View {
211        VStack {
212            ProgressView()
213                .scaleEffect(1.5)
214                .padding()
215
216            Text("Loading Exercise...")
217                .font(.headline)
218                .padding(.top, 5)
219
220            VStack(alignment: .leading, spacing: 8) {
221                HStack {
222                    Image(systemName: cameraViewModel.modelWarmedUp ? "checkmark.circle.fill" : "circle")
223                        .foregroundColor(cameraViewModel.modelWarmedUp ? .green : .gray)
224                    Text("Model Warm-up")
225                        .foregroundColor(cameraViewModel.modelWarmedUp ? .green : .primary)
226                }
227
228                HStack {
229                    Image(systemName: cameraViewModel.modelsLoaded ? "checkmark.circle.fill" : "circle")
230                        .foregroundColor(cameraViewModel.modelsLoaded ? .green : .gray)
231                    Text("Loading Models")
232                        .foregroundColor(cameraViewModel.modelsLoaded ? .green : .primary)
233                }
234            }
235            .padding(.top, 15)
236
237            if cameraViewModel.modelWarmedUp && cameraViewModel.modelsLoaded {
238                Text("Initializing Camera...")
239                    .foregroundColor(.green)
240                    .padding(.top, 8)
241            }
242        }
243        .frame(maxWidth: .infinity, maxHeight: .infinity)
244        .background(Color.black.opacity(0.8))
245    }
246
247    private var statsView: some View {
248        VStack(spacing: 8) {
249            HStack(spacing: 10) {
250                StatsCard(title: "REPS", value: "\(exerciseData.reps)")
251                StatsCard(title: "MAX ACCURACY", value: "\(Int(exerciseData.maxAccuracy))%", color: .green)
252            }
253
254            HStack(spacing: 10) {
255                StatsCard(title: "CURRENT ACCURACY", value: "\(Int(exerciseData.currentAccuracy))%", color: .cyan)
256
257                if exerciseData.mistake != "--" && !exerciseData.mistake.isEmpty {
258                    StatsCard(title: "MISTAKE", value: exerciseData.mistake, color: .red)
259                } else {
260                    StatsCard(title: "MISTAKE", value: "--", color: .gray)
261                }
262            }
263        }
264        .padding(.horizontal)
265    }
266}
267
268// Helper Views
269struct StartPage: View {
270    let onStartTapped: () -> Void
271
272    var body: some View {
273        VStack(spacing: 40) {
274            Spacer()
275            Text("Exercise Tracker")
276                .font(.system(size: 36, weight: .bold))
277            Image(systemName: "figure.run")
278                .font(.system(size: 100))
279                .foregroundColor(.blue)
280            Spacer()
281            Button(action: onStartTapped) {
282                Text("Start Exercise")
283                    .font(.headline)
284                    .foregroundColor(.white)
285                    .padding()
286                    .frame(width: 250, height: 60)
287                    .background(Color.blue)
288                    .cornerRadius(15)
289                    .shadow(radius: 5)
290            }
291            Spacer()
292        }
293        .padding()
294    }
295}
296
297struct StatsCard: View {
298    let title: String
299    let value: String
300    var color: Color = .white
301
302    var body: some View {
303        VStack(alignment: .center, spacing: 4) {
304            Text(title)
305                .font(.caption.weight(.medium))
306                .textCase(.uppercase)
307                .foregroundColor(.gray)
308            Text(value)
309                .font(.title2.weight(.bold))
310                .foregroundColor(color)
311                .lineLimit(1)
312                .minimumScaleFactor(0.7)
313        }
314        .padding(12)
315        .frame(maxWidth: .infinity, minHeight: 70)
316        .background(Material.ultraThinMaterial)
317        .cornerRadius(12)
318    }
319}
320
321struct ResultsPage: View {
322    @ObservedObject var exerciseData: ExerciseData
323    let onDoneTapped: () -> Void
324
325    var body: some View {
326        VStack(spacing: 30) {
327            Text("Exercise Summary")
328                .font(.system(size: 32, weight: .bold))
329                .padding(.top, 50)
330            Spacer()
331            VStack(spacing: 20) {
332                ResultCard(
333                    icon: "flame.fill",
334                    title: "Total Reps",
335                    value: "\(exerciseData.reps)",
336                    color: .orange
337                )
338                ResultCard(
339                    icon: "target",
340                    title: "Max Accuracy",
341                    value: "\(Int(exerciseData.maxAccuracy))%",
342                    color: .green
343                )
344            }
345            .padding(.horizontal, 20)
346            Spacer()
347            Button(action: onDoneTapped) {
348                Text("Back to Start")
349                    .font(.headline)
350                    .foregroundColor(.white)
351                    .padding()
352                    .frame(width: 250, height: 60)
353                    .background(Color.blue)
354                    .cornerRadius(15)
355                    .shadow(radius: 5)
356            }
357            .padding(.bottom, 50)
358        }
359        .frame(maxWidth: .infinity, maxHeight: .infinity)
360        .background(Color(UIColor.systemGroupedBackground).edgesIgnoringSafeArea(.all))
361    }
362}
363
364struct ResultCard: View {
365    let icon: String
366    let title: String
367    let value: String
368    let color: Color
369
370    var body: some View {
371        HStack(spacing: 20) {
372            Image(systemName: icon)
373                .font(.system(size: 30))
374                .foregroundColor(color)
375                .frame(width: 50)
376            VStack(alignment: .leading, spacing: 2) {
377                Text(title)
378                    .font(.callout)
379                    .foregroundColor(.secondary)
380                Text(value)
381                    .font(.title.weight(.semibold))
382                    .foregroundColor(.primary)
383            }
384            Spacer()
385        }
386        .padding()
387        .background(Material.regularMaterial)
388        .cornerRadius(15)
389    }
390}
391
392#Preview {
393    ContentView()
394}

Need Help?

Our team is ready to assist with your integration.

Contact Support
KinesteX

Personal AI Fitness Trainer & Motion Analysis SDK for Health & Fitness Apps

Contacts

hello@kinestex.com

Supported by

AMKM Investments supporting KinesteXin5 Tech
© 2024. All rights reserved. KinesteX
KinesteX TwitterKinesteX Inst