Tracking and visualizing hand movement

At a glance

Project structure

Source bundle/
└── RealityKit-HandTracking/
    ├── App/
    │   └── HandTracking.swift  # HandTracking
    ├── Components/
    │   └── HandTrackingComponent.swift  # HandTrackingComponent
    ├── Systems/
    │   └── HandTrackingSystem.swift  # HandTrackingSystem
    ├── Views/
    │   ├── HandTrackingView.swift  # HandTrackingView
    │   └── MainView.swift  # MainView
    └── Entities/
        ├── Bone.swift  # Bone
        ├── Finger.swift  # Finger
        └── Hand.swift  # Hand

Structure observations

• The runtime boundary is WindowGroup + ImmersiveSpace; the pruned tree lists only files that explain lifecycle, state, or framework integration.
• App code is split into role-named views, models, managers, providers, components, or systems.

Overall architecture

flowchart LR
    HandTracking_1["HandTracking"]
    WindowGroup___ImmersiveSpace_2["WindowGroup + ImmersiveSpace"]
    MainView_3["MainView"]
    HandTrackingSystem_4["HandTrackingSystem"]
    RealityView_entity_graph_5["RealityView entity graph"]
    ARKit_providers___RealityKit_6["ARKit providers + RealityKit"]
    HandTracking_1 --> WindowGroup___ImmersiveSpace_2
    WindowGroup___ImmersiveSpace_2 --> MainView_3
    MainView_3 --> HandTrackingSystem_4
    HandTrackingSystem_4 --> RealityView_entity_graph_5
    RealityView_entity_graph_5 --> ARKit_providers___RealityKit_6

Reference code

RealityKit-HandTracking/App/HandTracking.swift:11 — the app or executable entry declares the outer scene lifecycle.

struct HandTracking: App {
    // ...
}

The diagram is a responsibility flow, not a claim that every adjacent node directly calls the next. It keeps scene ownership, shared state, RealityKit content, and framework-provider work at separate levels.

Ownership and state

classDiagram
    HandTrackingSystem *-- ARKitSession : arSession
    HandTrackingSystem *-- HandTrackingProvider : handTracking
    HandTrackingComponent --> Chirality : chirality
    HandTrackingSystem --> HandAnchor : latestLeftHand

Ownership evidence

RealityKit-HandTracking/Systems/HandTrackingSystem.swift:13 — representative stored state or the nearest verified lifecycle anchor.

    static var arSession = ARKitSession()

    /// The provider instance for hand-tracking.
    static let handTracking = HandTrackingProvider()

    /// The most recent anchor that the provider detects on the left hand.
    static var latestLeftHand: HandAnchor?

Ownership here is deliberately narrow: @Environment and weak references are shared links, initialized @State or stored services are lifecycle ownership, and a RealityView content closure owns additions to its entity graph without making the SwiftUI view a reference-type owner.

Class and protocol design

The source defines no local substitution protocol in the reviewed boundary. Its protocol use is framework-facing (App, View, RealityKit/ARKit protocols, or platform adapters), so this document does not label the whole app protocol-oriented.

Access control

No reviewed declaration uses fileprivate, private(set), public, open; unmodified Swift declarations are internal.

Reference code

RealityKit-HandTracking/App/HandTracking.swift:11 — representative visibility boundary.

struct HandTracking: App {
    // ...
}

Logic ownership and placement

Design patterns

Naming conventions

• Role suffixes are evidence, not decoration: View: HandTrackingView, MainView.
• ECS names pair data and behavior: components HandTrackingComponent; systems HandTrackingSystem.
• Protocols: no app-defined protocol in the reviewed source.
• Commands use verb-led methods: runSession, update, addJoints, makeHandEntities.
• Files generally match their primary type; Views, Models, Managers, Providers, Components, Systems, and Packages folders describe architectural roles where present.

Architecture takeaways

• Treat HandTracking as the owner of scene declarations, not as the owner of every RealityKit entity created later.
• Keep view-local interaction in SwiftUI, but move provider sessions, playback resources, shared game state, or transport state into a stable owner when their lifetime exceeds one render pass.
• Run ARKit providers for the scene lifetime and consume their asynchronous updates in cancellable tasks; map anchors to entities at the boundary.
• Use RealityKit components for per-entity data and systems for repeated simulation instead of a monolithic view model.
• Model immersive-space open, transition, and close states explicitly so windows and immersive content cannot drift apart.

Source map

• Official Apple documentation