Rendering Gaussian splats with RealityKit

At a glance

Platform boundary: Apple’s documentation page lists Mac Catalyst. The verified project declares SUPPORTED_PLATFORMS = "iphoneos iphonesimulator macosx xros xrsimulator" and has no Mac Catalyst setting (RealityKit-GaussianSplats.xcodeproj/project.pbxproj:317). The code branches on os(macOS) || os(iOS) (RealityKit-GaussianSplats/WindowSplatView.swift:15). Front-matter platforms follow the official listing. Every macOS statement below is grounded in the native macosx destination.

Project structure

RealityKit-GaussianSplats/
├── RealityKitGaussianSplatsApp.swift   // scenes; ImmersiveSpace only on visionOS
├── AppModel.swift                      // @Observable shared state
├── ContentView.swift                   // two #if variants (visionOS / iOS+macOS)
├── SpatialSplatView.swift              // #if os(visionOS): immersive splat + snapping
├── SceneReconstructionManager.swift    // #if os(visionOS): ARKit meshes → collision
├── WindowSplatView.swift               // #if os(macOS) || os(iOS): turntable window
├── PLYLoader.swift                     // loadPLYEntity (Task.detached)
├── PLYParser.swift                     // nonisolated PLY parse + deinterleave
├── USDLoader.swift                     // loadUSDEntity via USDKit (Task.detached)
└── SplatEntityBuilder.swift            // GaussianSplatBuffers, makeSplatEntity
RealityKit-GaussianSplats.xcodeproj/
Configuration/SampleCode.xcconfig

Structure observations

• The files fall into two layers. Presentation is App, ContentView, SpatialSplatView, and WindowSplatView. Loading is PLYLoader, PLYParser, USDLoader, and SplatEntityBuilder. The loading layer has no types beyond value structs: its API is free functions such as loadPLYEntity(assetName:) and makeSplatEntity(from:isLinear:).
• Platform separation happens at the file level. SpatialSplatView.swift and SceneReconstructionManager.swift are entirely inside #if os(visionOS). WindowSplatView.swift is entirely inside #if os(macOS) || os(iOS) (RealityKit-GaussianSplats/SpatialSplatView.swift:16, RealityKit-GaussianSplats/SceneReconstructionManager.swift:13, RealityKit-GaussianSplats/WindowSplatView.swift:15). ContentView.swift holds two complete ContentView declarations separated by #else (RealityKit-GaussianSplats/ContentView.swift:13, RealityKit-GaussianSplats/ContentView.swift:67).
• Simulator builds compile throwing stubs instead of the GPU path: makeSplatEntity and loadUSDEntity sit behind #if targetEnvironment(simulator) (RealityKit-GaussianSplats/SplatEntityBuilder.swift:43, RealityKit-GaussianSplats/USDLoader.swift:25).
• The project uses SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor with SWIFT_VERSION = 5.0 (RealityKit-GaussianSplats.xcodeproj/project.pbxproj:319, RealityKit-GaussianSplats.xcodeproj/project.pbxproj:322). This is why parser functions are marked nonisolated.

Overall architecture

flowchart LR
    App["RealityKitGaussianSplatsApp"] -->|environment| Model["AppModel"]
    CV["ContentView"] -->|sets loadedAsset| Model
    Model -->|task id| Spatial["SpatialSplatView (visionOS)"]
    Model -->|task id| Window["WindowSplatView (iOS, macOS)"]
    Spatial --> Loaders["loadPLYEntity / loadUSDEntity"]
    Window --> Loaders
    Loaders --> Builder["makeSplatEntity"]
    Spatial --> Scene["SceneReconstructionManager"]

Reference code

RealityKit-GaussianSplats/RealityKitGaussianSplatsApp.swift:13 — RealityKitGaussianSplatsApp

struct RealityKitGaussianSplatsApp: App {
    @State private var appModel = AppModel()

    var body: some Scene {
        WindowGroup {
            ContentView()
                .environment(appModel)
        }
        // ...
        #if os(visionOS)
        ImmersiveSpace(id: appModel.immersiveSpaceId) {
            SpatialSplatView()
                .environment(appModel)
        }
        .immersionStyle(selection: .constant(.mixed), in: .mixed)
        #endif
    }
}

Interpretation

Observation: buttons never call a loader directly. They set AppModel.loadedAsset. The visible splat view reacts with .task(id: appModel.loadedAsset), clears its current entity, and loads the new one (RealityKit-GaussianSplats/SpatialSplatView.swift:100, RealityKit-GaussianSplats/WindowSplatView.swift:100). On visionOS, ContentView.load(_:) first opens the immersive space and sets loadedAsset only after .opened (RealityKit-GaussianSplats/ContentView.swift:42).

Inference: one enum value is the whole contract between the control UI and the renderer. That lets both platform views reuse the same trigger without a shared view model.

Ownership and state

classDiagram
    RealityKitGaussianSplatsApp *-- AppModel : @State
    ContentView --> AppModel : @Environment, mutates loadedAsset
    SpatialSplatView --> AppModel : @Environment, observes
    SpatialSplatView *-- SpatialSplatViewCoordinator : @State
    SpatialSplatViewCoordinator *-- SceneReconstructionManager : let sceneManager
    SpatialSplatViewCoordinator *-- EventSubscription : var releaseSubscription
    SceneReconstructionManager *-- ARKitSession : private let
    WindowSplatView *-- WindowSplatViewCoordinator : @State
    WindowSplatViewCoordinator *-- PerspectiveCamera : let cameraEntity

Ownership evidence

RealityKit-GaussianSplats/SpatialSplatView.swift:40 — state that must outlive view rebuilds

final class SpatialSplatViewCoordinator {
    let rootEntity = Entity()
    let sceneManager = SceneReconstructionManager()
    let spotLight = Entity()
    var releaseSubscription: EventSubscription?
    var attachedSplat: Entity?
    // ...
}

RealityKit-GaussianSplats/SpatialSplatView.swift:81 — the view stores it in @State

struct SpatialSplatView: View {
    @Environment(AppModel.self) private var appModel
    @State private var coordinator = SpatialSplatViewCoordinator()
    @State private var splatEntity: Entity?
    // ...
}

Note: the type names end in Coordinator, but the source describes them as holders for “RealityKit state that lives longer than one render of the view” (RealityKit-GaussianSplats/SpatialSplatView.swift:34, RealityKit-GaussianSplats/WindowSplatView.swift:52). They coordinate no navigation and no multi-object flow.

Concurrency, scheduling, and thread safety

Evidence limit: actor isolation, async/await, or Task creation does not by itself prove background-thread execution; Sendable conformance alone does not prove thread-safe mutation.

The target compiles with SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor (RealityKit-GaussianSplats.xcodeproj/project.pbxproj:319). Views, AppModel, both …Coordinator classes, SceneReconstructionManager, and the loader entry points are main-actor isolated, several also by explicit @MainActor. The parsing helpers opt out with nonisolated (RealityKit-GaussianSplats/PLYParser.swift:151, RealityKit-GaussianSplats/PLYParser.swift:285, RealityKit-GaussianSplats/USDLoader.swift:131).

No DispatchQueue, lock, or custom actor appears in the tree. Mutable shared state is limited to main-actor objects. The off-main work exchanges only freshly allocated buffers, returned through .value.

Inference on ordering: suppose the user switches assets while a parse is still running. The old .task is cancelled, but its detached parse keeps going, and the old loadAsset() can still assign splatEntity when it finishes. The source shows no guard against a slower earlier load finishing last.

Reference code

RealityKit-GaussianSplats/PLYLoader.swift:26 — loadPLYEntity(assetName:)

@MainActor
func loadPLYEntity(assetName: String) async throws -> Entity {
    let url = try bundlePLYURL(assetName: assetName)
    let buffers = try await Task.detached(priority: .userInitiated) {
        let splatData = try readGaussianSplatFile(url)
        return try deinterleaveGaussianSplatData(splatData)
    }.value
    return try makeSplatEntity(from: buffers)
}

RealityKit-GaussianSplats/SpatialSplatView.swift:100 — view-scoped tasks

.task(id: appModel.loadedAsset) {
    splatEntity = nil
    await loadAsset()
}
.task {
    await coordinator.sceneManager.run()
}

State propagation, frameworks, and dependencies

Evidence limit: an import proves a source-level compilation dependency at the cited line; it does not prove runtime use, architectural adoption, or whether a Swift package is a direct application dependency.

No Combine import, notification, or package manifest appears in the tree.

Class and protocol design

classDiagram
    class GaussianSplatBuffers {
        <<struct, unchecked Sendable>>
    }
    PLYLoader --> GaussianSplatBuffers : deinterleave
    USDLoader --> GaussianSplatBuffers : buildBuffers
    GaussianSplatBuffers --> SplatEntityBuilder : makeSplatEntity
    SplatEntityBuilder --> GaussianSplatComponent : assembles

Reference code

RealityKit-GaussianSplats/SplatEntityBuilder.swift:111 — the shared construction point

@MainActor
func makeSplatEntity(from buffers: GaussianSplatBuffers, isLinear: Bool = false) throws -> Entity {
    let comp = try assembleSplatComponent(from: buffers, isLinear: isLinear)
    let entity = Entity()
    entity.components.set(comp)
    return entity
}

No app-defined protocol exists. The two loaders converge on one value type and one builder function instead of sharing a loader protocol.

Access control

Reference code

RealityKit-GaussianSplats/SceneReconstructionManager.swift:29 — narrow internal surface

final class SceneReconstructionManager {
    // ...
    static let collisionGroup = CollisionGroup(rawValue: 1 << 2)

    let rootEntity = Entity()

    private let session = ARKitSession()
    private let provider = SceneReconstructionProvider()
    private var meshEntities: [UUID: Entity] = [:]
    // ...
}

Logic ownership and placement

Design patterns

Removed from the draft: Coordinator. The draft inferred it from the SpatialSplatViewCoordinator type name. The source documents these classes as state holders kept in @State across view rebuilds (RealityKit-GaussianSplats/SpatialSplatView.swift:34-38). They coordinate no navigation, flow, or collaborators.

Not claimed: Builder. SplatEntityBuilder.swift contains free functions that construct in one call, not an incremental builder object. Also not claimed: protocol-oriented design, since no app protocol exists.

Main application flow

sequenceDiagram
    actor User
    participant CV as ContentView
    participant Model as AppModel
    participant View as SpatialSplatView
    participant Loader as loadPLYEntity
    participant Detached as Task.detached
    User->>CV: Load PLY
    CV->>CV: openImmersiveSpace → .opened
    CV->>Model: loadedAsset = .ply
    Model-->>View: task(id:) restarts, splatEntity = nil
    View->>Loader: await loadPLYEntity
    Loader->>Detached: read + deinterleave
    Detached-->>Loader: GaussianSplatBuffers
    Loader-->>View: makeSplatEntity → Entity
    View->>View: update: syncSplat attaches, subscribes WillRelease

Reference code

RealityKit-GaussianSplats/SpatialSplatView.swift:109 — syncSplat(into:)

    private func syncSplat(into content: RealityViewContent) {
        if coordinator.attachedSplat === splatEntity { return }

        coordinator.attachedSplat?.removeFromParent()
        coordinator.releaseSubscription = nil
        coordinator.attachedSplat = nil

        guard let splatEntity else { return }

        configureManipulableObject(splatEntity)
        // ...
        coordinator.attachedSplat = splatEntity
        coordinator.releaseSubscription = content.subscribe(
            to: ManipulationEvents.WillRelease.self,
            on: splatEntity
        ) { event in
            snapToNearestSurface(event.entity)
        }
    }

Naming conventions

• Types: the Spatial… and Window… prefixes name the presentation surface (immersive space or window), not the platform. …ViewCoordinator names a view-owned reference holder (see the note above). …Manager owns an ARKit session. AppModel is the shared observable model. GaussianSplatBuffers and GaussianSplatData name data shapes. GaussianSplatError.invalidData(String) is the single error case.
• Functions: load…Entity(assetName:) is the async loader entry point. make… builds a value (makeSplatEntity, makeDescriptor, makeSHDescriptor). read…, parse…, deinterleave…, fill…Buffer, and copyPLYField are pipeline steps. configure… mutates an entity in place (configureSpotLight, configureManipulableObject). sync… reconciles state into the scene (syncSplat).
• State: immersiveSpaceState with closed/inTransition/open. loadedAsset with unloaded/ply/usd. settledSpin/spin distinguish the committed value from the in-drag value.
• Files: one primary type or format per file. SplatEntityBuilder.swift names a role but holds free functions. Rename from the earlier sample: SplatView → SpatialSplatView, SplatViewCoordinator → SpatialSplatViewCoordinator, GaussianSplatsOnVisionOSApp → RealityKitGaussianSplatsApp.

Architecture takeaways

• Use one observable enum (loadedAsset) as the contract between controls and renderer. .task(id:) turns each change into a restartable load without a dedicated view model.
• Keep RealityKit objects in a reference type stored in @State, and reconcile in RealityView’s update: by identity. Rebuilding the view then never recreates entities or subscriptions.
• Run CPU-heavy parsing as nonisolated functions inside an awaited Task.detached, and hand back an immutable buffer struct. @unchecked Sendable documents an invariant, it doesn’t enforce one, and detached work isn’t cancelled with the caller.
• Add platforms with whole-file #if boundaries and a same-named ContentView per branch. The loading layer stays shared, at the cost of small duplicated view logic.
• Read type suffixes against behavior. Here Coordinator means “view-lifetime state holder”, not the Coordinator pattern.

Source map

• Official Apple documentation