Sample CodeiOS 12.0+, iPadOS 12.0+, Mac Catalyst 12.0+Reviewed 2026-07-21View on Apple Developer

Streaming depth data from the TrueDepth camera

At a glance

Item	Summary
Purpose	Capture synchronized front-camera video and depth, then render a JET depth overlay or 3D point cloud.
App architecture	A storyboard-backed `CameraViewController` owns capture, synchronization, processing, and view coordination; renderer types isolate GPU work.
Main patterns	UIKit MVC, AVFoundation delegate, renderer strategy protocol, queued producer/consumer pipeline.
Project style	Flat camera/controller and renderer files, with separate PointCloud Objective-C++ and Metal shader folders.

Project structure

TrueDepthStreamer/
├── CameraViewController.swift
├── FilterRenderer.swift
├── DepthToJETConverter.swift
├── VideoMixer.swift
├── PreviewMetalView.swift
├── PointCloud/                 # Objective-C++ 3D renderer
├── Shaders/                    # Metal kernels
└── Base.lproj/Main.storyboard

This is a demonstration-oriented MVC target: the controller is the integration point, while expensive rendering algorithms are separate types and shader files.

Overall architecture

flowchart LR
    VC["CameraViewController"] --> Session["AVCaptureSession"]
    Session --> VideoOut["Video output"]
    Session --> DepthOut["Depth output"]
    VideoOut --> Sync["DataOutputSynchronizer"]
    DepthOut --> Sync
    Sync --> Converter["DepthToJETConverter"]
    Converter --> Mixer["VideoMixer"]
    Mixer --> Preview["PreviewMetalView"]
    Sync --> Cloud["PointCloudMetalView"]

Reference code

TrueDepthStreamer/CameraViewController.swift:442 — the controller establishes the synchronized boundary.

outputSynchronizer = AVCaptureDataOutputSynchronizer(
    dataOutputs: [videoDataOutput, depthDataOutput]
)
outputSynchronizer!.setDelegate(self, queue: dataOutputQueue)
session.commitConfiguration()

The synchronizer emits matched video/depth pairs on dataOutputQueue. The delegate selects either a 2D conversion/mix path or the point-cloud path.

Ownership and state

classDiagram
    CameraViewController *-- AVCaptureSession
    CameraViewController *-- AVCaptureVideoDataOutput
    CameraViewController *-- AVCaptureDepthDataOutput
    CameraViewController *-- DepthToJETConverter
    CameraViewController *-- VideoMixer
    CameraViewController --> PreviewMetalView
    CameraViewController --> PointCloudMetalView

Ownership evidence

TrueDepthStreamer/CameraViewController.swift:41 — retained capture and processing components (abridged).

private let session = AVCaptureSession()
private let sessionQueue = DispatchQueue(label: "session queue")
private let dataOutputQueue = DispatchQueue(label: "video data queue", qos: .userInitiated)
private let videoDataOutput = AVCaptureVideoDataOutput()
private let depthDataOutput = AVCaptureDepthDataOutput()
private let videoDepthMixer = VideoMixer()
private let videoDepthConverter = DepthToJETConverter()

Owner	Object or state	Relationship	Mutation authority
`CameraViewController`	Capture session, inputs, outputs	Creates and retains	Session configuration is dispatched to `sessionQueue`.
`CameraViewController`	Converter and mixer	Creates and retains	Delegate callback prepares and invokes them on `dataOutputQueue`.
`PreviewMetalView`	Current pixel buffer and Metal resources	Receives processed buffers	Its private sync queue protects render-facing state.
`PointCloudMetalView`	3D view state	Storyboard-owned outlet	Controller sends frames and gestures; Objective-C++ view mutates rendering state.

Class and protocol design

classDiagram
    class FilterRenderer {
        <<protocol>>
        prepare()
        render()
        reset()
    }
    DepthToJETConverter ..|> FilterRenderer
    CameraViewController --> DepthToJETConverter

Reference code

TrueDepthStreamer/FilterRenderer.swift:10 — a capability protocol defines renderer lifecycle and buffer contracts (abridged).

protocol FilterRenderer: AnyObject {
    var isPrepared: Bool { get }
    func prepare(with inputFormatDescription: CMFormatDescription,
                 outputRetainedBufferCountHint: Int)
    func reset()
    var outputFormatDescription: CMFormatDescription? { get }
    func render(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer?
}

DepthToJETConverter is the concrete strategy. VideoMixer has a similar lifecycle but does not conform to the protocol because its operation consumes two buffers.

Access control

Symbol	Access	Verified effect	Likely rationale
Controller session, outputs, queues	`private`	Other target code cannot mutate the pipeline components directly.	Preserves queue and configuration ordering.
`DepthToJETConverter.inputFormatDescription`	`private(set)`	Consumers can inspect prepared formats but only the converter changes them.	Exposes renderer output contract without exposing mutation.
`SessionSetupResult`	nested `private enum`	Setup states exist only inside the controller.	Keeps a local state machine from becoming app API.
`FilterRenderer`	internal	Available across the app target, not exported as a framework API.	The target has one reusable rendering capability boundary.

TrueDepthStreamer/DepthToJETConverter.swift:97:

class DepthToJETConverter: FilterRenderer {
    var isPrepared = false
    private(set) var inputFormatDescription: CMFormatDescription?
    private(set) var outputFormatDescription: CMFormatDescription?
    private var outputPixelBufferPool: CVPixelBufferPool!
}

Logic ownership and placement

Logic	Owning type or file	Placement rationale
Authorization, session lifecycle, routing	`CameraViewController`	UIKit lifecycle and capture lifecycle are coordinated in one demo controller.
Video/depth synchronization	`AVCaptureDataOutputSynchronizer` delegate in the controller	Rejects dropped or unmatched frames before GPU work.
Depth color conversion	`DepthToJETConverter` + `DepthToJET.metal`	Encapsulates buffer pools and compute pipeline resources.
Video/depth compositing	`VideoMixer` + `Mixer.metal`	Keeps two-input GPU composition separate from capture.
Point-cloud rendering	`PointCloud/` + `PointCloud.metal`	Isolates Objective-C++/Metal 3D concerns from Swift capture code.

Design patterns

Pattern	Source evidence	Purpose or tradeoff
MVC	`TrueDepthStreamer/CameraViewController.swift:15`	One controller coordinates UIKit, capture, and model-like state; simple but broad responsibility.
Delegate	`TrueDepthStreamer/CameraViewController.swift:733`	AVFoundation pushes synchronized frame pairs to the controller.
Strategy/capability protocol	`TrueDepthStreamer/FilterRenderer.swift:10`	Standardizes single-buffer renderer preparation and reset.
Queue confinement	`TrueDepthStreamer/CameraViewController.swift:148`	Keeps blocking/mutable session work off the main queue.

Naming conventions

Rendering types describe transformations: DepthToJETConverter, VideoMixer, PreviewMetalView.
Methods use lifecycle verbs (prepare, reset, configureSession) and event names for delegate callbacks.
Queue names describe owned work: sessionQueue, dataOutputQueue, syncQueue.
Metal and Objective-C++ files are grouped by implementation technology rather than Swift type role.

Architecture takeaways

Synchronize independent capture outputs before doing expensive rendering.
Keep AVFoundation session mutation on one serial queue and frame processing on another.
A small renderer protocol captures lifecycle and buffer-pool contracts without abstracting the whole pipeline.
This sample intentionally centralizes orchestration in one view controller; it is not a layered app architecture.

Source map

Source file	Relevant symbols
`TrueDepthStreamer/CameraViewController.swift`	Session owner, synchronizer delegate, routing
`TrueDepthStreamer/FilterRenderer.swift`	`FilterRenderer`, buffer-pool helper
`TrueDepthStreamer/DepthToJETConverter.swift`	Depth conversion strategy
`TrueDepthStreamer/VideoMixer.swift`	Video/depth compositor
`TrueDepthStreamer/PointCloud/PointCloudMetalView.mm`	3D renderer

Official Apple documentation