Enhancing live video by leveraging TrueDepth camera data

At a glance

Project structure

TrueDepthBackdrop/
├── CameraViewController.swift  # UI, session, synchronization, image processing
├── PreviewMetalView.swift      # Thread-safe image handoff and Metal render loop
├── Shaders/
│   └── PassThrough.metal       # Vertex and fragment functions
├── Base.lproj/Main.storyboard  # View hierarchy and outlets
└── AppDelegate.swift

Structure observations

• CameraViewController.swift is deliberately broad: it is the feature controller, capture owner, delegate, and image-processing coordinator.
• GPU presentation is separated into PreviewMetalView, which hides Metal objects and draw-loop state.
• There is no view model, service protocol, or separate domain layer.

Overall architecture

flowchart LR
    Storyboard["Storyboard"] --> Controller["CameraViewController"]
    Controller --> Session["AVCaptureSession"]
    Session --> Outputs["Video + depth + face outputs"]
    Outputs --> Sync["AVCaptureDataOutputSynchronizer"]
    Sync --> Controller
    Controller --> CI["Depth mask + Core Image blend"]
    CI --> Preview["PreviewMetalView"]
    Preview --> Metal["Metal pipeline + shaders"]

Reference code

TrueDepthBackdrop/CameraViewController.swift:407 and TrueDepthBackdrop/CameraViewController.swift:622 — synchronized callback boundary

outputSynchronizer = AVCaptureDataOutputSynchronizer(
    dataOutputs: [videoDataOutput, depthDataOutput, metadataOutput]
)
outputSynchronizer!.setDelegate(self, queue: dataOutputQueue)

func dataOutputSynchronizer(
    _ synchronizer: AVCaptureDataOutputSynchronizer,
    didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection
) { /* build the mask and composited image */ }

Interpretation

The view controller is the application-level owner and receives one synchronized collection per processing step. It transforms the depth buffer into a matte, blends the color frame with the selected background, then hands a CIImage to the specialized renderer.

Ownership and state

classDiagram
    Storyboard *-- CameraViewController : instantiates
    CameraViewController *-- AVCaptureSession : creates
    CameraViewController *-- AVCaptureVideoDataOutput : creates
    CameraViewController *-- AVCaptureDepthDataOutput : creates
    CameraViewController *-- AVCaptureMetadataOutput : creates
    CameraViewController *-- AVCaptureDataOutputSynchronizer : creates
    CameraViewController o-- PreviewMetalView : weak outlet
    PreviewMetalView *-- CIContext : creates
    PreviewMetalView *-- MTLCommandQueue : creates

Ownership evidence

TrueDepthBackdrop/CameraViewController.swift:18 — controller-owned graph and storyboard-owned view

@IBOutlet weak private var previewView: PreviewMetalView!
private let session = AVCaptureSession()
private let sessionQueue = DispatchQueue(label: "session queue")
private let dataOutputQueue = DispatchQueue(label: "video data queue")
private let videoDataOutput = AVCaptureVideoDataOutput()
private let depthDataOutput = AVCaptureDepthDataOutput()
private let metadataOutput = AVCaptureMetadataOutput()

Class and protocol design

TrueDepthBackdrop/CameraViewController.swift:14 — framework callback composition

class CameraViewController: UIViewController,
    AVCaptureDataOutputSynchronizerDelegate,
    UIImagePickerControllerDelegate,
    UINavigationControllerDelegate {
    // ...
}

This is protocol-oriented at framework callback boundaries, not through app-defined capabilities.

Access control

Reference code

TrueDepthBackdrop/PreviewMetalView.swift:21 — narrow input with private synchronized copy

var image: CIImage? {
    didSet {
        syncQueue.sync { internalImage = image }
    }
}
private var internalImage: CIImage?
private let syncQueue = DispatchQueue(label: "Preview View Sync Queue")

Logic ownership and placement

Design patterns

Naming conventions

• Types name framework role: CameraViewController and PreviewMetalView.
• Capture properties name the exact framework object: videoDataOutput, depthDataOutput, outputSynchronizer.
• State Booleans use is/past-participle forms: isSessionRunning, renderingEnabled.
• Methods use lifecycle/action verbs: configureSession, sessionWasInterrupted, chooseBackground, configureMetal.

Architecture takeaways

• Make one queue-confined owner responsible for an AVCaptureSession graph.
• Synchronize heterogeneous outputs before applying per-frame business logic.
• Keep the Metal draw loop behind a small renderer input surface.
• A large controller is understandable for a focused sample, but production growth would justify extracting capture and image-processing services.

Source map

• Official Apple documentation