AVMultiCamPiP: Capturing from Multiple Cameras

At a glance

Project structure

AVMultiCamPiP/
├── CameraViewController.swift   # Session graph, UI, routing, recording coordination
├── PiPVideoMixer.swift          # Metal compositing
├── PiPMixer.metal               # Compute kernel
├── MovieRecorder.swift          # AVAssetWriter wrapper
├── PreviewView.swift            # Preview-layer-backed UIView
├── Utilities.swift
├── Base.lproj/Main.storyboard
└── AppDelegate.swift

Structure observations

• The controller owns front/back video and microphone endpoints and their explicit connections.
• Rendering and file writing are extracted into concrete helper objects, but there is no app-defined protocol abstraction.
• Front/back preview views are storyboard-owned; recording uses mixed buffers rather than recording either preview layer.

Overall architecture

flowchart LR
    Controller["ViewController"] --> Session["AVCaptureMultiCamSession"]
    Session --> Front["Front video output"]
    Session --> Back["Back video output"]
    Session --> Audio["Front/back audio outputs"]
    Front --> Router["Sample routing"]
    Back --> Router
    Router --> Mixer["PiPVideoMixer"]
    Mixer --> Recorder["MovieRecorder"]
    Audio --> Recorder
    Recorder --> Writer["AVAssetWriter"]
    Writer --> Photos["PhotoKit save"]

Reference code

AVMultiCamPiP/CameraViewController.swift:869 — frame routing and composition

func captureOutput(_ output: AVCaptureOutput,
                   didOutput sampleBuffer: CMSampleBuffer,
                   from connection: AVCaptureConnection) {
    if let videoDataOutput = output as? AVCaptureVideoDataOutput {
        processVideoSampleBuffer(sampleBuffer, fromOutput: videoDataOutput)
    } else if let audioDataOutput = output as? AVCaptureAudioDataOutput {
        processsAudioSampleBuffer(sampleBuffer, fromOutput: audioDataOutput)
    }
}

Interpretation

The multi-camera session produces independent streams. The controller classifies each frame as fullscreen or PiP, retains the latest PiP frame, and asks the mixer to composite when a fullscreen frame arrives. Only mixed video and the selected microphone stream reach the recorder.

Ownership and state

classDiagram
    ViewController *-- AVCaptureMultiCamSession : creates
    ViewController *-- AVCaptureVideoDataOutput : front and back
    ViewController *-- AVCaptureAudioDataOutput : two connections
    ViewController *-- PiPVideoMixer : creates
    ViewController *-- MovieRecorder : creates per recording
    ViewController o-- PreviewView : storyboard outlets
    MovieRecorder *-- AVAssetWriter : creates per movie
    MovieRecorder *-- AVAssetWriterInput : audio and video

Ownership evidence

AVMultiCamPiP/CameraViewController.swift:272 and :702 — core owned objects

private let session = AVCaptureMultiCamSession()
private let dataOutputQueue = DispatchQueue(label: "data output queue")
private var movieRecorder: MovieRecorder?
private var currentPiPSampleBuffer: CMSampleBuffer?
private var videoMixer = PiPVideoMixer()

Class and protocol design

No app-defined protocol separates the three main concrete collaborators. The controller conforms directly to AVFoundation’s video and audio sample-buffer delegate protocols.

Access control

Reference code

AVMultiCamPiP/MovieRecorder.swift:13 — recorder boundary

private var assetWriter: AVAssetWriter?
private var assetWriterVideoInput: AVAssetWriterInput?
private var assetWriterAudioInput: AVAssetWriterInput?
private(set) var isRecording = false

No public, open, or fileprivate declaration appears in the Swift source.

Logic ownership and placement

Design patterns

Naming conventions

• Types: responsibility nouns (PiPVideoMixer, MovieRecorder, PreviewView); the main type uses generic ViewController despite the filename’s camera context.
• Methods: configuration names identify device side (configureBackCamera, configureFrontCamera) and processing stage (processFullScreenSampleBuffer).
• Properties: front... and back... prefixes make parallel graph endpoints explicit.
• State: Boolean names use is... or ...Enabled; queue names state their affinity.
• Files: one concrete helper per file; Metal kernel separated by implementation technology.

Architecture takeaways

• MultiCam requires explicit capture-port/connection ownership, which naturally concentrates graph coordination.
• Realtime routing, mixing, and recording share one data-output queue, making ordering assumptions visible.
• A small recorder wrapper prevents asset-writer lifecycle details from leaking through the controller.
• private(set) exposes status for coordination while preserving the component that may transition it.
• The design is concrete and sample-focused; it does not introduce protocols where only one implementation exists.

Source map

• Official Apple documentation