AVCamFilter: Applying filters to a capture stream

At a glance

Project structure

AVCamFilter/
├── CameraViewController.swift       # Capture graph, UI state, frame/photo coordination
├── FilterRenderer.swift             # Renderer capability and buffer-pool helpers
├── RosyMetalRenderer.swift
├── RosyCIRenderer.swift
├── DepthToGrayscaleConverter.swift
├── VideoMixer.swift
├── PreviewMetalView.swift
├── Shaders/                         # Metal kernels
├── minMaxFromBuffer.{h,m}           # Bridged depth utility
├── Base.lproj/Main.storyboard
└── AppDelegate.swift

Structure observations

• Capture and screen coordination remain in a large CameraViewController; GPU/Core Image algorithms are separate concrete objects.
• Each renderer owns its own buffer pool, format descriptions, and framework resources.
• Live and photo paths use separate renderer/mixer instances, avoiding shared mutable GPU preparation state across those paths.

Overall architecture

flowchart LR
    UI["CameraViewController"] --> Session["AVCaptureSession"]
    Session --> Video["Video data output"]
    Session --> Depth["Depth data output"]
    Session --> Photo["Photo output"]
    Video --> Filter["FilterRenderer strategy"]
    Depth --> Converter["DepthToGrayscaleConverter"]
    Filter --> Mixer["VideoMixer"]
    Converter --> Mixer
    Mixer --> Preview["PreviewMetalView"]
    Photo --> PhotoPipeline["Photo filter + depth mixer"]
    PhotoPipeline --> Photos["PhotoKit"]

Reference code

AVCamFilter/CameraViewController.swift:1040 — live-frame pipeline

var finalVideoPixelBuffer = videoPixelBuffer
if let filter = videoFilter {
    filter.prepare(with: formatDescription, outputRetainedBufferCountHint: 3)
    finalVideoPixelBuffer = filter.render(pixelBuffer: finalVideoPixelBuffer)!
}
// The optional depth mixer updates finalVideoPixelBuffer.
previewView.pixelBuffer = finalVideoPixelBuffer

Interpretation

AVFoundation delivers frames to the controller, which chooses and sequences renderer objects before handing the final buffer to a custom Metal view. The forced unwrap above abbreviates the source’s guarded result; the source handles failure before assignment.

Ownership and state

classDiagram
    CameraViewController *-- AVCaptureSession : creates
    CameraViewController *-- AVCaptureVideoDataOutput : creates
    CameraViewController *-- AVCaptureDepthDataOutput : creates
    CameraViewController *-- AVCapturePhotoOutput : creates
    CameraViewController *-- RosyMetalRenderer : live and photo instances
    CameraViewController *-- RosyCIRenderer : live and photo instances
    CameraViewController *-- DepthToGrayscaleConverter : live and photo instances
    CameraViewController *-- VideoMixer : live and photo instances
    CameraViewController o-- PreviewMetalView : storyboard outlet

Ownership evidence

AVCamFilter/CameraViewController.swift:62 — retained pipeline objects

private let session = AVCaptureSession()
private let videoDataOutput = AVCaptureVideoDataOutput()
private let depthDataOutput = AVCaptureDepthDataOutput()
private let photoOutput = AVCapturePhotoOutput()
private let filterRenderers: [FilterRenderer] = [RosyMetalRenderer(), RosyCIRenderer()]
private let videoDepthMixer = VideoMixer()
private let videoDepthConverter = DepthToGrayscaleConverter()

Class and protocol design

classDiagram
    class FilterRenderer {
        <<protocol>>
        prepare()
        render()
        reset()
    }
    RosyMetalRenderer ..|> FilterRenderer
    RosyCIRenderer ..|> FilterRenderer
    DepthToGrayscaleConverter ..|> FilterRenderer
    CameraViewController --> FilterRenderer : selects
    CameraViewController --> VideoMixer : composes depth
    CameraViewController --> PreviewMetalView : presents final buffer

Reference code

AVCamFilter/FilterRenderer.swift:10 — renderer capability

protocol FilterRenderer: AnyObject {
    var isPrepared: Bool { get }
    func prepare(with inputFormatDescription: CMFormatDescription,
                 outputRetainedBufferCountHint: Int)
    func reset()
    func render(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer?
}

Access control

Reference code

AVCamFilter/RosyMetalRenderer.swift:18 — externally readable renderer state

private(set) var inputFormatDescription: CMFormatDescription?
private(set) var outputFormatDescription: CMFormatDescription?
private var outputPixelBufferPool: CVPixelBufferPool?
private var computePipelineState: MTLComputePipelineState?

No public, open, or fileprivate declaration appears in the Swift source.

Logic ownership and placement

Design patterns

Naming conventions

• Types: operation-plus-role names (DepthToGrayscaleConverter, RosyMetalRenderer, VideoMixer).
• Protocol: capability role FilterRenderer; concrete types name technology where alternatives matter (Metal, CI).
• Methods: lifecycle verbs (prepare, render, reset) and event processing (processVideo, processDepth).
• State: ...Enabled, ...On, and isPrepared make Boolean scope explicit.
• Files: one renderer or visual component per file; shader filenames match their effects.

Architecture takeaways

• Keep capture coordination concrete while moving expensive transformation algorithms behind a small renderer protocol.
• Separate live and photo renderer instances when preparation state and buffer pools must not race.
• Use a custom render view when frames need modification before presentation.
• private(set) is useful for renderer lifecycle state that callers inspect but must not mutate.
• The controller remains large; the source optimizes sample visibility over a fully layered application architecture.

Source map

• Official Apple documentation