Supporting custom media formats and decoders

At a glance

Project structure

FormatReaderAndDecoder.xcodeproj   # 3 targets: app + 2 extensionkit-extension
FormatReaderAndDecoder/            # host app (Swift UI + ObjC exporter)
├── FormatReaderAndDecoderApp.swift
├── ContentView.swift              # VideoPlayer, drop target, export UI
├── MovieModel.swift               # @Observable owner; registration Task
├── MovieSavePanel.swift           # NSSavePanel ViewModifier
├── Info.plist                     # exports the .simpl UTType
└── RLEMovieExporter/              # ObjC: AVFoundation read -> RLE -> write
    ├── RLEMovieExporter.{h,m}     # pipeline + RLEMovieExporterDelegate
    ├── AssetVideoReader / AssetAudioReader
    ├── RLECompressor              # 2vuy pixels -> XRLE bytes
    ├── AssetSampleWriter          # QuickTime via AVAssetWriter
    └── SimpleContainerWriter      # .simpl file writer
SimpleContainerReader/             # format reader extension
├── Info.plist                     # EXPrincipalClass = ExtensionFormatReaderFactory
├── ExtensionFormatReader.{h,m}    # factory + MEFormatReader
├── ExtensionTrackReader.{h,m}     # METrackReader (+ video/audio/timecode)
├── ExtensionSampleCursor.{h,m}    # MESampleCursor (+ video/audio/timecode)
└── SimpleContainerReader.{h,m}    # container parser over MEByteSource
RLEVideoDecoder/                   # video decoder extension
├── Info.plist                     # EXPrincipalClass = ExtensionRLEDecoderFactory
├── ExtensionRLEDecoder.{h,m}      # factory + MEVideoDecoder
└── RLEDecompressor.{h,m}          # XRLE bytes -> 2vuy pixel buffer
Shared/
├── SimpleContainerFormat.h        # on-disk layout (writer <-> reader)
└── RLECodec.h                     # kRLECodecCode 'XRLE' (exporter <-> decoder)

Structure observations

• Target types: the app is com.apple.product-type.application (FormatReaderAndDecoder.xcodeproj/project.pbxproj:310). Both extensions are com.apple.product-type.extensionkit-extension (FormatReaderAndDecoder.xcodeproj/project.pbxproj:327, FormatReaderAndDecoder.xcodeproj/project.pbxproj:344).
• Embedding: the app depends on both extension targets (FormatReaderAndDecoder.xcodeproj/project.pbxproj:304, FormatReaderAndDecoder.xcodeproj/project.pbxproj:305). An “Embed ExtensionKit Extensions” copy phase places both .appex bundles in $(EXTENSIONS_FOLDER_PATH) (FormatReaderAndDecoder.xcodeproj/project.pbxproj:56).
• Source membership is strictly per target. The app compiles only the four Swift files and the six RLEMovieExporter/*.m files. The reader compiles only its four .m files. The decoder compiles ExtensionRLEDecoder.m and RLEDecompressor.m. No implementation file is shared.
• Shared/ contains headers only. SimpleContainerFormat.h is imported by the app’s writer (FormatReaderAndDecoder/RLEMovieExporter/SimpleContainerWriter.h:13) and the reader’s parser (SimpleContainerReader/SimpleContainerReader.h:13). RLECodec.h is imported by the exporter (FormatReaderAndDecoder/RLEMovieExporter/RLEMovieExporter.m:9) and the decoder (RLEVideoDecoder/ExtensionRLEDecoder.m:10). These two headers are the only compile-time coupling between targets.
• The app target is Swift 5 (FormatReaderAndDecoder.xcodeproj/project.pbxproj:503), deploys to macOS 15.0 (FormatReaderAndDecoder.xcodeproj/project.pbxproj:494), and bridges Objective-C through FormatReaderAndDecoder-Bridging-Header.h (FormatReaderAndDecoder.xcodeproj/project.pbxproj:501). That header imports only RLEMovieExporter.h (FormatReaderAndDecoder/RLEMovieExporter/FormatReaderAndDecoder-Bridging-Header.h:8).

Overall architecture

flowchart LR
    subgraph App["FormatReaderAndDecoder.app"]
        UI["ContentView"] --> Model["MovieModel"]
        Model --> Exporter["RLEMovieExporter (ObjC)"]
    end
    subgraph Reader["SimpleContainerReader.appex"]
        RF["ExtensionFormatReaderFactory"] --> FR["ExtensionFormatReader"]
    end
    subgraph Decoder["RLEVideoDecoder.appex"]
        DF["ExtensionRLEDecoderFactory"] --> DD["ExtensionRLEDecoder"]
    end
    Exporter -->|"writes .simpl / XRLE .mov"| Files[("Movie files")]
    Model -->|"AVPlayer + MT/VT registration"| Sys["System media stack"]
    Sys -->|"EXPrincipalClass, UTType simpl"| RF
    Sys -->|"EXPrincipalClass, CodecType XRLE"| DF
    Files --> Sys

Reference code

SimpleContainerReader/Info.plist:5 — the reader’s extension-point binding

<key>EXAppExtensionAttributes</key>
<dict>
    <key>ClassImplementationID</key>
    <string>com.example.apple-samplecode.formatreader.SimpleContainer</string>
    <key>EXExtensionPointIdentifier</key>
    <string>com.apple.mediaextension.formatreader</string>
    <key>EXPrincipalClass</key>
    <string>ExtensionFormatReaderFactory</string>
    <key>MTFileNameExtensionArray</key>
    <array>
        <string>simpl</string>
    </array>
    <key>MTUTTypeArray</key>
    <array>
        <string>com.example.apple-samplecode.simpleformat</string>
    </array>
    <!-- ... -->
</dict>

RLEVideoDecoder/Info.plist:9 — the decoder advertises the codec four-character code it handles

<key>CodecInfo</key>
<array>
    <dict>
        <key>CodecName</key>
        <string>MediaExtension RLE Video Codec</string>
        <key>CodecType</key>
        <string>XRLE</string>
    </dict>
</array>
<key>EXExtensionPointIdentifier</key>
<string>com.apple.mediaextension.videodecoder</string>
<key>EXPrincipalClass</key>
<string>ExtensionRLEDecoderFactory</string>

Interpretation

Observed: no target calls into another. The app never imports an extension header. The reader never references the decoder. The decoder knows only the kRLECodecCode constant. Each extension states what it handles declaratively: the reader by UTType and file extension, the decoder by CodecType. Each also names its principal class and carries an entitlement for its extension point (SimpleContainerReader/SimpleContainerReader.entitlements:5, RLEVideoDecoder/RLEVideoDecoder.entitlements:5).

Inference: the system media stack is the only broker between the two extensions. MediaExtension APIs do this routing: they choose the reader from the file’s type and the decoder from the format description’s codec type. Nothing in the sample’s source makes that selection. This is a plug-in architecture with the framework as host. The sample’s own composition is therefore three independently loadable products connected only by shared data contracts (file layout and codec code), not by calls.

Ownership and state

classDiagram
    ContentView *-- MovieModel : @State
    MovieModel *-- AVPlayer : let player
    MovieModel *-- Task : private let registration (detached)
    MovieModel *-- RLEMovieExporter : var exporter (per export)
    RLEMovieExporter o-- MovieModel : weak delegate
    ExtensionFormatReaderFactory ..> ExtensionFormatReader : creates per byte source
    ExtensionFormatReader *-- SimpleContainerReader : m_reader
    ExtensionFormatReader *-- ExtensionTrackReader : m_tracks
    ExtensionTrackReader o-- SimpleContainerReader : shared m_reader
    ExtensionSampleCursor o-- SimpleContainerReader : shared m_reader
    SimpleContainerReader o-- MEByteSource : m_byteSource
    ExtensionRLEDecoderFactory ..> ExtensionRLEDecoder : creates per format
    ExtensionRLEDecoder *-- RLEDecompressor : m_decoder
    ExtensionRLEDecoder o-- MEVideoDecoderPixelBufferManager : m_pixelBufferManager

Ownership evidence

FormatReaderAndDecoder/MovieModel.swift:13 — app-side state owner

@Observable class MovieModel: NSObject {
    // ...
    var exporter: RLEMovieExporter?
    // ...
    var inputURL: URL?
    // ...
    let player = AVPlayer()
    // ...
    var progressValue = 0.0
    // ...
    private let registration = Task.detached {
        VTRegisterProfessionalVideoWorkflowVideoDecoders()
        MTRegisterProfessionalVideoWorkflowFormatReaders()
    }
    // ...
}

SimpleContainerReader/ExtensionFormatReader.m:35 — one format reader per byte source parses eagerly and shares one container parser with its tracks

- (instancetype)initWithByteSource:(MEByteSource *)byteSource
                           options:(MEFormatReaderInstantiationOptions *)options
                             error:(NSError **)error {
    // ...
    self = [super init];
    if (self) {
        // Set up the reader.
        m_byteSource = byteSource;
        m_reader = [[SimpleContainerReader alloc] initWithByteSource:m_byteSource];
        // ...
        if (err == noErr) {
            // ...
            [self setupTracks];
            [self setupFileInfo];
            [self setupMetadata];
        }
    }
    // ...
    return self;
}

Who creates the factories is not visible in the source. The principal classes are named in each Info.plist and have only a plain init (SimpleContainerReader/ExtensionFormatReader.m:17, RLEVideoDecoder/ExtensionRLEDecoder.m:18). Inference: the extension host instantiates them.

Concurrency, scheduling, and thread safety

Evidence limit: actor isolation, async/await, or Task creation does not by itself prove background-thread execution. A Sendable conformance alone does not prove thread-safe mutation.

The app target is Swift 5 (FormatReaderAndDecoder.xcodeproj/project.pbxproj:503) and sets no default actor isolation. MovieModel carries no actor annotation, so its methods are nonisolated. Main-actor placement happens only where the source writes @MainActor.

Reference code

FormatReaderAndDecoder/MovieModel.swift:97 — the delegate callback arrives off the main actor and hops before touching observable state

func progressChanged(_ progress: Double) {
    // The exporter calls this from its own background queue, so update the observable
    // properties that the content view reads on the main actor.
    Task { @MainActor in
        // The content view observes `progressValue`, so the progress appears in the window.
        self.progressValue = progress

        // Releasing the exporter when the export finishes tells the content view to
        // hide the export controls overlay.
        if self.progressValue == 1 {
            self.exporter = nil
        }
    }
}

FormatReaderAndDecoder/RLEMovieExporter/RLEMovieExporter.m:86 — serial queue + group for the QuickTime path (abbreviated)

if (err == 0) {
    decodingQueue = dispatch_queue_create("com.example.apple-samplecode.FormatReaderAndDecoder.export.decode", DISPATCH_QUEUE_SERIAL);
    decodingTrackGroup = dispatch_group_create();

    __weak id<RLEMovieExporterDelegate> delegate = _delegate;

    // Handle the video track.
    dispatch_group_enter(decodingTrackGroup);
    [qtWriter videoRequestMediaDataWhenReadyOnQueue:decodingQueue
                                         usingBlock:^{
                                             // ...
                                         }];
}

Observations and inferences:

• Observed: the comment at FormatReaderAndDecoder/MovieModel.swift:98 says progress arrives on “its own background queue”. In the QuickTime path, intermediate progress does come from the serial decodingQueue block (FormatReaderAndDecoder/RLEMovieExporter/RLEMovieExporter.m:153). In the Simple Container path, the loop runs on whatever thread runs the Swift export task (FormatReaderAndDecoder/RLEMovieExporter/RLEMovieExporter.m:341). Both paths send the final progressChanged:1.0 from that calling thread (FormatReaderAndDecoder/RLEMovieExporter/RLEMovieExporter.m:228, FormatReaderAndDecoder/RLEMovieExporter/RLEMovieExporter.m:364). The @MainActor hop makes all of these safe for the UI.
• Inference: MovieModel is not actor-isolated, so the compiler does not enforce main-actor access to exporter. The export task reads exporter (FormatReaderAndDecoder/MovieModel.swift:54) at roughly the same time the main actor assigns it. The design relies on ordering, not on isolation.
• Inference: the export task blocks a Swift concurrency thread for the whole export, because writeRLE… is synchronous and itself waits on a dispatch group. The comment calls this “a background thread”. A dedicated queue would avoid tying up a cooperative-pool thread.

State propagation, frameworks, and dependencies

Evidence limit: an import proves a source-level compilation dependency at the cited line. It does not prove runtime use, architectural adoption, or whether a Swift package is a direct application dependency.

The project declares no Swift package. The app target’s link phase lists no framework. A MediaToolbox.framework file reference exists in the project but is not in any link phase.

Class and protocol design

classDiagram
    class MEFormatReaderExtension {
        <<protocol>>
    }
    class MEFormatReader {
        <<protocol>>
    }
    class METrackReader {
        <<protocol>>
    }
    class MESampleCursor {
        <<protocol>>
    }
    class MEVideoDecoderExtension {
        <<protocol>>
    }
    class MEVideoDecoder {
        <<protocol>>
    }
    ExtensionFormatReaderFactory ..|> MEFormatReaderExtension
    ExtensionFormatReader ..|> MEFormatReader
    ExtensionTrackReader ..|> METrackReader
    ExtensionVideoTrackReader --|> ExtensionTrackReader
    ExtensionAudioTrackReader --|> ExtensionTrackReader
    ExtensionTimecodeTrackReader --|> ExtensionTrackReader
    ExtensionSampleCursor ..|> MESampleCursor
    ExtensionVideoSampleCursor --|> ExtensionSampleCursor
    ExtensionAudioSampleCursor --|> ExtensionSampleCursor
    ExtensionTimecodeSampleCursor --|> ExtensionSampleCursor
    ExtensionRLEDecoderFactory ..|> MEVideoDecoderExtension
    ExtensionRLEDecoder ..|> MEVideoDecoder

Reference code

SimpleContainerReader/ExtensionFormatReader.h:13 — principal class and per-file reader conformances

@interface ExtensionFormatReaderFactory : NSObject <MEFormatReaderExtension>

- (instancetype)init;
- (id<MEFormatReader>)formatReaderWithByteSource:(MEByteSource *)primaryByteSource
                                         options:(MEFormatReaderInstantiationOptions *)options
                                           error:(NSError **)error;

@end

RLEVideoDecoder/ExtensionRLEDecoder.h:15 — decoder principal class

@interface ExtensionRLEDecoderFactory : NSObject <MEVideoDecoderExtension>
- (instancetype)init;
- (id<MEVideoDecoder>)videoDecoderWithCodecType:(CMVideoCodecType)codecType
                         videoFormatDescription:(CMVideoFormatDescriptionRef)videoFormatDescription
                     videoDecoderSpecifications:(NSDictionary<NSString *, id> *)videoDecoderSpecifications
             extensionDecoderPixelBufferManager:(MEVideoDecoderPixelBufferManager *)extensionDecoderPixelBufferManager
                                          error:(NSError **)error;
@end

Observed: MovieModel subclasses NSObject only to satisfy RLEMovieExporterDelegate <NSObject>, as its own comment says (FormatReaderAndDecoder/MovieModel.swift:12). The track readers and cursors use Objective-C subclassing for media-type differences. ExtensionFormatReader.setupFileInfo also tests exact classes with [meTrack class] == [ExtensionVideoTrackReader class] (SimpleContainerReader/ExtensionFormatReader.m:134).

Access control

Objective-C headers are not equivalent to Swift privacy. Every method declared in an @interface can be messaged by any code that imports the header. In this project, cross-target isolation comes from build membership, not from access modifiers.

Reference code

FormatReaderAndDecoder/MovieSavePanel.swift:15 — private modifier type behind an internal View extension

private struct MovieSavePanelModifier: ViewModifier {
    @Binding var isPresented: Bool
    let defaultFilename: String
    let contentTypes: [UTType]
    let onCompletion: (URL, UTType) -> Void
    // ...
}

Logic ownership and placement

Design patterns

Removed from the draft: “Protocol-oriented abstraction”. The protocols that shape the design (MEFormatReader, METrackReader, MESampleCursor, MEVideoDecoder, and their extension protocols) are framework contracts that the sample conforms to. They are not abstractions the sample designs or substitutes across. Its only local protocol is the single-method exporter delegate, and its variation uses Objective-C subclassing.

Main application flow

Playing a dropped .simpl movie: the scope is the source-visible handoffs. The system’s internal routing is shown as one participant.

sequenceDiagram
    actor User
    participant Model as MovieModel
    participant Sys as System media stack
    participant FR as ExtensionFormatReader
    participant Cur as ExtensionVideoSampleCursor
    participant Dec as ExtensionRLEDecoder
    User->>Model: performDrop(info:)
    Model->>Model: await registration.value (MainActor task)
    Model->>Sys: player.replaceCurrentItem(AVPlayerItem(url:))
    Sys->>FR: factory formatReaderWithByteSource: (UTType simpl)
    FR-->>Sys: track readers (format desc codec = file formatID)
    Sys->>Cur: generateSampleCursor... / sampleLocationReturningError:
    Cur-->>Sys: MESampleLocation (byte source, offset, length)
    Sys->>Dec: factory videoDecoderWithCodecType: XRLE
    Sys->>Dec: decodeFrameFromSampleBuffer:
    Dec-->>Sys: CVImageBuffer via completion handler

Reference code

SimpleContainerReader/ExtensionSampleCursor.m:264 — the reader hands back where the bytes are, not the decoded or even the compressed frame

- (MESampleLocation *)sampleLocationReturningError:(NSError **)error {
    int err;
    uint64_t offset, size;
    AVSampleCursorStorageRange storageRange;
    MESampleLocation *sampleLocation = nil;

    err = [m_reader getVideoSampleInfoAtTime:m_pts sampleOffset:&offset sampleSize:&size];
    if (err == noErr) {
        storageRange.offset = offset;
        storageRange.length = size;
        sampleLocation = [[MESampleLocation alloc] initWithByteSource:[m_reader byteSource] sampleLocation:storageRange];
    } else if (error != nil) {
        *error = [NSError errorWithDomain:MediaExtensionErrorDomain code:MEErrorLocationNotAvailable userInfo:nil];
    }

    return sampleLocation;
}

RLEVideoDecoder/ExtensionRLEDecoder.m:84 — the decoder receives a CMSampleBuffer and fills a system-managed pixel buffer

    if (sampleBuffer != NULL && m_decoder != nil && ![options doNotOutputFrame]) {
        // Create the output pixel buffer.
        outputPixelBuffer = [m_pixelBufferManager createPixelBufferAndReturnError:&error];
        // Decode the frame.
        err = [m_decoder decodeFrame:sampleBuffer toBuffer:outputPixelBuffer];
        if (err == noErr) {
            // ...
        } else {
            // ...
        }
    } else {
        // ...
    }

Reader → decoder data flow, as source-visible facts. (1) The exporter writes kRLECodecCode as the video track formatID. (2) The reader turns formatID into the track’s CMVideoFormatDescription codec type. (3) The reader returns sample byte ranges in an MESampleLocation. (4) The decoder’s Info.plist claims CodecType XRLE, and its factory re-checks kRLECodecCode. (5) The decoder consumes a CMSampleBuffer whose data begins with a 4-byte big-endian size (read in RLEDecompressor’s decodeFrame:toBuffer:). Inference: the system reads the located bytes, wraps them in a sample buffer with the reader’s format description, and selects the decoder by codec type. No sample code performs those steps. The same inference explains why an exported XRLE QuickTime movie needs only the decoder extension.

Naming conventions

• Types: the Extension… prefix marks classes that conform to MediaExtension protocols (ExtensionFormatReader, ExtensionTrackReader, ExtensionSampleCursor, ExtensionRLEDecoder). The …Factory suffix marks the EXPrincipalClass types. Media-type subclasses insert the medium (ExtensionVideoTrackReader, ExtensionTimecodeSampleCursor). Helpers without the prefix are format or codec logic with no framework role (SimpleContainerReader, RLEDecompressor, RLECompressor, SimpleContainerWriter). App-side Swift uses …Model, …View, and …Modifier.
• Protocols: the one local protocol uses the Cocoa …Delegate suffix (RLEMovieExporterDelegate).
• Methods: Objective-C methods follow Cocoa conventions: copy…/new… return retained Core Foundation objects (copyFormatDescriptionForMediaType:, newVideoFormatDescriptionFromSource:), …ReturningError: uses an error out-parameter, load…WithCompletionHandler: delivers asynchronously, and init… initializes. Container-parser methods return C int status codes. Swift methods are commands (exportMovie(to:as:), cancelMovieWriting()).
• Instance variables: an m_ prefix in Objective-C (m_reader, m_pixelBufferManager). The exporter’s ivars have no prefix.
• Identifiers and files: one {h,m} pair per primary class, with media-type subclasses in the same pair. Constants use k plus four-character codes (kRLECodecCode 'XRLE', kSimpleContainerMovieID 'smpl'). Bundle and type identifiers share the com.example.apple-samplecode. prefix.

Architecture takeaways

• A MediaExtension feature is a set of independent binaries joined by declarative metadata (EXPrincipalClass, UTTypes, CodecType, entitlements) and shared data contracts. Keep cross-target coupling to headers that define formats, as Shared/ does.
• Make the principal class a thin factory and put state in per-file or per-stream objects. In this sample the reader object owns the parser, and every track and cursor shares that parser.
• Readers describe and locate. Decoders transform. The reader returns MESampleLocation byte ranges and a format description whose codec type selects the decoder. It never decodes and never calls the decoder.
• On the host side, keep blocking framework calls (…Register…, synchronous export) off the main actor, and hop back with Task { @MainActor in } before mutating observable state. Without actor annotations on the model, that discipline is by convention, not checked by the compiler.
• For cancellation across a Swift/Objective-C boundary, a C11 atomic flag polled by the work loop is a minimal, explicit mechanism. Note that the sample’s Swift task handle is not retained, so the task itself is never cancelled.

Source map

• Official Apple documentation