Evaluating a Core AI model

At a glance

Project structure

CoreAIEvals.xcodeproj           # 2 targets: app + hosted unit-test bundle
CoreAIEvals/                    # app target (synchronized folder)
├── CoreAIEvalsApp.swift        # @main, one WindowGroup
├── Models/
│   └── ClassifierModel.swift   # @Observable screen state + Task lifecycle
├── Inference/
│   ├── ClassifierBackend.swift # .coreAI / .onDeviceDefault selector
│   ├── CoreAIModelStore.swift  # actor: load-once model cache
│   ├── SentimentClassifier.swift # stateless service (app + tests)
│   └── SentimentVerdict.swift  # @Generable output schema
└── Views/
    ├── SentimentClassifierView.swift
    ├── ResultView.swift
    └── VerdictView.swift
CoreAIEvalsTests/               # test target, @testable import CoreAIEvals
├── SentimentClassificationEvaluationTests.swift
└── SentimentDataset.swift      # 60 labeled ModelSample values
scripts/prepare-model.sh        # exports Models/coreai-model (not committed)

Structure observations

• The project has two native targets. CoreAIEvals is an application (CoreAIEvals.xcodeproj/project.pbxproj:133). CoreAIEvalsTests is a unit-test bundle (CoreAIEvals.xcodeproj/project.pbxproj:156) that depends on the app (CoreAIEvals.xcodeproj/project.pbxproj:146) and runs inside it through TEST_HOST (CoreAIEvals.xcodeproj/project.pbxproj:469).
• The test target links no package product of its own. It reaches app code through @testable import CoreAIEvals (CoreAIEvalsTests/SentimentClassificationEvaluationTests.swift:11). That is why the evaluation exercises the shipping classification path instead of a copy.
• The model is not source code. prepare-model.sh exports Qwen/Qwen3-0.6B into Models/coreai-model (scripts/prepare-model.sh:27, scripts/prepare-model.sh:28). The project adds that folder to the app’s Resources phase (CoreAIEvals.xcodeproj/project.pbxproj:206), and CoreAIModelStore looks it up by name at runtime.
• The folders reflect responsibility: Inference/ holds everything that talks to models, Models/ holds UI-facing state, and Views/ holds rendering only.

Overall architecture

flowchart LR
    App["CoreAIEvalsApp"] --> View["SentimentClassifierView"]
    View -->|"@State owns, @Bindable edits"| Model["ClassifierModel (@Observable)"]
    Model -->|"Task: classify(_:using:)"| Svc["SentimentClassifier (service)"]
    Tests["SentimentClassificationEvaluation"] -->|"subject(from:)"| Svc
    Svc -->|".coreAI"| Store["CoreAIModelStore (actor)"]
    Store --> CoreAI["CoreAILanguageModel"]
    Svc -->|".onDeviceDefault"| FM["LanguageModelSession (.default)"]

Reference code

CoreAIEvals/Inference/SentimentClassifier.swift:37 — makeSession(for:) is the single branch point between the two backends

private static func makeSession(for backend: ClassifierBackend) async throws -> LanguageModelSession {
    switch backend {
    case .coreAI:
        let coreAIModel = try await CoreAIModelStore.shared.model()
        return LanguageModelSession(model: coreAIModel, instructions: instructions)
    case .onDeviceDefault:
        return LanguageModelSession(model: .default, instructions: instructions)
    }
}

CoreAIEvalsTests/SentimentClassificationEvaluationTests.swift:113 — the evaluation calls the same entry point the UI uses

func subject(from sample: ModelSample<SentimentVerdict>) async throws -> ModelSubject<SentimentVerdict> {
    ModelSubject(value: try await SentimentClassifier.classify(sample.prompt, using: backend))
}

Interpretation

Observed: both the UI (CoreAIEvals/Models/ClassifierModel.swift:96) and the evaluation (CoreAIEvalsTests/SentimentClassificationEvaluationTests.swift:114) call SentimentClassifier.classify(_:using:). The only difference between the two measured configurations is the ClassifierBackend value. Both backends go through the same LanguageModelSession API, with the same instructions and the same @Generable output type.

Inference: this is the main architectural lesson of the sample. Because the classifier boundary is the single seam that both production code and evaluation code call, the evaluation scores exactly what the app ships. Comparing two models becomes a one-parameter change, not two code paths.

Ownership and state

classDiagram
    SentimentClassifierView *-- ClassifierModel : @State (created in init default)
    ClassifierModel *-- Phase : private(set) phase
    ClassifierModel *-- Task : private task (retained, cancelled)
    ClassifierModel --> SentimentClassifier : calls static classify
    SentimentClassifier --> CoreAIModelStore : shared.model()
    CoreAIModelStore *-- CoreAILanguageModel : private cached
    SentimentClassificationEvaluationTests *-- SentimentClassificationEvaluation : static let per backend

Ownership evidence

CoreAIEvals/Views/SentimentClassifierView.swift:12 — the view creates and owns the screen model

struct SentimentClassifierView: View {
    @State private var model: ClassifierModel

    /// Accepts a model so previews can supply one parked in a fixed phase;
    /// defaults to a live model for the running app.
    init(model: ClassifierModel = ClassifierModel()) {
        _model = State(initialValue: model)
    }
    // ...
}

CoreAIEvals/Inference/CoreAIModelStore.swift:29 — a process-wide actor owns the loaded model

actor CoreAIModelStore {
    static let shared = CoreAIModelStore()
    // ...
    private var cached: CoreAILanguageModel?
    // ...
}

Derived values are computed properties, not stored state. These are selectedBackend (CoreAIEvals/Models/ClassifierModel.swift:43), unavailableMessage (CoreAIEvals/Models/ClassifierModel.swift:52), and canClassify (CoreAIEvals/Models/ClassifierModel.swift:71). So the only lifecycle variable the view has to observe is phase.

Concurrency, scheduling, and thread safety

Evidence limit: actor isolation, async/await, or Task creation does not by itself prove background-thread execution. A Sendable conformance alone does not prove thread-safe mutation.

No DispatchQueue, lock, Task.detached, MainActor.run, Combine, or completion handler appears in the source. The project enables SWIFT_APPROACHABLE_CONCURRENCY on both targets (CoreAIEvals.xcodeproj/project.pbxproj:401, CoreAIEvals.xcodeproj/project.pbxproj:465) under Swift 6 (CoreAIEvals.xcodeproj/project.pbxproj:405). The sample does not depend on any behavior of those settings beyond what the table shows.

Reference code

CoreAIEvals/Models/ClassifierModel.swift:83 — cancel the previous task, publish .classifying, then retain the new task

func classify() -> Task<Void, Never> {
    task?.cancel()

    phase = .classifying
    let task = Task {
        // ...
        if let unavailableMessage {
            phase = .failed(unavailableMessage)
            return
        }
        do {
            let verdict = try await SentimentClassifier.classify(Prompt(text), using: selectedBackend)
            phase = .classified(verdict)
        } catch {
            phase = .failed(Self.message(for: error))
        }
    }
    self.task = task
    return task
}

CoreAIEvals/Inference/CoreAIModelStore.swift:42 — the load-once path

func model() async throws -> CoreAILanguageModel {
    if let cached { return cached }

    guard let url = Bundle.main.url(forResource: Self.modelResourceName, withExtension: nil) else {
        throw CoreAIModelStoreError.modelNotBundled(resourceName: Self.modelResourceName)
    }

    let model = try await CoreAILanguageModel(resourcesAt: url)
    cached = model
    return model
}

Observations and inferences on these boundaries:

• Observed: the task body checks neither Task.isCancelled nor which task is current. Inference: if a caller cancels a run that then finishes or throws, the old task can still write phase. In the shipped UI this cannot happen through the button: canClassify returns false while phase is .classifying (CoreAIEvals/Models/ClassifierModel.swift:72), and the button is disabled on that value (CoreAIEvals/Views/SentimentClassifierView.swift:37). The cancel() path matters only to programmatic callers of the returned task.
• Observed: CoreAIModelStore.model() checks cached before the await and assigns it after. Inference: two concurrent first callers could each start a load. The actor keeps cached free of data races, but it does not deduplicate in-flight loads. The source does not show whether the Evaluations framework calls subject(from:) concurrently.
• Inference: the explicit nonisolated on ClassifierBackend.displayName exists because the test target is nonisolated by default. The tests use the name synchronously when they build Evaluation.name and the @Test trait arguments.

State propagation, frameworks, and dependencies

Evidence limit: an import proves a source-level compilation dependency at the cited line. It does not prove runtime use, architectural adoption, or whether a Swift package is a direct application dependency.

The test code reaches the bundled model through Bundle.main, which works because the bundle is hosted in the app (TEST_HOST). Inference: this is why the tests need no resource or package setup of their own.

Class and protocol design

classDiagram
    class Evaluation {
        <<protocol>>
    }
    class SentimentClassificationEvaluation {
        backend: ClassifierBackend
        dataset
        evaluators
        subject(from:)
    }
    class SentimentClassifier {
        <<enum namespace>>
        classify(_:using:)
    }
    class ClassifierBackend {
        <<enum>>
        coreAI
        onDeviceDefault
    }
    SentimentClassificationEvaluation ..|> Evaluation
    SentimentClassificationEvaluation --> SentimentClassifier
    SentimentClassificationEvaluation --> ClassifierBackend
    ClassifierModel --> SentimentClassifier
    SentimentClassifier --> ClassifierBackend

Reference code

CoreAIEvalsTests/SentimentClassificationEvaluationTests.swift:37 — one evaluation type, parameterized by backend

struct SentimentClassificationEvaluation: Evaluation {
    let backend: ClassifierBackend

    var name: String { backend.displayName }

    let dataset = SentimentDataset.all

    /// Fraction of all labeled samples whose predicted sentiment matched the label.
    let accuracy = Metric("Accuracy")
    // ...
}

CoreAIEvals/Inference/SentimentVerdict.swift:23 — the guided-generation schema is also the evaluation’s expected-value type

@Generable
struct SentimentVerdict: Codable, Equatable {
    @Guide(description: "One short sentence explaining why this sentiment was chosen.")
    let reasoning: String
    @Guide(description: "How confident you are in the sentiment choice.", .range(0...100))
    let confidence: Int
    @Guide(description: "The overall sentiment: negative, neutral, or positive.")
    let sentiment: Sentiment
}

The only protocol conformances are to framework protocols: App, View, Evaluation, LocalizedError, Codable, and Equatable. The sample declares no local protocol. Backend selection uses an enum switch, not protocol substitution, so this analysis does not claim protocol-oriented design or a Strategy object.

Access control

The remaining app types are implicitly internal. The test target can see them only because it uses @testable import (CoreAIEvalsTests/SentimentClassificationEvaluationTests.swift:11). The sample declares nothing public or open.

Reference code

CoreAIEvals/Models/ClassifierModel.swift:34 — read-only lifecycle and private task handle

@Observable
final class ClassifierModel {
    // ...
    /// The current stage of the classification lifecycle.
    private(set) var phase: Phase = .idle

    /// Any in-flight classification task.
    private var task: Task<Void, Never>?
    // ...
}

Logic ownership and placement

Design patterns

Removed from the draft: “Central store”. CoreAIModelStore is a load-once cache for one framework object, not a store of application state, and it has no dispatch or reducer path. The Store suffix describes its caching role only. The sample is not labeled MVVM. ClassifierModel does play a screen-model role, but the source has only one view and one model and declares no view-model abstraction.

Main application flow

sequenceDiagram
    actor User
    participant View as SentimentClassifierView
    participant Model as ClassifierModel
    participant Svc as SentimentClassifier
    participant Store as CoreAIModelStore
    User->>View: Classify (Cmd-Return)
    View->>Model: classify()
    Model->>Model: cancel old task, phase = .classifying
    Model->>Svc: await classify(Prompt(text), using: backend)
    Svc->>Store: await shared.model() (coreAI only)
    Store-->>Svc: cached or newly loaded model
    Svc->>Svc: session.respond(generating: SentimentVerdict)
    Svc-->>Model: SentimentVerdict or error
    Model-->>View: phase = .classified / .failed (observed)

Reference code

CoreAIEvals/Inference/SentimentClassifier.swift:56 — session per call, typed result

    static func classify(
        _ prompt: Prompt,
        using backend: ClassifierBackend
    ) async throws -> SentimentVerdict {
        let session = try await makeSession(for: backend)
        let response = try await session.respond(to: prompt, generating: SentimentVerdict.self)
        return response.content
    }

The evaluation path is the same from Svc onward. Evaluations calls subject(from:) for each dataset sample, and the evaluators score subject.value against input.expected.

Naming conventions

• Types: role suffixes match observed roles. …App is the entry, …View renders, ClassifierModel holds UI state, CoreAIModelStore caches (a cache, not a state store), SentimentClassifier is the service, ClassifierBackend is the choice enum, SentimentVerdict is the result value, and …Evaluation/…Tests/…Dataset are the test roles. CoreAIModelStoreError pairs its error type with its owner.
• Protocols: no local protocol. Conformances use framework names (Evaluation, LocalizedError).
• Methods: verbs for commands (classify(), classify(_:using:)), make… for construction (makeSession(for:)), noun-style async accessors (model()), and can… for Boolean gates (canClassify). recallEvaluator(for:metric:) / precisionEvaluator(for:metric:) name what they build.
• Enum cases: lifecycle states read as adjectives or past participles (idle, classifying, classified, failed).
• Files and extensions: one primary type per file, named after it. The #if DEBUG preview factory sits in an extension in the model’s own file, so it can reach the private(set) setter.

Architecture takeaways

• Put the model call behind one seam (SentimentClassifier.classify(_:using:)) and have the UI and the evaluation both call it. Then the evaluation measures the shipping path, and comparing models is a one-enum-case change.
• Use the guided-generation output type (SentimentVerdict) as the evaluation’s expected type too. That way the schema, the UI, and the scoring share one definition.
• Target-wide MainActor default isolation removes most annotations. But it creates a boundary with a test target that is nonisolated by default, and the sample handles that with an explicit nonisolated member.
• A retained, cancellable Task plus a closed Phase enum is enough lifecycle machinery for a single-request screen. The UI guard (canClassify), not task identity, is what prevents overlapping runs.
• An actor makes a cache race-free, but not deduplicated. If you need single-flight loading, store the in-flight Task too.

Source map

• Official Apple documentation