02
Scenario-based admission criteria
Dependency builds and repeatable tests
Select one existing OCI image and several representative dependency tasks. Do not begin with a synthetic “hello world” job. Choose a job that currently consumes meaningful CI capacity and produces an artifact used by a later stage.
Record the following evidence:
- The exact image reference and digest used by the job.
- The registry authentication method and the configuration source.
- The dependency lockfiles and build inputs.
- The produced artifact checksum or another deterministic identity.
- The behavior of the local cache on a cold run, a consecutive run, and after the node restarts.
- The logs from a failed pull, failed build, and interrupted task.
- The result of repeating the same job under the same declared inputs.
Do not publish cold-start time, throughput, or performance ratios unless the team has a retained test record. The acceptance question is whether the output and failure behavior are repeatable, not whether one successful run looks fast.
A workload can leave the native macOS workspace only when its inputs are explicit, its output is portable, and its cache does not silently depend on a developer’s host directory. If a dependency build requires an undocumented path or an interactive credential, keep it on the current worker until the dependency flow is redesigned.
Scenario score: 5/5 only when the image, inputs, output identity, cache behavior, and repeat execution are all documented.
Untrusted pull requests and task isolation
Treat an untrusted pull request as hostile even when the repository is internal. The test should attempt to discover or access:
- Host directories outside the declared workspace.
- SSH agent sockets and forwarded credentials.
- Sensitive environment variables.
- Previous task files in mounted paths.
- Other task processes or residue.
- Unexpected published ports.
- Writable paths that survive task deletion.
Build the test around the project’s documented controls. Use a read-only root filesystem where the workload allows it, restrict mounts to the minimum required paths, run as a non-root user, and apply path masking or other documented path-obscuring controls where appropriate. The official security and capability guidance must be checked against the exact release under evaluation before a control is accepted.
A passing test needs more than a configuration file. Preserve the command, the effective runtime configuration, the attempted access, the observed denial, and the cleanup result. Run the same test after a previous job deliberately creates files and exports non-sensitive marker variables. This checks whether isolation survives task reuse rather than only a clean first launch.
If the test fails, apply one of three dispositions:
- Tighten permissions, mounts, capabilities, or credential handling and repeat the test.
- Move the job to a disposable, one-time workspace with explicit cleanup.
- Remove that workload from the shared Apple Silicon node and keep it on an isolated worker pool.
Do not place production signing tasks in the same pool as untrusted Linux jobs merely because both jobs can technically start. Signing credentials, provisioning assets, and Keychain access require a separate trust decision.
Scenario score: 5/5 requires a successful hostile-job test, documented cleanup, and no unapproved host or credential exposure.
Private registries and internal dependencies
Network acceptance must distinguish three paths:
- Runtime networking: connections made by the running container.
- Image-pull networking: access to the private OCI registry and its authentication endpoint.
- BuildKit build networking: access needed while the image or dependency layer is being built.
One successful curl from a running container does not validate all three paths. Test each path under the same proxy, DNS, firewall, and certificate rules used by the intended CI worker.
The evidence package should include:
- Registry and proxy configuration sources.
- DNS resolution results for internal dependencies.
- Certificate and trust-store behavior.
- The policy for build-time secrets.
- A controlled test of required outbound and inbound ports.
- Failure logs with credentials removed.
- Credential revocation followed by a retest.
- Results from a restricted-network environment.
Do not copy long-lived tokens into an image layer, build argument, or persistent workspace. The test should prove that a revoked credential can no longer pull the private dependency and that the old secret is absent from logs and output layers.
Network success also needs a lifecycle check. Run a job with the proxy available, repeat it with the proxy unavailable, and confirm that the failure is clear rather than silently falling back to an unintended public endpoint. If the build can use public mirrors when policy requires private sources, the job is not ready for production.
Scenario score: 4/5 is the minimum pilot threshold; production requires separate evidence for runtime, pull, and build networks.
Shared-node concurrency and resource contention
Concurrency should be tested with representative jobs, not an arbitrary container count. Select the largest normal Linux job, a typical dependency job, and a native macOS pipeline that uses the same host. Run them under the planned scheduling policy and observe whether the container workload changes the macOS pipeline’s queue behavior, build stability, disk pressure, or network reliability.
Measure and retain the actual records for:
- CPU pressure and memory pressure.
- Disk consumption from layers, caches, logs, and workspaces.
- Network saturation or throttling events.
- Cache growth and cleanup behavior.
- Native macOS job failures while container work is active.
- Queue delay and task cancellation behavior.
These observations must come from the team’s own test records. The official documentation can establish available controls, but it cannot prove capacity for a particular CI workload.
Use the result to choose one deployment model:
- Dedicated node: appropriate when container jobs are untrusted, bursty, disk-heavy, or likely to interfere with signing.
- Shared node with strict quotas: acceptable only when resource limits, cleanup, and failure behavior are proven.
- Separate pools: preferred when Linux support tasks and macOS signing tasks have different trust or availability requirements.
The documented volume and mount behavior should be reviewed before cache directories or workspaces are placed on shared storage. A cache that improves one job but grows without a tested cleanup policy is an operational liability.
Scenario score: 3/5 is not a production approval. Resource ownership, cleanup, and pool routing must be explicit.
Restart, upgrade, and unattended recovery
A CI node is not ready because a service starts once. Test the events that remove human intervention from the recovery path:
- Reboot the Apple Silicon Mac during an idle period and confirm that the intended container service and CI agent return to the expected state.
- Interrupt a running task and verify that the scheduler records failure rather than success.
- Stop or disrupt the container service and confirm that queued and active jobs receive a clear status.
- Fill the approved test volume to the documented operational threshold, then verify cleanup and recovery.
- Remove unused images and caches according to the release-supported procedure.
- Upgrade from the currently approved release to the candidate release in a disposable environment.
- Re-run image pulls, private dependencies, isolation tests, artifact checks, and native macOS jobs after the upgrade.
- Exercise the documented rollback path and preserve the resulting logs.
The release boundary matters. The project’s main branch may contain work that is not part of the formal 1.3.0 release. Acceptance records must name the release tag, host macOS version, image digest, CI agent version, and configuration revision. Do not approve a production feature based only on a main-branch description or an untagged command example.
Each recovery test should record five fields:
- Action performed.
- Expected result.
- Actual evidence.
- Responsible owner.
- Rollback or containment path.
Scenario score: 5/5 requires unattended recovery evidence for reboot, interruption, storage pressure, and version change.
