test: classify the runner XCTests — pure decisions to a macOS host lane, simulator semantics gated os(iOS) (#1781 A7) - #1861
Conversation
Size Report
Startup median (7 runs, lower is better):
Top changed chunks:
|
|
Re-reviewed exact P2 architecture blocker: Readiness: still draft; Coverage is red from the inherited main |
27ede96 to
ba30f37
Compare
|
CI green on the actual head The lane this PR adds reports its own evidence: Undrafting. Note this is ratchet-class (see the section in the body): it must be re-run against current main immediately before merge, and it should merge last among runner-touching PRs in flight. |
|
Re-reviewed exact Split conditional-compilation/platform-reach evaluation from workflow/lane manifest validation, give each focused mirrored tests, and keep the entry script orchestration-only. Please also correct the understated growth accounting. |
ba30f37 to
5c8f81d
Compare
|
Review fixes pushed — head Both false greens are fixed, and the counts moved with them:
The unbackstopped- Still ratchet-class — re-run against current main immediately before merge. |
…ne, simulator semantics gated os(iOS) (#1781 A7) Every declared AgentDeviceRunnerUITests method now belongs to a lane, and the #if guard is the classification: AGENT_DEVICE_RUNNER_UNIT_TESTS alone means a pure runner decision (runs on the macOS host on every PR — ci.yml's existing compile job now executes the bundle it builds), '&& os(iOS)' means runner/XCTest semantics (simulator lanes only). check:xctest-selection evaluates the guards per platform, derives each lane's reach, and fails on a flagged identifier that is undeclared or uncompiled on that lane, on a declared test no lane reaches (found the two tvOS-only tests, dark since birth — widened to os(tvOS) || os(macOS)), and on testCommand reaching any lane. The host and nightly lanes assert executed == derived reach, so a missing -D flag or a guard that compiles a file out reads red, not as a smaller green. One duplicate test deleted (sparse-verdict assertions folded into its twin).
5c8f81d to
c2e91f0
Compare
Architecture blocker: closed at head
|
| module | LOC | question | test | LOC |
|---|---|---|---|---|
swift-conditional-compilation.ts |
152 | does this line compile for platform X? | swift-conditional-compilation.test.ts |
69 |
xctest-declarations.ts |
87 | what does the target declare, on which platforms? | xctest-declarations.test.ts |
86 |
check-xctest-selection.ts |
378 | which lane reaches what, is the manifest honest? | xctest-selection.test.ts |
414 |
xctest-run-summary.ts |
125 | did a lane execute what the source says it reaches? | xctest-run-summary.test.ts |
149 |
At the reviewed head these were 591 / 539 in two files; the aggregated test file is gone.
2. Entry script cohesion. main() is four calls: loadReport → reportFailures → formatSummary → exit code. The Swift-reading half (readSwiftSources, parseDeclaredTests, parseDeclaredTestsByPlatform, the DECLARATION/SWIFT_SOURCE regexes, SwiftSource/DeclaredTest) moved to xctest-declarations.ts, so the entry module reads workflow manifests and renders a verdict — one question. It is 378 lines of which 296 are code (55 comment, 27 blank): inside the ≤300 target on code, over it on rationale prose, and roughly a third of that code is the failure text the verdict prints. I did not split the verdict's messages away from the verdict to buy a smaller number.
3. Growth accounting — corrected in the body. The old line said "≈ +240 net LOC", which understated it. Real:
| main | now | Δ | |
|---|---|---|---|
| source (4 files) | 369 | 742 | +373 |
| tests (4 files) | 455 | 718 | +263 |
| harness total | 824 | 1,460 | +636 |
What it buys: the #if evaluator plus its refusal-to-guess tests, per-platform attribution, three new gate failure modes (uncompiled / dark / entry-point-reachable) each with a planted-red case, and the executed-count assertion on two lanes.
Behaviour is unchanged across both extraction passes — the check still derives 155 declared · host 132 · PR 46 · nightly 153 · 0 dark, and CI's host lane still asserts 132 of 132.
|
CI green on |
thymikee
added
the
ready-for-human
|
Re-reviewed exact One readiness cleanup remains: the PR body’s Validation and The four lines sections still report the superseded 134-test host run, while this head derives and executes 132/132. Please refresh the canonical body evidence to This remains ratchet-class: rerun against current main immediately before merge and merge last among runner-touching PRs. |
|
1 participant
Summary
The classify half of A7 (#1781): every one of the 157
AgentDeviceRunnerUITestsmethods on main (156 after this PR's one deletion) is sorted into a bucket, the sort is written into the code as#ifguards, and the pure-decision majority now runs on every PR with no simulator.Buckets (table below has every method, one reason each):
test-without-buildingstep. No new job, no simulator boot.return false). They stay on the iOS Simulator lanes: 11 on ios.yml's PR list, all 22 on the nightly.testCommand, the runner's 24-hour server entry point; skipped everywhere, now checked everywhere).testSnapshotAccessibilityUnavailableCarriesSparseVerdictduplicatedtestSnapshotAccessibilityUnavailableMarksSparseSnapshotRunnerFatal(same production call); its three verdict assertions are folded into the surviving test, which also checksreason— a field the deleted one never asserted. AndtestBlockingSystemAlertSnapshotIsNilOnTvOS, deleted rather than widened:blockingSystemAlertSnapshotis#if os(macOS) return nil, so on the only lane that could run it the assertion pins a compile-time literal. Nothing else deletes: the CLI/vitest suites observe the daemon through the runner's wire surface, not these in-process decision seams, so no XCTest here is "already observed by the CLI smoke" — and ios runner: navigation fallback helpers accept CGRect.infinite (tap point ≈ −9e307) #1812 is the standing counterexample for calling any of the dark set disposable.The classification is enforced, not documented. A test's
#ifguard is its bucket (convention comment inRunnerTests.swift):AGENT_DEVICE_RUNNER_UNIT_TESTSalone ⇒ pure, host + simulator;… && os(iOS)⇒ simulator only.check:xctest-selectionnow evaluates the guards per platform (small#ifevaluator that throws on vocabulary it doesn't know rather than guessing) and derives what each of the three lanes — host (ci.yml, whole macOS bundle), PR list (ios.yml-only-testing:), nightly (whole iOS bundle) — reaches, failing on:#if os(tvOS)tests, dark since birth; renamed…WithoutSpringBoardHostand widened toos(tvOS) || os(macOS)so the host lane runs their no-SpringBoard contract),testCommandreachable by any lane (new — the nightly's skip was checked before; ci.yml's skip now is too).The host lane and the nightly both assert executed == derived reach (
scripts/xctest-run-summary.ts), closing both the known "-D flag missing ⇒ silent 0-test green" and its quieter sibling, a guard compiling a whole file out.Swift changes: 12 sim-semantic tests gained
os(iOS)guards — 10 in+CommandExecution.swift, 2 in+Snapshot.swift(they launch apps / probe SpringBoard, or — the case review caught — pin a guard whose macOS arm is a barereturn false; 7 of the 12 demonstrably fail on the macOS host, and the other 5 pass there only because the branch they exist to assert is compiled out) — one tvOS test renamed + widened toos(tvOS) || os(macOS), 2 tests deleted. No test logic changed otherwise.The table
All 157 methods · test · file · bucket · reason (132 HOST · 22 SIM · 1 ENTRY · 2 DELETE)
testTapPointPolicyMatchesGoldenParityTableRunnerTapPointPolicy.swiftcontracts/fixtures/tap-point-policy.json(TS twintap-point-policy-parity.test.ts); pure geometry, no XCUIElement — the Swift half of the parity gate must staytestActionNamesAreCappedPerElementAndReportedRunnerTests+AXSnapshotFallback.swifttestCustomActionCoverageParsesOnlyCompletePairsRunnerTests+AXSnapshotFallback.swifttestCustomActionsRequestPinsPrivateAXBackendRunnerTests+AXSnapshotFallback.swiftCommand, checkspresentationOptionsbackend pin; puretestDeepExtensionCountsMissedFrontiersRunnerTests+AXSnapshotFallback.swiftRunnerAXSnapshotBridge.extendwith fake frontiers/AX client (FrontierSnapshot…ForTesting); no live treetestHungCustomActionReadIsContainedAndRecoversRunnerTests+AXSnapshotFallback.swifttestPartialCustomActionPassIsDisclosedAndCompleteOneIsNotRunnerTests+AXSnapshotFallback.swiftlegacyQualityMessagewording for partial coverage; puretestPrivateAXAcceptedDepthMemoryMatchesBundleProcessAndExpiresRunnerTests+AXSnapshotFallback.swifttestPrivateAXAcceptedDepthMemoryRequiresProcessIdentifierToRecordRunnerTests+AXSnapshotFallback.swifttestPrivateAXAttemptDepthsAppliesRememberedDepthRunnerTests+AXSnapshotFallback.swiftprivateAXAttemptDepths); puretestPrivateAXDepthLimitedRequiresEveryFrontierResolvedRunnerTests+AXSnapshotFallback.swiftprivateAXDepthLimitedverdict matrix; puretestPrivateAXInteractiveFiltersLoginLikeHiddenDrawerRunnerTests+AXSnapshotFallback.swiftprivateAXPresentationinteractive filter over a dictionary tree; puretestPrivateAXNodesCarryAnnotatedCustomActionsRunnerTests+AXSnapshotFallback.swiftprivateAXPresentationover a dictionary tree; pure (usesXCUIElement.ElementTyperaw values only)testPrivateAXScopeSelectsSubtreeNotMatchingLabelsRunnerTests+AXSnapshotFallback.swiftprivateAXPresentationscope selection over a dictionary tree; puretestRequestPinnedBackendReportsItsOwnReasonRunnerTests+AXSnapshotFallback.swiftxcTestChannelStateFirstFailurereason codes; puretestViewportReadSkippedWhileXCTestChannelPenalizedRunnerTests+AXSnapshotFallback.swiftshouldReadPrivateAXViewportViaXCTestover penalty + abandoned-capture state; no apptestAlertAcceptTreatsOpenAsAffirmativeRunnerTests+Alert.swiftisAcceptButtonlabel table; pure string ruletestRemoteHostProbeRunsOnlyWhenSpringboardModalHasNoActionsRunnerTests+BlockingSystemModalResolution.swiftRemoteHostedSystemModalPolicydecision on an action count; puretestRemoteHostStateGateFailsClosedToForegroundRunnerTests+BlockingSystemModalResolution.swiftRemoteHostedSystemModalPolicystate gate; pure enum ruletestResolveBlockingSystemModalIsAbsentWithoutSpringBoardHostRunnerTests+BlockingSystemModalResolution.swift…OnTvOSunder#if os(tvOS)— reachable by no lane; the contract (no SpringBoard host →.absent) is macOS's too, so widened toos(tvOS) || os(macOS)and now runs on the host lanetestCanonicalPlannedGestureResponseOmitsDragFrameAndPreservesDiagnosticsRunnerTests+CommandExecution.swiftcanonicalPlannedGestureResponseprojection; puretestExecuteDispatchedReturnsBusyBeforeMainThreadFastPathRunnerTests+CommandExecution.swiftexecuteDispatchedshort-circuits to RUNNER_BUSY on abandoned main-thread work before touching any app; no apptestExecuteDispatchedReturnsWedgedBeforeMainThreadFastPathRunnerTests+CommandExecution.swifttestGestureResponseIncludesMaestroNonHittableFallbackUsageRunnerTests+CommandExecution.swiftgestureResponsepayload shape; puretestGestureResponseIncludesSynthesizedTapFallbackDiagnosticsRunnerTests+CommandExecution.swiftgestureResponsepayload shape; puretestInjectedTapRecordedFailureGateIsTapOnlyAndCountGatedRunnerTests+CommandExecution.swifttestPostSnapshotDelayMarkDoesNotQueueBehindAbandonedTreeCaptureRunnerTests+CommandExecution.swiftsetNeedsPostSnapshotInteractionDelaymust not enqueue on main behind abandoned work; real GCD, no apptestRunMainThreadWorkExecutesOffMainCallerOnMainThreadRunnerTests+CommandExecution.swiftrunMainThreadWorkhops an off-main caller to main; real GCD, no apptestRunMainThreadWorkTimeoutMarksAbandonedUntilDrainedRunnerTests+CommandExecution.swiftrunMainThreadWorktimeout → abandoned/drained bookkeeping; real GCD + semaphores, no apptestXCTestRecordedFailureResponseDoesNotWrapReadOnlyOrRunnerFatalResponsesRunnerTests+CommandExecution.swiftxctestRecordedFailureResponsefor reads / runnerFatal; puretestXCTestRecordedFailureResponseFailsMutatingSuccessesRunnerTests+CommandExecution.swiftxctestRecordedFailureResponsefor mutations; puretestCommandJournalKeepsErrorMetadataWhenResponseJsonIsDroppedRunnerTests+CommandJournal.swifttestCommandJournalRetainsCompletedSequenceResultsRunnerTests+CommandJournal.swifttestCommandJournalRetainsFailedSequenceResultsRunnerTests+CommandJournal.swifttestCommandJournalRetentionPolicyRunnerTests+CommandJournal.swifttestJournalStoredResponseStaysUnstampedRunnerTests+CommandJournal.swiftRunnerCommandJournalstores unstamped JSON; puretestStampingCurrentUptimeCreatesPayloadWhenNilRunnerTests+CommandJournal.swifttestStampingCurrentUptimePreservesPayloadRunnerTests+CommandJournal.swiftResponse.stampingCurrentUptimeMs; puretestStampingCurrentUptimeSkipsErrorResponsesRunnerTests+CommandJournal.swifttestUptimeBypassesCommandJournalRunnerTests+CommandJournal.swiftexecute(uptime)must not be journaled; runs the command dispatcher without a target apptestFlatSnapshotFilterDecisionCarriesSubtreeScopeStateRunnerTests+FlatSnapshotFiltering.swifttestFlatSnapshotFilterDecisionMatrixCoversOptionsRunnerTests+FlatSnapshotFiltering.swiftflatSnapshotFilterDecisionoption matrix; puretestFlatSnapshotProjectionMatchesElementReverseScrollCaptureRunnerTests+FlatSnapshotFiltering.swifttestPrivateAXInteractiveCandidatesPreserveBackendInputsRunnerTests+FlatSnapshotFiltering.swiftprivateAXInteractiveCandidatetype rule; puretestDesktopScrollWheelDeltaEventsHonorDurationAndPreservePixelsRunnerTests+Interaction.swifttestDesktopScrollWheelDeltaEventsKeepInstantScrollSingleEventRunnerTests+Interaction.swifttestDesktopScrollWheelDeltasMapDirectionsRunnerTests+Interaction.swifttestNativeSynthesizedPointRotatesByInterfaceOrientationRunnerTests+Interaction.swifttestNativeSynthesizedVectorRotatesByInterfaceOrientationRunnerTests+Interaction.swifttestOrientedSynthesizedScreenshotReferenceFrameUsesLandscapeLogicalDimensionsRunnerTests+Interaction.swifttestPlannedMultiTouchGestureAcceptsMatchingInBoundsTrajectoriesRunnerTests+Interaction.swiftRunnerGesturePlandecode + validation; puretestPlannedMultiTouchGestureRejectsMismatchedOffsetsRunnerTests+Interaction.swifttestSinglePointerEndpointHoldUsesFastSwipeExecutionRunnerTests+Interaction.swifttestSinglePointerFlingUsesFastSwipeExecutionRunnerTests+Interaction.swiftplannedGestureExecutionprofile mapping; puretestSinglePointerGestureRejectsMissingExecutionProfileRunnerTests+Interaction.swifttestSinglePointerTimedPanUsesSampledExecutionRunnerTests+Interaction.swifttestSynthesizedScreenshotReferenceFrameRejectsInvalidSizeRunnerTests+Interaction.swifttestSynthesizedScreenshotReferenceFrameUsesScreenshotSizeRunnerTests+Interaction.swifttestRunnerScreenshotStabilitySettledFalseOnFailedCaptureRunnerTests+Keyboard.swifttestRunnerScreenshotStabilitySettledFalseOnMidWindowMismatchRunnerTests+Keyboard.swifttestRunnerScreenshotStabilitySettledNeedsEnoughSamplesRunnerTests+Keyboard.swiftrunnerScreenshotStabilitySettledwindow rule; pure overDatasamplestestRunnerScreenshotStabilitySettledOnlyLooksAtTheTrailingWindowRunnerTests+Keyboard.swifttestRunnerScreenshotStabilitySettledRejectsDegenerateRequirementRunnerTests+Keyboard.swifttestRunnerScreenshotStabilitySettledTrueWhenWindowMatchesRunnerTests+Keyboard.swifttestCachedTargetInvalidationClearsProcessBoundStateRunnerTests+LifecycleCacheTests.swiftinvalidateCachedTargetclears process-bound fields; assigns the unlaunchedXCUIApplicationproxy as a token onlytestCachedTargetRefreshRequiresChangedPositiveProcessIdentityRunnerTests+LifecycleCacheTests.swiftshouldRefreshCachedTargetPID rule; puretestSnapshotPenaltyCanBeClearedAcrossTargetProcessReplacementRunnerTests+LifecycleCacheTests.swifttestSnapshotPenaltyWarmupExemptionIsConsumedOnceRunnerTests+LifecycleCacheTests.swifttestTargetResetInvalidatesProcessBoundStateWithoutRestartingRunnerRunnerTests+LifecycleCacheTests.swiftresetTargetAfterExternalRelaunchbookkeeping on runner state; unlaunched proxy as token, no apptestTextEntryTapWitnessIsBoundToTargetIdentityRunnerTests+LifecycleCacheTests.swiftTextEntryTapWitness.matchesidentity rule; puretestNavigationBackControlRankPrefersBackThenCloseThenCancelRunnerTests+Navigation.swifttestNavigationBackPredicateUsesTheSharedKeywordTableRunnerTests+Navigation.swiftNSPredicatefrom the keyword table; puretestNavigationFallbackRequiresObservedVisualChangeRunnerTests+Navigation.swiftDatacomparison; puretestTopLeadingNavigationFallbackPointRejectsInvalidFrameRunnerTests+Navigation.swift.infinite/.zeroguard; pure geometry, fixed on main and green heretestTopLeadingNavigationFallbackPointTargetsHeaderControlBandRunnerTests+Navigation.swifttestTopNavigationControlFrameAcceptsOnlyHeaderBandRunnerTests+Navigation.swift.infinite; puretestPrivateAXGeometrylessSemanticsAreNeverActionableOrScrollContextsRunnerTests+PrivateAXPresentation.swifttestPrivateAXPresentationKeepsOffscreenSubtreeExcludedWhenChildFramesAreClampedRunnerTests+PrivateAXPresentation.swifttestPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHintRunnerTests+PrivateAXPresentation.swiftprivateAXPresentationviewport projection over a dictionary tree; puretestRecordStopIsIdempotentAfterNativeRecorderAlreadyStoppedRunnerTests+RecordingTests.swiftexecute(recordStop)with no active recording; dispatcher path, no app or recordertestRunnerScrollGesturePlanMatchesParityTableRunnerTests+ScrollGesture.swiftcontracts/fixtures/scroll-gesture.json(TS twinscroll-gesture.test.ts); puretestRunnerScrollGesturePlanRejectsInvalidAmountAndPixelsRunnerTests+ScrollGesture.swifttestRunnerScrollGesturePlanRejectsUnknownDirectionRunnerTests+ScrollGesture.swifttestRunnerScrollGesturePlanUsesParityTableConstantsRunnerTests+ScrollGesture.swifttestDirectSelectorAcceptsOneRawHittableMatchRunnerTests+SelectorMatchPolicyTests.swifttestDirectSelectorRejectsTwoRawMatchesBeforeHittabilityPreferenceRunnerTests+SelectorMatchPolicyTests.swiftclassifyDirectSelectorCandidatesmutation row; puretestMaestroSelectorKeepsExpectedPointAndNonHittableFallbackSemanticsRunnerTests+SelectorMatchPolicyTests.swifttestReadSelectorDoesNotAdoptTheNonHittableCoordinateFallbackRunnerTests+SelectorMatchPolicyTests.swifttestReadSelectorPrefersTheHittableMatchOverANonHittableDuplicateRunnerTests+SelectorMatchPolicyTests.swifttestReadSelectorStillRejectsTwoHittableMatchesRunnerTests+SelectorMatchPolicyTests.swifttestAssembleSequencePreservesOrderOnSuccessRunnerTests+SequenceExecution.swiftassembleSequenceresult ordering with stubbed step results; puretestAssembleSequenceStopsAtFirstFailureRunnerTests+SequenceExecution.swiftassembleSequencefail-fast; puretestSequenceAcceptsDoubleTapKindRunnerTests+SequenceExecution.swiftexecuteSequencevalidation before any executor call; unlaunched proxy asactiveApptestSequenceDecodesStepsFromWireRunnerTests+SequenceExecution.swiftCommanddecode of sequence steps; puretestSequenceHasSynthesizedCoordinateStepRunnerTests+SequenceExecution.swifttestSequenceRejectsEmptyRunnerTests+SequenceExecution.swifttestSequenceRejectsTooManyStepsRunnerTests+SequenceExecution.swifttestSequenceRejectsUnknownKindRunnerTests+SequenceExecution.swifttestSequenceWorstCaseResponseStaysUnderJournalCapRunnerTests+SequenceExecution.swifttestDispatchRecoverySkipsBookkeepingWhileXCTestChannelOccupiedRunnerTests+Snapshot.swiftexecuteDispatchedWithRecoverywith a stub perform closure over abandoned-work state; real GCD, no apptestRawSnapshotTooLargeFailureIsStructuredRunnerTests+Snapshot.swifttestRecoveredSnapshotMessagePreservesHintRunnerTests+Snapshot.swifttestSnapshotAccessibilityUnavailableMarksSparseSnapshotRunnerFatalRunnerTests+Snapshot.swiftsnapshotAccessibilityUnavailablepayload shape + target invalidation + (merged in) the sparse verdict; unlaunched proxy as tokentestSystemModalProbeSliceSharesAndClampsToPlanDeadlineRunnerTests+Snapshot.swiftsystemModalProbeSliceclamp arithmetic; puretestAbandonedTreeCaptureSkipsOnlyXCTestBackedSnapshotTiersRunnerTests+SnapshotCapturePlan.swiftshouldSkipSnapshotBackendForAbandonedTreeCapture; puretestCollapsedLeafIndexesFlagsMergedContainersOnlyRunnerTests+SnapshotCapturePlan.swifttestDecodedPreferredBackendReachesOptionsAndApplicablePlanRunnerTests+SnapshotCapturePlan.swifttestEffectiveSnapshotCapturePlanDefersXCTestBackedTiersOnlyWhenPenalizedRegularPlanRunnerTests+SnapshotCapturePlan.swifttestEffectiveSnapshotCapturePlanUsesBoundedXCTestProbeWhenNoIndependentBackendRunsRunnerTests+SnapshotCapturePlan.swifttestLegacyQualityMessageStatesFallbackMeaningRunnerTests+SnapshotCapturePlan.swiftlegacyQualityMessagewording; puretestPreferredPrivateAXBackendPlansAsPenalizedRunnerTests+SnapshotCapturePlan.swifttestSnapshotXCTestChannelPenaltyMatchesBundleAndExpiresRunnerTests+SnapshotCapturePlan.swifttestSparsePayloadReasonMatrixRunnerTests+SnapshotCapturePlan.swifttestSuppressedAxSnapshotIssueClassifierRunnerTests+SnapshotCapturePlan.swiftisSuppressedAxSnapshotIssueDescriptionclassifier; pure (record(_:) itself deliberately not invoked)testTerminalFailsClosedOnInteractiveAxFailureRegardlessOfSparseBestRunnerTests+SnapshotCapturePlan.swifttestXCTestChannelStateFirstFailureStampsDeferredCodeOnlyForDeferralRunnerTests+SnapshotCapturePlan.swiftxcTestChannelStateFirstFailurecodes; puretestSnapshotPresentationOwnsBackendNeutralEligibilityRunnerTests+SnapshotPresentationTests.swiftSnapshotPresentationbackend-neutral eligibility (#1850, landed mid-classification); puretestSnapshotPresentationPreservesCurrentWireShapeRunnerTests+SnapshotPresentationTests.swiftSnapshotPresentation→ JSON wire shape; puretestSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBoundsRunnerTests+SnapshotTraversalIdentityTests.swiftsnapshotTraversalIdentity— platform-dependent by design (#if os(iOS)NotEqual, else Equal), so both lanes assert their own branch; puretestCoordinateTapTextInputProbeSkipsPenalizedXCTestChannelRunnerTests+SynthesizedGesturePolicy.swiftshouldProbeCoordinateTapTextInput; puretestSynthesizedDragCoordinateFallbackAllowsUnknownButNotUnavailableAccessibilityRunnerTests+SynthesizedGesturePolicy.swiftSynthesizedFallbackPolicytable; puretestSynthesizedFallbackPolicyRequiresPrivateSynthesisForScrollWhenAxUnavailableOrUnknownRunnerTests+SynthesizedGesturePolicy.swiftSynthesizedFallbackPolicytable; puretestSynthesizedGesturePoliciesMatchCommandContractsRunnerTests+SynthesizedGesturePolicy.swiftsynthesizedGesturePolicy(kind)contract table; puretestSynthesizedKeyboardPolicyKeepsUnknownDragProbeButNotUnknownScrollProbeRunnerTests+SynthesizedGesturePolicy.swiftSynthesizedKeyboardPolicytable; puretestResolvedCoordinateTextEntryFallsBackWhenSynthesizedFocusIsUnavailableRunnerTests+TextEntryPolicyTests.swiftshouldFallbackFromSynthesizedTextEntryFocus; puretestResolvedCoordinateTextEntryRouteRequiresReplacementCoordinatesAndPenalizedXCTestRunnerTests+TextEntryPolicyTests.swiftshouldUseResolvedCoordinateTextEntryRoutematrix; puretestSynthesizedFirstResponderTypeRequiresHiddenKeyboardTapWitnessRunnerTests+TextEntryPolicyTests.swiftshouldUseSynthesizedFirstResponderTypematrix; puretestSynthesizedReplacementPacesCharactersAfterSelectingOnceRunnerTests+TextEntryPolicyTests.swiftsynthesizedReplacementStepspacing; puretestSynthesizedReplacementRequiresPenalizedXCTestAndCoordinatesRunnerTests+TextEntryPolicyTests.swiftshouldUseSynthesizedFirstResponderReplacementmatrix; puretestSynthesizedTextCommitProgressWalksExpectedPrefixOnlyRunnerTests+TextEntryPolicyTests.swifttestCoordinateTextInputCandidateMustBeEnabledAndContainTheTouchPointRunnerTests+TextInputCandidatePolicy.swiftisCoordinateTextInputCandidategeometry rule; puretestDuplicateCommandIdCoalescesOntoInFlightExecutionRunnerTests+Transport.swiftattachToInFlightCommandIfNeeded/deliverCommandResult) on runner state; no transport, no apptestAlertResolutionCannotBypassRequestedDeadlineRunnerTests+CommandExecution.swiftcurrentApp = springboard+ alert resolution deadline through the real dispatch path (failed on macOS: no SpringBoard) — newly gatedtestBareDelayedTypeFailsWhenTappedInputDisappearsMidCommandRunnerTests+CommandExecution.swiftos(iOS)testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHiddenRunnerTests+CommandExecution.swiftos(iOS)testExecuteDispatchedReturnsBusyBeforeBlockingSystemModalProbeDrainsRunnerTests+CommandExecution.swiftapp.launch()+ a real bounded SpringBoard probe timeout (13 s; failed on macOS) — newly gatedtestMissingBundleCommandInvalidatesCompleteCachedTargetStateRunnerTests+CommandExecution.swiftapp.launch()+prepareActiveCommandContextagainst a live foreground app (newly gatedos(iOS); passed on macOS by launching the macOS host app, but the contract is about a real target)testPrepareActiveCommandContextRoutesBlockingSystemModalToSpringboardRunnerTests+CommandExecution.swiftspringboard(XCUIApplication(bundleIdentifier: com.apple.springboard)), iOS-only routing (failed on macOS) — newly gatedtestSelectorTapFallsBackToXCTestCoordinateWhenPrivateSynthesisFailsRunnerTests+CommandExecution.swiftos(iOS)testSinglePointerFlingFallsBackToXCTestCoordinateDragWhenPrivateSynthesisFailsRunnerTests+CommandExecution.swiftos(iOS)testSkipAppActivationPreflightIncludesAlertCommandsRunnerTests+CommandExecution.swift#if os(iOS)inshouldRouteToSpringboardBlockingSystemModal; failed on macOS) — newly gatedtestSkipAppActivationPreflightIncludesForegroundCachedCoordinateOnlyTapsRunnerTests+CommandExecution.swiftapp.launch(); foreground-state-dependent preflight (failed on macOS: no foreground state) — newly gatedtestSkipAppActivationPreflightKeepsDragScrollAndSequenceOnForegroundGuardRunnerTests+CommandExecution.swiftapp.launch(); foreground preflight rows — newly gatedtestSkipAppActivationPreflightRejectsMissingChangedAndBackgroundTargetsRunnerTests+CommandExecution.swiftapp.launch()/terminate()to produce changed and background targets — newly gatedtestSkipAppActivationPreflightRejectsSelectorAndMixedSequenceGesturesRunnerTests+CommandExecution.swiftapp.launch(); foreground preflight rows — newly gatedtestSkipAppActivationPreflightRequiresCachedForegroundTargetRunnerTests+CommandExecution.swiftshouldSkipAppActivationPreflightis#if os(iOS) …guards… #else return false #endif, so on macOS this asserted a compile-time literal — moved to SIM in review; the guard it pins (currentApp.state == .runningForeground) only exists on iOStestTypeWithoutResolvedInputReturnsTypedFailureBeforeDispatchingTextRunnerTests+CommandExecution.swiftXCUIApplicationtype path with no resolved input (12 s); alreadyos(iOS)testPenalizedCoordinateTapOnNonTextControlDoesNotAuthorizeBareTypeRunnerTests+CoordinateTextEntryTests.swift--agent-device-text-entry-regressionfixture (17 s); alreadyos(iOS)testActivateTargetSkipsForegroundAndActivatesNonForegroundApplicationRunnerTests+LifecycleCacheTests.swiftXCUIApplication.state/activate; alreadyos(iOS)testQuerySelectorPrefersHittableMatchOverNonHittableDuplicateRunnerTests+SelectorMatchPolicyTests.swift--agent-device-selector-read-regressionfixture (7 s); alreadyos(iOS)testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrainRunnerTests+Snapshot.swiftrunMainThreadWorktimeout of the bounded SpringBoard probe throughsnapshotFast(5 s); the probe body returns nil on macOS (failed there) — newly gatedtestBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrainForSnapshotRawRunnerTests+Snapshot.swiftsnapshotRaw— deliberately parametrised over both entry points, kept — newly gatedtestSynthesizedTextEntryFallsBackOnlyWhenPrivateSynthesisIsUnavailableRunnerTests+TextEntryPolicyTests.swiftPrivateXCTestTextEntrySynthesizeris an iOS-only type; alreadyos(iOS), stays SIM by compile constrainttestTypeTextReliablyPacesSynthesizedReplacementThroughProductionCallerRunnerTests+TextEntryPolicyTests.swifttypeTextReliablywith a fake synthesizer conforming to the iOS-onlyTextEntrySynthesizing; alreadyos(iOS)testCommandRunnerTests.swift-skip-testing:on both whole-bundle lanes, absent from the PR list; the check fails if any lane reaches ittestSnapshotAccessibilityUnavailableCarriesSparseVerdictRunnerTests+SnapshotCapturePlan.swifttestSnapshotAccessibilityUnavailableMarksSparseSnapshotRunnerFatal(same production call, only thesnapshotQualityassertions differed); all three folded into the survivor, which also assertsreason— a field the deleted one never checkedtestBlockingSystemAlertSnapshotIsNilOnTvOSRunnerTests+BlockingSystemModalResolution.swiftblockingSystemAlertSnapshotis#if os(macOS) return nil, so on the only lane that could run it the assertion pins a compile-time literal. Its twintestResolveBlockingSystemModalIsAbsentWithoutSpringBoardHostcovers the same #1351 contract at the layer that decides it at runtime (hasSpringBoardSystemModalHost) and is widenedLane arithmetic (so the numbers reconcile against the xcresult, not the exit code)
xcodebuildexits 0 on a selection that matches nothing, so every count below is read off theresult bundle's executed total, never off an exit code.
os(tvOS)-only)os(iOS), −1 deleted duplicate, +1 widened)os(iOS), −1 deleted duplicate, −1 deleted vacuous)testCommand)testCommand)-only-testing:)The nightly's 155-executed night predates two tests that have since landed and the ones this PR
moves; against current main the same command would execute 154, and against this head it
executes 153 — the delta is exactly the deleted duplicate. macOS and tvOS both land on 135
because the 11 newly
os(iOS)-gated tests used to compile (and, on macOS, fail) on everyplatform; they now compile only where they can pass. No lane runs a tvOS destination at all —
which is why the two
…OnTvOStests were dark since birth, and why the fix routes their contractthrough the macOS host lane instead of inventing a tvOS lane. Their tvOS branch is still never
executed; that is stated as residual risk rather than papered over.
Validation
Swift Runner Host XCTests):Executed 134 tests, with 0 failures (0 unexpected) in 3.152 seconds, reporterExecuted 134 test(s); the source reaches 134 on this lane.— 82 s for the whole job (06:03:28 → 06:04:50Z) including checkout, toolchain, and the cached runner build the job already did before this PR.Executed 134 tests, with 0 failures in 8.1svia the same command; reporter134 of 134.Executed 153 tests, with 0 failures in 119s; reporter passes with153 of 153for the nightly lane. (Nightly ran 155 before: −2 tvOS-gated tests that never executed there anyway, −1 deleted duplicate, +1 new test from feat(ios): unify snapshot eligibility #1850 which landed mid-classification and is bucketed HOST in the table.)-skip-testing:→ fails as unknown identifier and entry point reachable by lane(s): host (the 24-hour hang the check exists for);#if os(tvOS)→ fails as reachable by no lane;XCTAssertNil→NotNilintestTopLeadingNavigationFallbackPointRejectsInvalidFrame(the ios runner: navigation fallback helpers accept CGRect.infinite (tap point ≈ −9e307) #1812 test), rebuilt: host lane runsExecuted 133 tests, with 1 failure, exit 65 — the moved tests genuinely assert on macOS.XCTEST_LANE=nightlyfails withexecuted 134 … reaches 153.os(iOS)(SpringBoard probes and app launches; several of the records are the same test's cascading issues). The remaining 4 of the 11 gated tests passed on macOS only because their iOS-only branch compiles out there, so passing was not evidence of coverage — that is why the bucket is drawn by what the test asserts, not by what happens to go green. After gating: 134 executed / 0 failures, repeated across runs.check:xctest-selectiongreen on this head; its vitest suite grew planted-guard/uncompiled/dark/entry-point cases (planted-red for each new failure mode, asserted red in-suite); gate-manifest, scripts/gate + check-affected suites, typecheck, lint, format green;pnpm check:affected --rungreen before push.CODE_SIGN_IDENTITYon the build (now indocs/agents/testing.md); GitHub's macOS runners run the unsigned build as-is — this PR's own CI is the proof.What independent review changed
An adversarial review re-derived the whole count table with its own parser (exact match) and found two false greens — the precise class this PR's bucket rule exists to prevent — plus three smaller items. All fixed here:
testSkipAppActivationPreflightRequiresCachedForegroundTargetwas HOST but vacuous there.shouldSkipAppActivationPreflightis#if os(iOS) …guards… #else return false #endif, so on macOS theXCTAssertFalsepinned a compile-time literal and no edit to the iOS body could turn it red. Its five siblings in the same matrix were already gated. Moved into theos(iOS)region → HOST 133 → 132, SIM 21 → 22.testBlockingSystemAlertSnapshotIsNilOnTvOSwas widened into the same trap.blockingSystemAlertSnapshotis#if os(macOS) return nil, so its macOS run asserted the compiler. Deleted rather than widened; its twintestResolveBlockingSystemModalIsAbsentWithoutSpringBoardHostcovers the same tvOS: every snapshot/alert resolution fails because the runner probes com.apple.springboard (no SpringBoard on tvOS) #1351 contract where the decision is actually taken at runtime (hasSpringBoardSystemModalHost) and keeps the widening. The honest statement is therefore that one of the two contracts now runs on macOS — corrected in the Swift comment, which had claimed both.check-xctest-selection.tshad grown 269 → 591 LOC, crossing both AGENTS.md tripwires, and a re-review added that the split had to separate platform-reach evaluation from workflow/lane manifest validation with mirrored tests and an orchestration-only entry point. Done in two passes, ending at four modules each answering one question, each with its own test file:swift-conditional-compilation.tsswift-conditional-compilation.test.tsxctest-declarations.tsxctest-declarations.test.tscheck-xctest-selection.tsxctest-selection.test.tsxctest-run-summary.tsxctest-run-summary.test.tsmain()is now four calls (load → failures → summary → exit code). The entry module is 378 lines of which 296 are code (55 comment, 27 blank) — inside the ≤300 target on code, over it on the rationale prose this repo asks for; the remaining content is one question (lane manifest → verdict), with roughly a third of it the failure messages that verdict prints.Follow-up sizing was 2× off (see above): 11, not ~21.
docs/agents/testing.mdsigning recipe did not work on a fresh Mac. Rewritten as machine-dependent after testing both spellings on this one: the genericCODE_SIGN_IDENTITY="Apple Development"fails here withNo signing certificate "Mac Development" found(with and withoutCODE_SIGN_STYLE=Manual), while the certificate SHA-1 works; the reviewer's Mac is the inverse. The doc now says try both and notes the XCUITest automation permission a local host run needs.Also fixed while in there:
TEST_FLAGmatched only the first flag per line (.exec, notmatchAll) — latent today but silently permissive in the skip direction, where an unseen-skip-testing:is a lane that stops skipping the 24-hour entry point. Now global, with a regression test.Review also confirmed, and worth keeping in the record: no executed coverage is lost (the 12 newly gated tests were compiled-but-never-executed on macOS before this PR, since no lane ran the bundle there); the sweep found exactly three production symbols with degenerate macOS arms and the third (
testSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBounds) asserts both branches via its own in-body#ifand is correctly bucketed; and the evaluator resisted attempts to break it on precedence,!, nesting,#elseif,os(visionOS), and trailing comments while throwing onswift(>=),compiler(>=), and/* */.One more thing the table makes visible: the golden-table parity tests (
testTapPointPolicyMatchesGoldenParityTable,testRunnerScrollGesturePlanMatchesParityTable) and both #1812 navigation tests are on no-only-testing:list — their Swift half ran nightly-only until now and gates every PR from here.The four lines
Catches: regressions in the 134 pure runner-decision tests on every PR in ~20 s of an existing job, independent of ios.yml (15 min, ~19% cancel rate, and its list names only 45 of them); plus three new gate classes — dark tests, wrong-platform selections, silent partial runs.
Evidence: ios runner: navigation fallback helpers accept CGRect.infinite (tap point ≈ −9e307) #1812 came from exactly this test class (a geometry guard) on its first-ever execution; the two tvOS tests were dark from birth and the new reachability rule found them mechanically before any human read the table.
Cost: ~20 s wall on a PR job that already built the bundle (measured 3.4 s test execution in CI); no new macOS job, no simulator. Harness growth, stated properly rather than as a net figure — an earlier draft of this line said "≈ +240 net LOC", which understated it:
check-xctest-selection.tsswift-conditional-compilation.tsxctest-declarations.tsxctest-run-summary.tsxctest-selection.test.tsswift-conditional-compilation.test.tsxctest-declarations.test.tsxctest-run-summary.test.tsWhat the +636 buys: the
#ifevaluator and its refusal-to-guess tests, per-platform attribution, three new gate failure modes (uncompiled / dark / entry-point-reachable) each with a planted-red case, and the executed-count assertion on two lanes. It is a real cost and worth naming as one.Kill-criterion: a host-lane test that proves flaky where its simulator run is not gets an
os(iOS)guard and returns to the simulator lanes (review already exercised this direction — two rows moved out of HOST); more than a handful means the host lane is the wrong tool and goes back to compile-only. The nightly's own kill criterion is updated in its header with the measured figure below.Ratchet-class — re-run against main immediately before merge
This PR tightens
check:xctest-selection: it adds three failure modes (uncompiled selection,dark test, entry point reachable) and both whole-bundle lanes now assert an exact executed
count. Any PR that lands a runner XCTest between this PR's last CI run and its merge can turn
main red the moment this merges — a new test whose guard names a platform no lane runs, or one
added to ios.yml's list under the wrong guard, fails the gate rather than the author's PR.
So: merge this last among the runner-touching PRs in flight, and re-run its CI against
current main immediately before merging. It has already been rebased twice mid-review for
exactly this reason (absorbing #1850's new test, then #1860).
Scope
17 files: 6 Swift (guards, 1 rename+widening, 2 deletions, convention comment), 2 workflows (ci.yml job, xctest-nightly.yml env + honest comments), 4 scripts + 4 script tests (the evaluator and the declaration scan became their own modules under review, each with a mirrored test file), AGENTS.md gate bullet, docs/agents/testing.md row. ios.yml is deliberately untouched — its
-only-testing:list is the follow-up's business, so this PR does not tighten the PR lane.Residual risk
testHungCustomActionReadIsContainedAndRecoversis the one moved test with real timing in it (a 1 s read deadline plus a drain poll). It passed on every local host run, but it is the first candidate to re-gateos(iOS)if the host lane ever flakes — named in the kill criterion for that reason.#ifevaluator is a small parser, not the Swift compiler. It refuses vocabulary it does not know (test asserts this), so a new guard shape fails the gate loudly rather than being silently mis-bucketed — but a supported guard used in a way I did not anticipate could still mis-attribute a test. The executed-count assertion on both whole-bundle lanes is the backstop: source-derived reach and real executed count have to agree.#ifinside a/* */block comment or a"""string literal reads as a real directive. The evaluator is line-based, so commented-out or quoted directive text would shift its frame stack. On the host and nightly lanes the executed-count assertion backstops this (source-derived reach and real executed count must agree); the PR lane has no count assertion, so there it is unbackstopped until the follow-up adds one. No such construct exists in the tree today.Follow-up (separate PR, merges last per the ratchet rule): drop ios.yml's
-only-testing:list in favor of the whole iOS bundle + executed-count assertion, then retire xctest-nightly.yml.Sizing that follow-up honestly: what only the nightly reaches is
nightly ∖ (host ∪ pr)= 11 tests / ~42 s of simulator time (measured off the first night's xcresult; an earlier draft of this body said "~21 / ~2.5 min", which double-counted the tests ios.yml's list already runs — the kill-criterion paragraph was the correct half and the two contradicted each other). The trade is not free: those 11 are exactly the app-launching, SpringBoard-probing class — the flakiest tests in the suite, with the longest ones at 13 s, 7 s, 6 s, 6 s — and moving them onto ios.yml puts them on a blocking 15-minute PR gate that is already cancelled ~19% of the time. Measure their flake rate on the nightly before moving them.