DATASET AND EXPERIMENT CARD
Purpose: controlled finite-plan workflow comparison; not real-world agent certification.
Authoring: original synthetic data/code, AI-assisted under Eye172 project ownership. Actual October 2026 dates. No copied benchmark answers, private user files or human participant data.
Templates: ranking, capacity, dependency, recovery, validation, freshness. All semantic rules in public prompts; all IDs finite.
Counts: 168 instances, 24 development, 24 qualification, 120 evaluation; six shared templates, four structural settings, five evaluation instances/cell. Seeds and every public input disclosed.
Lineage: separate split offsets before generation. Templates and rules intentionally shared. Disjoint random seeds are not unseen-task-template holdout. Deterministic pseudorandom draws are part of the disclosed bank, not a probability sample of all programming work.
Labeling: deterministic exact oracles; gold/no-op/incorrect controls all checked. Five templates have separate author algorithms; recovery has positive/omission/alternative controls but no separately implemented full oracle before freeze. A post-freeze grammar reference cross-checks actual outcomes under increasing-target assumptions, including public length; it is not independent human review. No human annotation or independent assessor.
Evaluation: four conditions; three repeats; two related Qwen 0.5B variants; all original assignments listed before main inference. Related models and repeats are not independent task units.
Acquisition: public model data from official Ollama registry over TLS, SHA/size verified. Two main upstream model licenses Apache-2.0; no weights included in compact results package. Tokenizer pinned to upstream commit and verified wheel; runtime binary pin retained.
Metrics: full semantic correctness, template-specific partial groups, eligible solved-claim error, wrong acceptance, actual checks, calls, tokens, time, stop reasons. Missing/infra outcomes have explicit status, unknown tokens remain unknown.
Intended uses: educational research and apparatus audits. Not intended: deploying unrestricted agents, diagnosing medical conditions, testing third-party services, claiming state-of-the-art performance or validated hundreds of architecture entries.
Bias/coverage: short ASCII tasks; synthetic conventions; small task banks; structural settings not universal difficulty; oracle/author correlation; constrained schema and output budget. Low model capability may produce floor effects.
Privacy/security: no personal data or secrets; model output never resolves host paths or executes code. Trusted native runtime still has normal user permissions; filesystem sandbox against compromised binaries not established.
Availability: full local evidence on D. Public release and asset license selection require owner review. No public preregistration, independent replication or peer review claimed.

POST-FREEZE OVERLAP AUDIT: 120 evaluation draws contain 115 unique semantic payloads after removing IDs/seeds/split metadata. Five exact repeats occur in recovery. No task removed or rerun; deduplicated sensitivity is descriptive and explicitly post hoc.

Post-freeze measurement clarification: the frozen trial controller enforces an effective 16,192 aggregate input+output token cap, below the outer 16,384 protocol ceiling; prompts advertise 16,192. This reporting correction does not alter admission or outcomes. Separate seeds did not ensure semantic holdout: 6 evaluation recovery draws/5 payloads match preflight splits, retained with a post-hoc exclusion sensitivity.
