SemaMig-Bench
Does an AI migration preserve the behavior of the system it replaces?
SemaMig-Bench evaluates agentic COBOL-to-Java migration by running the executable source and generated Java candidate on the same held-out workloads. It scores contract-relevant behavior, not source similarity or compilation alone.
Release status
The method is public. The first benchmark run comes after freeze.
Protocol
Published
Task design, source-oracle evaluation, scoring, and release criteria.
Task corpus
In construction
50 practitioner-reviewed synthetic COBOL-to-Java tasks planned for v1.
Baseline results
Forthcoming
Scores publish only after the corpus and verifier admission gates are frozen.
Source-oracle evaluation
One hidden workload. Two isolated executions. One behavioral verdict.
01
Task
COBOL repository, contract, and public cases
02
Agent
Produce a native Java candidate
03
Paired run
Execute source and target on the same hidden input
04
Verdict
Pass the task or expose the first divergence
The candidate runs in a target-only image without the COBOL runtime, source files, or network access. Verifiers must accept alternate correct implementations and reject wrappers, public-case hard coding, and COBOL-specific semantic mutants.
Planned v1 scope
Repository migration across three horizons.
These are release targets, not completed corpus statistics. Every task must pass determinism, acceptance-breadth, verifier-strength, and independent-review gates before freeze.
Development split
10 public tasks
Evaluation split
40 hosted tasks
Benchmark v1 release
Dataset and baseline results are in preparation.
The task corpus, verifier audit, full agent trajectories, and first leaderboard will be published together after the v1 suite is frozen. No placeholder scores will be presented as results.
01Public development tasks and a hosted evaluation split
02Reference and alternate-correct Java implementations
03COBOL-specific semantic mutants and verifier audit results
04Shared-harness baseline trajectories, cost, and failure analysis
05Versioned task cards, runner, comparator, and signed result bundles
Research and participation
Read the protocol, then bring a migration worth measuring.
The earlier STRUST-v1 study validates the differential verifier on controlled pairs. SemaMig-Bench turns that method into a shared evaluation for migration agents.