Behavioral preservation benchmark
SemaMig-Bench
The executable COBOL source and native Java candidate run on the same held-out workloads. SemaMig-Bench scores contract-relevant behavior, not resemblance or compilation alone.
Measurement contract
Compilation is an input. Behavior is the score.
- Executable reference
- A pinned COBOL system establishes the observed source behavior.
- Native candidate
- The generated Java system runs independently in a target-only environment.
- Behavior contract
- Declared outputs, state changes, ordering, messages, and failure behavior.
- Task verdict
- A task passes only when every required held-out case agrees.
- Compilation without behavioral agreement
- Source-code similarity
- Translated unit tests alone
- A wrapper around the original runtime
Source-oracle evaluation
One hidden workload. Two isolated executions. One behavioral verdict.
- Task
COBOL repository, contract, and public cases
- Agent
Produce a native Java candidate
- Paired run
Execute source and target on the same hidden input
- Verdict
Pass the task or expose the first divergence
The candidate runs without the COBOL runtime, source files, or network access. The comparator evaluates only observations named by the behavior contract.
Corpus design
Three horizons. One preservation contract.
Tier describes implementation horizon, not presumed difficulty. A short packed-decimal case may be harder than a regular multi-file workflow.
One or two programs isolating a semantic hazard
Multi-file business workflow with shared records or state
Multiple programs, copybooks, and batch or service boundaries
Verifier admission
A task enters only after the evaluator proves it can tell right from wrong.
Determinism
Repeated source runs under the same image, seed, locale, and fixture produce identical observations.
Acceptance breadth
A reference implementation and a structurally different correct Java implementation both pass.
Verifier strength
Wrappers, public-case hard coding, null candidates, and declared semantic mutants fail.
Independent review
Reviewers who did not author the candidate approve the contract, hidden partitions, and evidence.
Result bundles identify task versions, artifact digests, exclusions, timing, cost, and verifier audit evidence so every published verdict has a reviewable chain of custody.
Read the specification. Bring a migration worth measuring.
The STRUST-v1 study develops the differential-verification method. SemaMig-Bench applies that method to comparison across migration agents and coding products.
