Evaluation infrastructure for agent-written software

Agents write the change. Strust shows what it did.

Strust closes the testing loop: deterministic feedback for agents, behavioral diffs for reviewers, and approved examples that become the next eval. Built first for COBOL-to-Java migration.

Run an example changeContact sales

One change through Strust

Agent + reviewer views

One change. The agent fixes it; the reviewer decides it.

Agent view isolates the first changed decision and the next patch. Reviewer view translates that evidence into impact and a human decision.

COBOL to Java

Agent receives

Trace-local cause, the first divergent decision, a suggested patch, and the exact rerun command.

Agent diagnosis / F-02

12 divergent traces

First divergence: Branch action

Batch branch B-17 / null-account path · Trace 417 / 1,215

Cause isolated. Patch the changed branch and rerun.

Reference system

COBOL

IF ACCOUNT-ID = SPACES

NEXT SENTENCE

ELSE PERFORM WRITE-CLAIM

END-IF

Candidate system

Java

if (accountId == null) {

writeFallback(claim);

} else { writeClaim(claim); }

batch.advance();

Execution checkpoints

Source value / target value

  1. NULL

    Account input

    NULL

    Equivalent

  2. TRUE

    Guard B-17

    TRUE

    Equivalent

  3. SKIP_WRITE

    Branch action

    WRITE_FALLBACK

    Origin

  4. 0

    Rows emitted

    1

    Changed

  5. 4,808

    Batch count

    4,809

    Changed

First divergence

The first changed decision is Branch action: the source returns SKIP_WRITE while Java calls writeFallback.

Capture → Compare → Isolate → Patch → Rerun

02

Use the lightest check that can answer the question.

Change criticality

Selected feedback

Add replay and broader input coverage.

4/ 5

  1. Types + static analysis

    Global invariants

    Immediate

  2. Expect-style examples

    Readable behavior

    Milliseconds

  3. Deterministic replay

    Controlled time + effects

    Seconds

  4. Property + fuzz checks

    Input exploration

    Selective

  5. Historical + formal checks

    Highest assurance

    Critical paths

Evaluator firewall

The agent does not grade itself.

Strust separates implementation changes, eval changes, and new expected behavior. If a patch changes both the code and the rule judging it, that decision is surfaced for human review.

Implementation diffEval diffHuman approval
03

An eval is a readable example of intent.

.strust/evals/null-account-output.evalRepository eval case
case "null account emits no row"

risk critical
input account_id = null
expect rows = 0
expect effects = []
owner payments-platform

Text first

The test is also the explanation.

Each case records why the behavior matters, the controlled input, expected outputs and side effects, its owner, and its criticality. Humans can read it. Agents can act on it.

Survives after the reference system is retired

CLI

strust eval --changed

Agent tool

Focused failure context

CI

Risk-sensitive gates

Pull request

Behavioral diff

04

Migration was the first proving ground.

First product laneCOBOL → Java
COBOL — observed
01 CUSTOMER-CODE PIC X(8).

IF CUSTOMER-CODE = 'VIP'
  MOVE 'PRIORITY' TO QUEUE-NAME
Java — converted
String code = record.customerCode();
if (code.equals("VIP")) { // ← flagged
  queueName = "PRIORITY";
}

A legacy system is already an executable reference. Strust turns representative source behavior into evals for the system replacing it. Here, COBOL pads the shorter fixed-width value; Java treats VIP····· and VIP as different strings. Both compile; the behavior changes.

✓Capture the source system as executable behavior
✓Review each source-to-target divergence
✓Keep approved cases for every later change
Read the original migration research

Careers · 3 open roles

We are hiring engineers who care about behavior.

Join Strust as a Member of Technical Staff or Software Engineering Intern and help make agent-written software easier to test and understand.

View open roles

Behavioral eval infrastructure

Bring one important workflow into the loop.

Start with a migration, refactor, or agent-written system. Strust returns focused feedback to the agent and a behavior change the responsible human can understand.

Discuss your eval stackRun the example