Evaluate translations with operator answers
The Test Suite accepts UTF-8 CSV files as well as existing Excel test workbooks.
The interaction format records the question an operator reviewed and the answer to supply.
It follows the production interaction API, including current interaction revisions.
Run a test in the interface
- Download the interaction example CSV, or use a converted demo CSV.
- In Test Suite, select the CSV and press Upload & validate. This parses the table; it does not translate the rows.
- Choose Hybrid v3. Leave model assistance unchecked for a repeatable local evaluation.
- Choose First response only to measure the initial response, or Use reviewed answers from CSV to complete the scripted conversations.
- Press Start test. Download the result CSV and the full question/answer traces when it finishes. Stop keeps partial results.
The results show initial and final exact matches separately. A successful clarification does not count as first-turn success.
The result CSV preserves the original columns and appends Eval … columns. Full traces retain original responses,
including requested separator rendering. Standalone TACK is displayed as a hyphen only in test comparisons and reports.
Eval Reference Output uses the same display normalization as the generated outputs;
the original Command and English cells remain available unchanged.
Legacy XLSX tests retain their existing workbook workflow. Convert to interaction CSV to supply reviewed answers.
For CSV tests, Direction selects encode or decode for each row; the family hint from a sheet name is unnecessary.
CSV format
Retain Command (or Code Sequence) and English (or Reference English).
For encoding, English is the input and Command is the reference used only after execution.
For decoding, Command is the input and English is the reference used only after execution.
| Column | Meaning |
|---|
| Interaction Schema Version | mtp-interactions/v1 |
| Case ID | Unique stable ID. Subsets should retain a link to their main case. |
| Direction | encode or decode |
| Script Mode | scripted or one-shot; used by the CLI's fixture mode. The UI's chosen interaction mode overrides this. |
| Call Sign Notation | Optional per-case none, C/S, or CALL SIGN. A blank cell uses the run setting. This requests the output representation; comparisons still retain every callsign token. |
| Questions JSON | Ordered array of exact reviewed prompt guards: kind, question text, offered labels, and field definitions. |
| Answers JSON | Matching ordered array of explicit actions and field answers or displayed choice labels. Never save runtime option IDs. |
| Transmission Context JSON | Optional operating evidence, such as flaghoist role and usage. Context is an input, with its provenance recorded. |
| Answer Basis / Book Reference | Explain how answers were obtained. Mark additional operating facts absent from the original sentence. |
| Reference Status / Reference Issue | Record book disputes separately. disputed references cannot pass. A nonconformant signal can pass an explicitly expected rejection test; it cannot pass as a translated original. |
| Proposed Corrected Command / English | Review suggestions only; execution and strict scoring continue to use the original columns. |
| Expected Outcome | Optional terminal status, e.g. translated or nonconformant. Explicit rejection tests are separate from translation matches. |
For a field question, an answer can be {"action":"submit","values":{"speed_knots":13}}.
A later confirmation needs its own reviewed question and {"action":"confirm"}.
A choice uses {"action":"submit","option_label":"the exact displayed meaning"}.
A select field value uses {"option_label":"the exact displayed label"}.
Ordinary CSV quoting applies to JSON cells; use a CSV writer or edit in a spreadsheet editor.
Numeric answers use the unit shown by the question. For a field labelled in yards,
enter 300 for 300 yards. An explicit quantity such as 300 yards
has the same meaning in an authored answer. Counts must be whole numbers where the
question asks for a number of people, aircraft or other objects.
The runner refuses changed questions, missing or unused answers, ambiguous labels, and incomplete field sets.
It never chooses the first option automatically. A recovery confirmation cannot make a nonconformant original signal valid.
A corrected-input answer is explicit and remains visible in the trace.
Run the same workflow locally
From the project directory, use the project's Python environment with the web application's dependencies installed:
python tools/run_interaction_regression.py \
web_translator/app/static/interaction-template.csv \
--output outputs/my-interaction-run --mode scripted --assistance off
python tools/run_interaction_regression.py \
outputs/interactive-evaluation-20261018/converted/demo1_encode_comparison.interactions.csv \
outputs/interactive-evaluation-20261018/converted/demo2_encode_comparison.interactions.csv \
--output outputs/my-demo-run --mode scripted --assistance off
Add --mode one-shot for the initial-response comparison, --timeout 120 for the hard per-case limit,
or --fail-on-mismatch when every selected case is required to pass.
Each run writes scored CSVs, raw JSONL responses, summaries and an implementation/source manifest.
Reusing an output folder resumes only if sources, implementation and settings match exactly; otherwise choose a fresh folder.
Use --backend http --base-url http://127.0.0.1:8000 --token-env MTP_TEST_TOKEN
to test a running server after setting that environment variable to your login token. Local mode needs no running server or Docker.