Keep a simulation result reproducible across tool changes

A simulation result can change when the vehicle software stays the same. A sensor model, road description, solver setting or evaluation script may have changed. If those inputs are not preserved, the programme can spend time investigating a product regression that actually originated in its virtual environment.

Treat a simulation run as an experiment with an identifiable configuration. The scenario is one input among several. Reproducibility requires a record sufficient to reconstruct the relevant environment or explain precisely why a later run differs.

Identify the full experiment

Record the system under test, scenario, environment models and evaluation logic. Include versions and configuration for the components that influence the intended conclusion. Decide which random or variable inputs need controlled seeds or recorded values so the run's behaviour can be understood later.

ASAM OpenSCENARIO describes dynamic scenarios, while OpenDRIVE describes road-network information. These standards organise important parts of an experiment. Neither a scenario file nor a road file alone defines every model and setting that a simulator will use.

Distinguish product and environment baselines

Maintain a baseline for the vehicle function separately from the simulation environment. When either changes, document the reason and expected effect. A team needs to know whether it is comparing two product versions in one environment or comparing two environments around one product version.

Avoid changing several influential inputs without preserving an interpretable transition. Where multiple changes are necessary, plan checks that help identify their contributions. This does not require freezing tools indefinitely; it requires acknowledging that the test environment has behaviour that can change alongside the product.

Version the evaluation as carefully as the scenario

The criterion used to judge a run can affect the reported outcome even when the recorded behaviour is identical. Preserve the evaluation script or method and its relevant settings. Explain whether a reported comparison uses the original criterion or a revised interpretation.

Retain enough raw evidence to revisit a conclusion when an evaluation defect is found. A pass label by itself provides little opportunity for reanalysis. The required retention should follow the importance of the decision and the programme's evidence needs, rather than a blanket assumption that every possible output must be stored forever.

Verify repeatability and explain limits

Repeat selected runs to understand variation. If the environment is not fully deterministic, determine how much variation is expected and how the decision accounts for it. A different outcome should prompt investigation rather than an automatic choice of the more favourable run.

Include supplier-controlled models in that review. The customer needs a practical notification route for changes that could alter an existing result, even when the supplier retains proprietary implementation details.

Keep reproduction instructions usable by another team. Tools, access and external dependencies can disappear even when configuration files remain. Reproducibility is therefore an operating capability, not a label on a test report. It gives reviewers a clear route to separate product behaviour from modelling and evaluation changes, making simulation evidence more useful across programme and supplier boundaries.