Prospective vs. Retrospective Testing

Retrospective (or back-test) analysis evaluates a model's performance against historical data that was available when the model was built. In earthquake science, this means training a model on seismic catalogues from past decades and measuring how well it "predicts" earthquakes that are already known to have occurred.

Retrospective results are necessary and useful for model development - you cannot deploy a model with no performance history. But they are inherently optimistic: the model has, in some sense, been shaped by the data it is tested on, either through direct training or through design choices made by researchers who knew the data.

Prospective testing means issuing a prediction before the outcome is known, then recording whether the prediction was correct when the outcome is revealed. The model had no access to the future data at prediction time.

Prospective performance is the scientific gold standard because it eliminates all forms of look-ahead bias. A model that achieves 98% precision in a retrospective back-test but 30% in prospective testing has overfit its training data; its back-test claim is scientifically misleading.

Seismikon is explicit about which figures are retrospective and which are prospective:

Any earthquake prediction system - or any forecasting system in any domain - should be evaluated primarily on its prospective performance. If a vendor only cites back-test figures, ask how long the prospective record is and whether the evaluation protocol was fixed before predictions were issued. Seismikon publishes both, clearly labelled, so users can make informed assessments.