Evaluation Protocol

1. Purpose

This protocol defines the exact rules applied to every prospective Seismikon prediction to determine whether it should be counted as a true positive (TP), false positive (FP), or - when a real earthquake occurs in the forecast region but was not predicted - a false negative (FN). The rules are deliberately simple and deterministic: given a prediction record and an earthquake catalogue, any independent analyst with the same data must reach the same result.

All prospective predictions are issued before the forecast window opens. The protocol is applied retrospectively once the window closes; no retroactive adjustment of tolerances is permitted.

2. Prediction Qualification Criteria

The public Track Record endpoint (/api/track-record) returns the full set of prediction records for the requested time window, deduplicated by event_uid (most-recent write wins). Every returned record is scored - the browser evaluator does not apply an additional is_current or magnitude filter on the prediction side.

The following constraints apply at the data-loading level rather than the matching level:

• A record must carry a valid event_time. Records without a parseable target timestamp are skipped during evaluation.
• Records must have valid latitude and longitude values to participate in spatial matching.
• The is_current field is stored per-row and is available in the download export; analysts who wish to replicate the active forecast set at issuance can filter on it independently.

The minimum magnitude threshold applies to the real earthquake catalogue (Section 12), not to predictions. Predictions of any magnitude are eligible as match candidates.

3. Forecast Window

Each model run produces predictions with an associated run_tag (the timestamp at which the run completed). The forecast window is the 48-hour period starting at run_tag. A prediction's event_time must fall within this window for the record to be active (is_current = TRUE).

The forecast window is set at ingest time. It is never recalculated retroactively. A prediction's active status can only be changed manually by an administrator (for data quality corrections), and such changes are logged.

4. Spatial Tolerance

A real earthquake is considered spatially consistent with a prediction if the great-circle distance (Haversine) between the predicted coordinates (lat, lon) and the real earthquake epicentre is ≤ 100 km.

The Haversine formula is used with an Earth radius of 6,371 km. Depth is not used in the spatial match - only surface epicentre coordinates.

5. Magnitude Tolerance

A real earthquake is considered magnitude-consistent with a prediction if its reported magnitude satisfies:

Magnitude type is not differentiated at this protocol version; whatever magnitude value the USGS FDSN catalogue reports for the event is used directly. When USGS updates a magnitude after initial report, the revised value is used if available at the time of evaluation.

6. Time Tolerance

A real earthquake is considered temporally consistent with a prediction if the absolute difference between the real event time and the predicted event time satisfies:

All timestamps are compared in UTC. The tolerance is symmetric: a real event that precedes the predicted time by up to 36 hours counts the same as one that occurs afterward.

7. Depth Tolerance

A real earthquake is considered depth-consistent with a prediction if the absolute difference between the predicted focal depth and the real earthquake depth satisfies:

Depth from the USGS FDSN catalogue carries uncertainty, particularly for shallower events. The tolerance is intentionally wide to account for this. Both the predicted depth and the reported real depth are in kilometres below the surface.

8. Match Logic

A prediction is scored as a true positive (TP) if there exists at least one real earthquake in the USGS FDSN catalogue that simultaneously satisfies all four of:

• Spatial tolerance (Section 4), AND
• Magnitude tolerance (Section 5), AND
• Time tolerance (Section 6), AND
• Depth tolerance (Section 7).

A prediction for which no real earthquake satisfies all four criteria simultaneously is scored as a false positive (FP). Satisfying three out of four criteria does not yield a TP.

9. Matching Mode, Tie-breaking & One-to-One Assignment

The evaluator operates in one of two modes, configured by the site administrator:

• Default - one-to-one (Hopcroft-Karp maximum bipartite matching): Each prediction is matched to at most one real earthquake and each real earthquake is matched to at most one prediction. The algorithm finds a maximum cardinality matching - it maximises the number of matched pairs. When multiple maximum matchings of equal size are possible, the specific pairing selected is determined by the deterministic DFS traversal order over the adjacency lists. Candidates are enumerated in ascending real-event-index order within the ±time-tolerance window; no cost function is applied to break ties between equally-sized matchings. This means that two predictions equidistant from the same real event may be matched differently depending on their positions in the adjacency lists. Independent replication using a minimum-cost bipartite matching algorithm may produce different specific pairings (though the same total TP count in non-degenerate cases). This is the mode in effect unless an administrator has explicitly enabled multi-match.
• Optional - multi-match (greedy): Each prediction independently picks its nearest qualifying real earthquake (lowest composite score). Multiple predictions may map to the same real earthquake. In this mode a single real event can produce multiple TPs. This mode is intended for research use and is not the public default.

The current active matching mode is: multi-match (greedy).

The Track Record page always reflects the currently configured mode. The data download exposes the raw prediction records and the /api/track-record endpoint returns the active matching criteria, so independent analysts can reproduce the result under either mode.

10. Aftershock Handling

Aftershocks are not filtered from the earthquake catalogue used for matching. The natural time methodology (Bikos 2024) explicitly does not require catalogue declustering; the LSTM was trained on raw seismic sequences including aftershock swarms. Filtering aftershocks would misrepresent what the model was trained to predict.

Consequence: if the model issues a prediction that is subsequently matched to an aftershock of a larger event, it counts as a TP. This is by design and is noted as a limitation in the Limitations section.

11. False Negative (FN*) Counting

A strict FN rate cannot be computed for this model because Seismikon does not issue a prediction for every possible location within the operational region - only for the specific coordinates the model identifies as elevated risk. An earthquake that occurs in a location for which no prediction was issued is therefore ambiguous: it could reflect a model miss or simply a location that was not in scope for that run.

The Track Record page displays FN* - a liberal estimate computed as the number of real M ≥ 0.0 earthquakes within the operational bounding box during the evaluation period that were not matched to any active prediction. This figure intentionally over-counts misses (because it includes earthquakes the model was never intended to predict in that specific form), and it should be interpreted as an upper bound on false negatives.

12. Minimum Earthquake Magnitude Threshold

No minimum magnitude filter is currently applied to the real-earthquake catalogue used in evaluation. All earthquakes reported by USGS FDSN within the spatial and temporal windows are eligible as match candidates, regardless of magnitude.

This means a predicted M 5.0+ event can technically be matched against a smaller real earthquake if it falls within the magnitude tolerance window. Administrators can set a minimum threshold via the public metrics configuration panel to restrict evaluation to significant events only (e.g. M ≥ 5.0).

13. Metric Calculations

Metrics are computed from the TP, FP, and FN* counts as follows:

All metrics are computed client-side on the Track Record page directly from the public prediction records and the USGS FDSN earthquake feed. No pre-computed metric stored on our servers is used in the public-facing track record display; users can verify the computation by inspecting the browser JavaScript source.

14. Reference Catalogue & Data Sources

• Real earthquake data: USGS FDSN Earthquake API (earthquake.usgs.gov/fdsnws/event/1/) - queried in real time from the visitor's browser during Track Record evaluation.
• Prediction data: Seismikon's own prediction database, exported via the public /api/track-record JSON endpoint and available for bulk download on the Data Downloads page.
• Coordinate system: WGS-84 decimal degrees.
• Timezone: All timestamps stored and compared in UTC.

Protocol Version History