flare+

Rare events make liars of metrics

Big solar flares are uncommon. Over the fortnight of real data this system was trained on there were eight M class events, and on most days nothing much happened at all. That has an awkward consequence: a forecaster that answers "none" every single time will be right on most days, and any metric that rewards being right on most days will hand it a medal.

So the hard part of this project was never predicting flares. It was building a scoreboard that a model which knows nothing cannot win. I failed that on the first attempt, and published the result. This page is the post mortem.

Ingest, label, fit, serve

Two model families over one feature pipeline. Everything below is running code in the repository rather than a diagram of an intention.

The split is chronological, not random: survival_pipeline.py holds out the tail of the dataset and trains on the head, so the training set never contains the future. That part was right from the start. The scoring was not.

93% precision, from a model that was never asked

Here is the loop that produced the published figures. It is short enough to read in full, which is exactly why it survived review.

for pred in predictions:
    pred_time = pred["timestamp"]
    pred_window_end = pred_time + timedelta(hours=pred.get("window_hours", 48))
    window_actuals = [a for a in actuals
                      if pred_time <= a["peak_time"] <= pred_window_end]
    matches.append({"prediction": pred, "actuals": window_actuals,
                    "hit": len(window_actuals) > 0})

n_hits = sum(1 for m in matches if m["hit"])
precision = n_hits / n_predictions

A hit is recorded when a flare happened. Not when the model said one would. The forecast is carried into the loop and never read, so the model could have been replaced with a constant, or with nothing, without moving the number by a thousandth.

And window_hours was set to the model's full range, 168 hours. Over an active fortnight, a seven day window almost always contains a flare. Thirteen of the fourteen sampled windows did. Thirteen over fourteen is 0.929, which is the 93% precision that went into the deployment notes. It is not a property of the model. It is a property of the sun.

This is a synthetic fortnight, not the real run. I do not have the model's original per day outputs, so nothing here is a reconstruction of them. What it does reproduce is the one statistic that matters: at a 48 hour window, thirteen of fourteen windows contain a flare, and a forecaster emitting the same number every day therefore scores precision 0.929 and F1 0.963. Those are the published figures, earned by a model that knows nothing. The arithmetic is the same arithmetic as backtest_scoring.py, decision threshold fixed at 0.5.

Four numbers, withdrawn

These were in DEPLOYMENT.md, on the dashboard hero, and in two tabs of the app. They are gone from all of them. The section now opens with the only honest summary available: status, not yet measured.

The arithmetic behind them was internally consistent, which is what made it convincing. 13/14 is 0.929, 13/(13+3) is 0.8125, and those give F1 0.867 exactly. Every number checked out against every other number. None of them was measuring the model.

Score against what you get for free

A forecast is a probability, and a probability is not a decision. To count hits at all you have to turn it into a yes or a no with a stated threshold, over a stated horizon, because "a flare is coming" means nothing without "by when". The new scoring takes both as arguments and prints them at the top of every report.

That alone would not have caught this. What catches it is refusing to report any number on its own. Every score now sits next to climatology, which is what a forecaster gets for ignoring every input and repeating the base rate, and the headline is the fraction of climatology's error the model actually removed.

# the fraction of climatology's error the model removed. zero means the
# model is worth exactly as much as knowing the base rate and nothing
# more. below zero means it is worth less than that.
bss = (1 - brier / brier_climatology) if brier_climatology > 0 else 0.0

The CI gate reads that, not F1. F1 climbs on its own when events are common, so gating on it would pass a model for having been pointed at an active fortnight. The report also prints an explicit warning whenever answering "yes" every day would have scored at least as well, and sweeps twenty one thresholds, because one threshold is one arbitrary choice and a genuinely informative model separates the classes at more than one of them.

Extracted into src/models/backtest_scoring.py, pure standard library, no database and no modelling stack, so it can be tested on its own. Fourteen tests came with it. The load bearing one builds a forecaster that emits the same number every day and asserts that it cannot score well: pull request 42.

Eight events is not a measurement

The scoring is fixed. The model is still unmeasured, and saying so is the point of the exercise. Training ran on thirteen days, 28 October to 10 November 2025: 125 C class, 8 M class, 1 X class. Eight positives cannot support a claim about M class performance, whatever a backtest prints.

Reserved