What is backtesting and why is it critical?

What is backtesting?

Backtesting is the practice of running a predictive model against historical data to see how its predictions would have compared to what actually happened. The alternative is forward testing: forecast an upcoming date, then wait for that date to arrive before you know if the forecast was right. That works, but it's slow. If you want to test a forecast at a 90-day lead time, you wait 90 days for a single data point. Extrapolate that to collecting data across all 12 months, and you have to wait a full year. Testing across many lead times and many properties this way would take months or years to accumulate enough evidence to trust the results. It also means if continuous improvements to the model are being pushed out, any existing data is immediately irrelevant for validating performance.

Backtesting shortcuts that wait and allows for immediate model validation. Because the historical period being tested already happened, you can score the forecast immediately, and you can generate many test points at once by picking different past dates as your "forecast origin" rather than waiting for each one to occur in real time.

The idea shows up anywhere someone has to decide whether to trust a model before it affects a real decision: quantitative trading, weather prediction, epidemiology, demand forecasting. The mechanics are similar in each case. A model can perform differently depending on the sample of time periods or conditions it's tested against, so backtesting is typically done across a range of periods and conditions rather than a single one.

Why is backtesting critical for accurate forecasting?

A forecast's accuracy number alone doesn't tell you how it behaves. Two forecasts can carry the same average error and behave differently in practice. One might be off by a small, consistent amount every time. The other might be right more often than not, but occasionally far off.

It's also possible to have a directional bias within a forecast. If a forecast is consistently over or understated, the relative weakness or strength of demand could be misinterpreted and, ultimately, lead to suboptimal pricing. Backtesting is one of the main ways to surface these differences.

It's also how you compare a new model to one already in use. Running both against the same historical data shows where one outperforms the other, and where it doesn't.

How does FLYR Hospitality approach backtesting?

Our production backtest runner evaluates the current forecasting model against a rolling window of historical outcomes, broken out by how far in advance the forecast was made. A forecast made 30 days before a stay date and one made 90 days before are solving different problems with different error patterns, so we evaluate them separately rather than blending everything into one number that hides which part of the system actually needs attention.

We track MAPE¹ and symmetric MAPE (sMAPE)² to measure how accurate the forecast is, since sMAPE holds up better than standard MAPE during low-demand periods, and mean bias error (MBE)³ to see whether the model consistently forecasts too high or too low. A model that's reliably off by a small, predictable amount is more useful in practice than one that's right half the time and badly wrong the rest, even if their average error looks the same on paper.

Want to learn more? See: Why do we track both MAPE and sMAPE?

This runs on a monthly cadence, per property, not as a one-time check. That continuous backtest feeds back into the model's learning, so a property's forecast keeps getting evaluated and refined against its own track record over time, rather than being scored once and left alone.

It's also the mechanism we use to decide whether a new model is allowed to replace an old one. When our science team evaluates a candidate forecasting model, they run it against the current production model across multiple, separately sampled groups of properties, then compare results in each lead-time bucket independently, because a model can win at long lead times and lose at short ones, or perform differently depending on which properties happen to be in the sample.

In addition to datapoints like MAPE and sMAPE, part of that evaluation includes checking residuals⁴ for patterns, since a model that's consistently off on certain days or property types has a fixable problem, not just noise. When a candidate model adds a new input, we also look at its partial R⁵ to confirm that input is actually earning its place, rather than adding complexity without adding accuracy. A model only earns a production rollout in the lead-time range where it actually demonstrates an improvement. If a candidate looks better in one sample and roughly equivalent in another, that gets reported as exactly that, not rounded up to "better."

We recently went through this process testing a new forecasting approach against our existing production model, and the result was a partial rollout: switch for the lead times where the new model clearly outperformed, hold the rest for further testing.

Because this backtesting runs continuously on a property's own data starting at onboarding, a new property doesn't have to wait months to build up a track record before its forecast can be trusted enough to automate. Our backtesting methodology starts generating that track record from day one.

Learn more about FLYR Hospitality Optimize

Glossary

1. MAPE — The average size of a forecast's error, shown as a percentage.
2. sMAPE (symmetric MAPE) — A version of MAPE that treats over- and under-forecasting equally, and holds up better than MAPE during low-demand periods.
3. MBE (mean bias error) — Whether a model tends to consistently forecast too high or too low, on average.
4. Residuals — The difference between what actually happened and what the model predicted.
5. Partial R² — How much one specific input adds to a model's accuracy, on its own.