The situation
The science was sound and the model was proven. The problem was everything around it. The forecast ran as one long notebook on one machine, so a failure anywhere meant starting over, and nobody could run two forecasts at once. A 36-hour turnaround made it impossible to offer customers anything close to a daily product.
What I did
- Broke the notebook into discrete processing stages and mapped their real data dependencies, which were fewer than anyone assumed.
- Rebuilt the pipeline on Step Functions for orchestration, Lambda for the processing stages, and EventBridge for scheduling, so stages run in parallel and a failed stage retries on its own.
- Moved intermediate data to S3 with a layout the scientists could still open and inspect directly.
- Delivered the whole thing as Terraform, with the scientists able to deploy a change to a stage without touching infrastructure.
The outcome
Forecasts now complete in under 30 minutes and multiple regions can run at once. That changed what the company could sell: a daily forecast instead of a weekly one. The team owns the pipeline and has extended it since without me.