- Role
- Solo ML Infrastructure & Modeling Engineer
- Reported to
- Company CEO
- Project advisors
- Former Head of Algorithms at a Fortune 500 streaming company; former Head of Data Science at an online retailer acquired by Amazon for over $1 billion
The client who approached me for this project knew they were underutilizing the decade of company data they had, but weren't sure how to create a data science capability internally, given that their existing engineering team didn't have the bandwidth to double as a data science team.
On one hand, there was real excitement about building an ML and data analytics system from scratch. On the other, the task was colossal. I would be the only one executing, and it was necessary to build every step: data cleanup, housing, pipelines, modeling, inference architecture, and testing. The only way to do this well was to leverage AI for coding execution and for red-teaming architecture decisions along the way. Luckily, at this point, Claude was becoming very adept at exactly that kind of work.
The company's CEO brought in two of the best data science minds in the business to offer guidance and make sure we didn't go off the rails, and we set to work.
Architecture Overview
We began by mapping out an AWS-native foundation that could handle three jobs at once: housing a decade of historical data, transforming it into clean, model-ready features, and eventually serving live predictions back into the client's operations system.
The core pattern was simple: raw operational data flows into automated ETL pipelines (AWS Glue), which build a structured feature store (S3 + Athena). Orchestrated workflows (Step Functions) manage the pipeline end to end, feeding trained models (SageMaker) that serve real-time predictions through Lambda and API Gateway, with a caching layer (DynamoDB) to keep response times low enough for production use.
This gave us a system that could be extended job by job, new data sources, new features, new models, without re-architecting each time, which mattered given I was the only engineer building it.
ETL Pipelines and Data Backfill
Once the architecture was in place, the focus shifted to building out the ETL pipelines and backfilling years of historical data. A few problems came up along the way, and solving them ended up shaping how we approached the rest of the build:
- Pricing data was misleading. The source data stored wholesale costs in a field that looked like retail pricing. We caught it, traced the correct values to a different table, and had to backfill the affected historical data once it was fixed.
- Look-ahead bias in aggregate features. Early versions of some features used data that wouldn't have been available at the actual moment of prediction. We rebuilt those features so they only ever draw on information that existed at that point in time, which is essential for a model that's meant to predict real outcomes rather than memorize them.
- Schema changes silently forked our tables. Incompatible schema updates caused the data catalog to quietly create duplicate, hash-named tables instead of updating the original. Once we understood the pattern, we built a process to drop and re-crawl cleanly instead of letting stale duplicates pile up.
Each of these took days to diagnose but, once understood, became repeatable fixes. What would have taken a larger team months to work through, we resolved in weeks.
Modeling
We started with customer clustering models to build a foundational understanding of who the customers were and how they behaved. From there, we moved into predictive modeling aimed at anticipating specific customer decisions, the kind where even a 5 to 8 percent improvement in accuracy could meaningfully move revenue.
A few key findings shaped the final approach:
- Simpler features beat complex balancing. Rather than using techniques like SMOTE to correct for imbalanced outcomes, narrowing down to a smaller set of high-signal features consistently produced better results.
- Collaborative filtering underperformed. We tested several approaches built on the idea that similar customers make similar choices. In this domain, that assumption didn't hold. What actually drove predictions was the content of the items themselves, not patterns across customers. A content-based approach outperformed collaborative filtering significantly.
- The final model reached strong, production-grade accuracy (AUC of roughly 0.75), built on gradient-boosted decision trees (XGBoost) with 37 engineered features. It's especially strong at flagging likely negative outcomes, making it most valuable as a filter that catches probable misses before they happen, rather than a system trying to guess a perfect match every time.
Inference and Integration
The last major piece before ongoing testing and validation was making the model available to the operations system the company already relied on day to day. Working closely with the client's head of engineering, we built a real-time inference layer.
The structure worked like this: predictions for all relevant customer and inventory combinations are pre-computed and stored ahead of time. When a request comes in, the system checks a cache first. On a cache hit, response times drop by roughly 600x compared to computing a prediction from scratch, which made the difference between a system that was technically accurate and one that was actually usable in a live operational workflow. The whole thing is exposed through a secured API, so the existing ops system could call it directly without needing to know anything about the ML underneath.
Outcome
In under a year, working largely solo with AI as a force multiplier for execution, we took the client from an underused decade of raw data to a fully deployed, production-grade ML system: automated pipelines, a validated predictive model, and real-time inference wired directly into daily operations.
What began as a question of whether a small team could build a serious data science capability without hiring a data science team became a working answer: yes, with the right architecture, the right advisors, and the discipline to fix problems properly instead of working around them.