Walk me through your framework for designing a machine learning system from an ambiguous business problem. What key decisions do you make at each stage?

Question

Accepted Answer

I follow a structured six-stage framework when designing ML systems from ambiguous requirements.

The first stage is problem definition and scoping. I ask clarifying questions to understand the business objective, success metrics, and constraints. Key decisions include whether ML is actually needed, what the primary optimization target is, and what latency or throughput requirements exist. I translate business goals into measurable ML metrics, understanding that improving user engagement might translate to maximizing click-through rate or session duration.

The second stage is data strategy. I determine what data is available, what needs to be collected, and how to handle labeling. Key decisions involve label quality versus quantity tradeoffs, handling class imbalance, and defining data freshness requirements. I also identify potential data leakage sources and design collection pipelines that avoid label contamination.

The third stage is feature engineering and representation. I decide how to encode different data types, what feature stores to use, and how to handle real-time versus batch features. Key decisions include embedding dimensions, handling missing values, and feature normalization strategies. I consider both offline feature computation for training and online feature serving for inference.

The fourth stage is model architecture selection. Based on the problem type and constraints, I choose appropriate model families. Key decisions involve model complexity versus interpretability, single model versus ensemble, and whether to use pre-trained models. I consider compute budget for both training and inference when making these choices.

The fifth stage is training and evaluation strategy. I design the evaluation framework including validation splits, online versus offline metrics, and A/B testing plans. Key decisions include how to simulate production conditions during evaluation, handling temporal data splits, and defining minimum performance thresholds for deployment.

The sixth stage is serving and monitoring architecture. I design the inference pipeline, caching strategies, and monitoring systems. Key decisions include deployment strategy such as shadow or canary, fallback mechanisms, and drift detection approaches. I also plan for model updates and retraining triggers.

Throughout all stages, I document assumptions and risks, and I ensure alignment with stakeholders on tradeoffs between accuracy, latency, cost, and maintainability.