DATA SCIENCE

From Tukey’s “future of data analysis” to today’s AI era: a discipline that blends statistics, computing, and domain knowledge to learn from data and drive decisions at scale.

175ZB Global datasphere by 2025 (IDC)
34% US data scientist job growth (2024–34)
1962→ Tukey formalizes modern data analysis
ε,δ Differential privacy as a guarantee

What is Data Science?

Data Science is the scientific study of extracting knowledge from data, integrating mathematics, statistics, computing, and domain expertise to generate predictions, explanations, and decisions. It spans the full path from raw signals to deployed, monitored models.

The core pillars: Mathematics (linear algebra, calculus, optimization, probability), Statistics (inference, uncertainty, design), Computer Science (algorithms, data systems), and Domain knowledge (problem framing, constraints).

Outcomes: inference • prediction • decision • explanation • visualization

Data Science Venn Diagram Three overlapping circles labeled Computer Science / IT, Math and Statistics, and Domains / Business Knowledge. The central intersection is labeled Data Science; the pairwise overlaps are labeled Machine Learning, Traditional Research, and Software & Data Products. Computer Science / IT Algorithms • Data Systems • DevOps Math and Statistics Probability • Inference • Optimization Domains / Business Knowledge Problem Framing • Constraints • Impact Machine Learning Traditional Research & Modeling Software & Data Products Data Science Mathematics × Statistics × Computing × Domain
Data Science merges Mathematics, Statistics, Computer Science, and Domain Expertise.
Mathematics is foundational: linear algebra, calculus, and probability theory underpin statistical inference and machine learning. Optimization and matrix methods drive nearly every modern algorithm.

Why Now?

Exponential data growth (web, sensors, genomics, logs) and accessible cloud compute created the conditions for modern analytics and AI. Organizations now operationalize data products that continuously learn and feed decisions.

At scale: petabyte data lakes, streaming pipelines, autoscaling training/inference, ML platforms, and governance for privacy, fairness, and reliability.

From ad-hoc analysis → repeatable, audited, production systems

Foundations & Milestones

1962 — John Tukey

“The Future of Data Analysis” reframes statistics around data-analytic practice and exploratory work.

1970s–90s — Data Analysis Culture

Cleveland, Chambers and colleagues advance graphics, EDA, and computing; databases become mainstream.

2001 — Breiman’s Two Cultures

Breiman contrasts data models vs. algorithmic models, urging validation by predictive accuracy.

2010s — Big Data & Deep Learning

Hadoop→Spark, GPUs, ImageNet breakthroughs, and end-to-end pipelines move models into production.

2017 — “50 Years of Data Science”

Donoho synthesizes the discipline’s activities and calls for rigorous, reproducible workflows.

2020s: Foundation models (LLMs, diffusion) + causal/Responsible-AI tooling bring a new synthesis.

The Scale

Global datasphere ≈ 175 ZB by 2025; Data scientist jobs +34% (US, 2024–34).

Demand for statistical + computational fluency keeps rising.

The Data Science Lifecycle

  1. Problem framing (objectives, metrics, constraints)
  2. Data acquisition (instrumentation, logs, surveys, experimental design)
  3. Cleaning & integration (ETL/ELT, schema, missingness, leakage control)
  4. EDA & visualization (patterns, anomalies, priors)
  5. Modeling (statistical + ML; CV; hyperparameters)
  6. Evaluation (offline metrics; uncertainty; fairness; ablation)
  7. Deployment (APIs, batch/stream, guardrails)
  8. Monitoring (drift, performance, incidents, retraining cadence)
  9. Governance (documentation, approvals, audits)
Design of Experiments and A/B testing connect modeling to causal impact in production.

Measurement & Metrics

Choose metrics aligned with decisions: accuracy/ROC/PR, calibration, cost-sensitive loss, uplift, time-to-detect, latency, energy, and risk budgets.

Calibration + uncertainty are often more valuable than a small accuracy gain.

Statistical Methods (Core)

Inference: likelihood, estimation (MLE/REML/Bayes), confidence/credible intervals.

Regression: linear/logistic/Poisson; regularization (ridge/LASSO/elastic net).

Design: randomization, blocking, power, multiple testing control (FDR).

Multivariate: PCA/FA, clustering (k-means, hierarchical, Gaussian mixtures), EM.

Time series: ARIMA/ETS/state space; anomalies; hierarchical forecasting.

Bayesian workflow: prior→likelihood→posterior; posterior predictive checks; hierarchical shrinkage; MCMC/VI.

Generalization

Cross-validation, bootstrap, learning curves, regularization paths, and domain shift assessment (IID→OOS) ensure reliability beyond the training set.

Prefer simple, well-calibrated models with honest uncertainty to brittle overfit wonders.

Machine Learning & Deep Learning

Supervised: trees/ensembles (RF/GBM), SVMs, GLMs, neural nets.

Unsupervised: clustering, density modeling, manifold learning.

Representation learning: embeddings, autoencoders, transformers.

Foundation models: LLMs and diffusion models as pre-trained priors; adapt via finetuning, adapters, retrieval, or prompting.

Evaluation: holdout tasks + robustness, calibration, fairness, privacy, and energy/latency—especially for LLM inference.

From Accuracy to Utility

Operational usefulness depends on decision-aware metrics, human factors, and cost constraints. Log-loss, expected utility, and policy value often dominate top-1 accuracy.

“The best model” is the best system, not just the best score.

Causality & Experiments

Randomized trials for clean identification. When not feasible: observational causal inference—propensity scores, matching, IVs, diff-in-diff, synthetic controls, front-door/back-door criteria.

Decision focus: uplift modeling & heterogeneous treatment effects (meta-learners; Bayesian hierarchical models) for “who benefits?”

Breiman’s Two Cultures (Revisited)

Blend data models for explanation and algorithmic models for prediction; validate each by its purpose and out-of-sample performance.

Fairness & Accountability

Equalized odds: equal false-positive/false-negative rates across protected groups; implement via post-processing or constrained training.

Disclose intended use, limitations, and data provenance; log decisions; enable recourse.

Governance: model cards, datasheets, risk registers, incident response, human-in-the-loop escalation.

Privacy & Safety

Differential Privacy (ε,δ) offers a mathematical guarantee that an individual’s inclusion changes outputs only slightly; combine with secure enclaves, federated learning, and robust evaluation against leakage.

Energy & footprint: track training and inference cost; seek efficient architectures and deployment (distillation, quantization, caching, batching).

Policy & practice: align with legal frameworks and internal standards; document privacy budgets and retention policies.

Workflow & Tooling

Data: SQL/lakes, parquet, catalogs; versioned data (Delta/Iceberg).

Code: notebooks + scripts; testing; packaging; containers.

ML Ops: feature stores, experiment tracking, CI/CD, canary/shadow deploys, monitors (drift, bias, latency, cost).

Viz: grammar-of-graphics, dashboards, narrative documents for stakeholders.

Reproducibility: seeds, env locks, lineage, and signed artifacts ease audits and replication.

Team Practices

Small cross-functional squads (data + ML + domain + infra). Write Decision Logs, maintain Model Cards, and run post-mortems for incidents and near-misses.

Great data science is a team sport with strong documentation.

Case Sketches

Public Health

Forecast admissions; detect anomalies; evaluate policy changes via difference-in-differences.

Manufacturing

Predictive maintenance from sensor streams; anomaly detection; causal impact of process tweaks.

Education

Early-warning indicators; uplift models for targeted interventions; RCTs for curricular changes.

Finance

Credit risk + fairness constraints; calibrated PD/LGD; drift monitoring under regime shifts.

From Notebook to Production

Refactor EDA into robust pipelines; version features; add tests; containerize; deploy with canaries; monitor; iterate.

If it isn’t monitored, it isn’t in production.

Frontiers (2025→)

LLMs as interfaces to data & models; agentic systems with tool use; efficient AI with large energy savings via smarter training/inference; privacy-preserving collaborative modeling.

Causal + generative synthesis (counterfactuals; policy optimization); auto-ML with guardrails; quantitative governance (risk, compliance, assurance).

Education: data science curricula deepen statistical foundations (design, inference, uncertainty) alongside practical ML, ethics, and governance.

Advice to Students & Teams

Master probability, inference, and linear algebra; practice honest validation and clear communication; build systems, not just models; document everything.

Clarity, calibration, causality, and care.