From Tukey’s “future of data analysis” to today’s AI era: a discipline that blends statistics,
computing, and domain knowledge to learn from data and drive decisions at scale.
175ZBGlobal datasphere by 2025 (IDC)
34%US data scientist job growth (2024–34)
1962→Tukey formalizes modern data analysis
ε,δDifferential privacy as a guarantee
What is Data Science?
Data Science is the scientific study of extracting knowledge from data, integrating
mathematics, statistics, computing, and
domain expertise to generate predictions, explanations, and decisions.
It spans the full path from raw signals to deployed, monitored models.
The core pillars:Mathematics (linear algebra, calculus, optimization, probability),
Statistics (inference, uncertainty, design),
Computer Science (algorithms, data systems), and
Domain knowledge (problem framing, constraints).
Data Science merges Mathematics, Statistics,
Computer Science, and Domain Expertise.
Mathematics is foundational: linear algebra, calculus, and probability theory underpin
statistical inference and machine learning. Optimization and matrix methods drive nearly every modern algorithm.
Why Now?
Exponential data growth (web, sensors, genomics, logs) and accessible cloud compute created the conditions
for modern analytics and AI. Organizations now operationalize data products that continuously learn
and feed decisions.
At scale: petabyte data lakes, streaming pipelines, autoscaling training/inference,
ML platforms, and governance for privacy, fairness, and reliability.
From ad-hoc analysis → repeatable, audited, production systems
Foundations & Milestones
1962 — John Tukey
“The Future of Data Analysis” reframes statistics around data-analytic practice and exploratory work.
1970s–90s — Data Analysis Culture
Cleveland, Chambers and colleagues advance graphics, EDA, and computing; databases become mainstream.
2001 — Breiman’s Two Cultures
Breiman contrasts data models vs. algorithmic models, urging validation by predictive accuracy.
2010s — Big Data & Deep Learning
Hadoop→Spark, GPUs, ImageNet breakthroughs, and end-to-end pipelines move models into production.
2017 — “50 Years of Data Science”
Donoho synthesizes the discipline’s activities and calls for rigorous, reproducible workflows.
2020s: Foundation models (LLMs, diffusion) + causal/Responsible-AI tooling bring a new synthesis.
The Scale
Global datasphere ≈ 175 ZB by 2025; Data scientist jobs +34% (US, 2024–34).
Demand for statistical + computational fluency keeps rising.
The Data Science Lifecycle
Problem framing (objectives, metrics, constraints)
Data acquisition (instrumentation, logs, surveys, experimental design)
Foundation models: LLMs and diffusion models as pre-trained priors; adapt via finetuning, adapters, retrieval, or prompting.
Evaluation: holdout tasks + robustness, calibration, fairness, privacy, and energy/latency—especially for LLM inference.
From Accuracy to Utility
Operational usefulness depends on decision-aware metrics, human factors, and cost constraints. Log-loss, expected utility, and policy value often dominate top-1 accuracy.
“The best model” is the best system, not just the best score.
Causality & Experiments
Randomized trials for clean identification. When not feasible: observational causal inference—propensity scores, matching, IVs, diff-in-diff, synthetic controls, front-door/back-door criteria.
Blend data models for explanation and algorithmic models for prediction; validate each by its purpose and out-of-sample performance.
Fairness & Accountability
Equalized odds: equal false-positive/false-negative rates across protected groups; implement via post-processing or constrained training.
Disclose intended use, limitations, and data provenance; log decisions; enable recourse.
Governance: model cards, datasheets, risk registers, incident response, human-in-the-loop escalation.
Privacy & Safety
Differential Privacy (ε,δ) offers a mathematical guarantee that an individual’s inclusion changes outputs only slightly; combine with secure enclaves, federated learning, and robust evaluation against leakage.
Energy & footprint: track training and inference cost; seek efficient architectures and deployment (distillation, quantization, caching, batching).
Policy & practice: align with legal frameworks and internal standards; document privacy budgets and retention policies.
Workflow & Tooling
Data: SQL/lakes, parquet, catalogs; versioned data (Delta/Iceberg).
Viz: grammar-of-graphics, dashboards, narrative documents for stakeholders.
Reproducibility: seeds, env locks, lineage, and signed artifacts ease audits and replication.
Team Practices
Small cross-functional squads (data + ML + domain + infra). Write Decision Logs, maintain Model Cards, and run post-mortems for incidents and near-misses.
Great data science is a team sport with strong documentation.
Case Sketches
Public Health
Forecast admissions; detect anomalies; evaluate policy changes via difference-in-differences.
Manufacturing
Predictive maintenance from sensor streams; anomaly detection; causal impact of process tweaks.
Education
Early-warning indicators; uplift models for targeted interventions; RCTs for curricular changes.
Refactor EDA into robust pipelines; version features; add tests; containerize; deploy with canaries; monitor; iterate.
If it isn’t monitored, it isn’t in production.
Frontiers (2025→)
LLMs as interfaces to data & models; agentic systems with tool use; efficient AI with large energy savings via smarter training/inference; privacy-preserving collaborative modeling.
Education: data science curricula deepen statistical foundations (design, inference, uncertainty) alongside practical ML, ethics, and governance.
Advice to Students & Teams
Master probability, inference, and linear algebra; practice honest validation and clear communication; build systems, not just models; document everything.