348: History of Data Science - Part 4

Episode Summary

Executive Summary: The episode traces data science from 2015 to 2020, showing its shift from hype-driven hiring and vague job specs to clearer roles, broader skills, and rising ethical responsibility. It highlights Dave Donoho’s definition-by-what-it-is-not, the growing dominance of Python and machine learning, AI’s rise as the next major technology wave, and the enduring view that data science is just beginning, not peaking.

Main Topics: From hype to maturity in data science hiring: The episode explains how organizations moved from seeking unrealistic 'data science unicorns' to defining more specialized roles and skill needs as the field matured. Dave Donoho’s clarification of data science: Donoho’s 2015 contribution is presented as a key step in defining data science by rejecting simplistic notions that it is only big data, only computing, or only statistics. Tool and language shifts: Hadoop to Python: The discussion contrasts early-2010s demand for SQL and Hadoop with the rapid rise of Python, which became central to data science and broader programming. AI takes center stage: Around 2017, artificial intelligence becomes the dominant hype topic, bringing new concerns about job displacement, ethics, interpretability, and robotics. Practical and ethical value of data science: The episode emphasizes that data science and AI improve customer service, operations, and healthcare, while also requiring bias awareness and ethical judgment. Data science as an expanding long-term trend: Using the electricity analogy, the speaker argues that AI and data science are early in adoption and will become ubiquitous across businesses. Education and specialization in the ecosystem: The host notes that more education for both practitioners and managers has reduced unrealistic expectations and enabled clearer career specialization.

Key Arguments: Data science hiring improved as organizations learned to specify needs more realistically, reducing the demand for impossible 'unicorn' candidates. Dave Donoho’s 2015 framing was important because it clarified data science by showing what it is not: not just big data, not just computing, and not just statistics/analytics. Python’s rise reflects a broader shift away from Hadoop-centric big data infrastructure toward more flexible, practical machine learning workflows. AI became the new center of technology hype around 2017, shifting the conversation from data infrastructure to automation, ethics, interpretability, and job impact. Data science and AI have clear real-world benefits in customer service, healthcare, and operational efficiency, not merely theoretical or hype-driven value. Ethics is now a core competency for data professionals; bias, accountability, and responsible use of data are no longer optional. The field is not at its peak; it is in early stages of adoption and will spread much like electricity did across homes and businesses.

Data Points: Time period covered: 2015–2020 - The episode focuses on the most recent five years in the history of data science. First Super Data Science course launch: May/June 2015 - The host mentions this as a brief personal milestone during the discussion of education in data science. Best job in America: 2016 - Glassdoor named data scientist the best job in America. Early in-demand skills: SQL and Hadoop - The host says these were the most in-demand data science skills around the start of 2016. Python release year: 1991 - Python is described as a long-standing language that rose sharply in popularity later. AI health platform claim: Almost a week earlier - A Canadian AI health-monitoring platform claimed it predicted a coronavirus outbreak nearly a week before major traditional health organizations. Electric power in US homes: Half of homes by 1925 - Used as an analogy for technology adoption and why AI will spread widely. Forecast for Python popularity: Around 2023 - The host predicts Python could overtake C and Java to become the most popular overall language. Forecast for business AI adoption: 10–20 years - The episode predicts most businesses will use AI/data science within this timeframe.

Pivotal Quotes: "AI is the new electricity." — Andrew Ng: Used to argue that AI adoption will become as ubiquitous and transformative as electricity. "Data science is not just about big data." — David Donoho: One of Donoho’s three clarifying rejections of simplistic definitions of data science. "Look to the past not for aspiration, but for inspiration." — Host: Closing advice summarizing the episode’s message about learning from history while building the future.

Implications: Listeners should expect data science and AI to keep expanding, with more specialized roles, stronger ethics requirements, and broader business adoption. The field is maturing, not fading, so continuous learning and adaptability will matter most.