Goldman Sachs CIO on How the Bank Is Actually Using AI

Episode Summary

Executive Summary: The episode centers on how Goldman Sachs is deploying generative AI safely and practically inside a regulated financial institution. Marco Argenti explains Goldman’s platform approach, governance, open-source strategy, and emphasis on retrieval, embeddings, and human oversight. The conversation also covers how AI is boosting developer productivity, changing work across knowledge jobs, and potentially reshaping chip demand and cloud infrastructure.

Main Topics: Goldman Sachs’ AI strategy and platform approach (Priority: 5/5): Argenti describes building a controlled internal AI platform (GSAI) rather than relying on raw off-the-shelf chatbots. The goal is to make existing models usable, safe, and interchangeable across applications. Governance, risk, and regulation in enterprise AI (Priority: 5/5): Because Goldman is a regulated financial institution, AI use cases go through business and risk committees, model-risk controls, entitlements, and human-in-the-loop review to reduce hallucination and data-security risks. Practical AI use cases inside the bank (Priority: 5/5): Goldman is using generative AI for banker assistants, document processing, entity extraction, public filing analysis, and content generation, especially where unstructured data and repetitive workflows create bottlenecks. Developer productivity and changing software work (Priority: 4/5): The bank has deployed AI coding tools widely across developers and sees meaningful productivity gains. Argenti argues AI shifts developers from low-level coding toward business understanding, documentation, and outcome-driven work. Open source versus proprietary models (Priority: 4/5): Argenti says Goldman prefers not to build what it doesn’t need to, favors open source where appropriate, and sees strong open-source models like Llama narrowing the gap with proprietary systems while reducing vendor lock-in. Compute, GPUs, inference, and hardware evolution (Priority: 4/5): The discussion explores where AI compute demand is headed: GPUs remain dominant for training, but inference may open the door to specialized accelerators. Cloud-hosted and self-hosted models may shape future hardware demand. How AI changes work across the firm (Priority: 3/5): Beyond developers, AI is expected to streamline pitch books, summaries, research synthesis, and other content-heavy tasks, potentially eliminating toil while expanding the scope of work employees can do.

Key Arguments: Enterprise AI cannot be treated like consumer chatbots because even small error rates are unacceptable in banking. Goldman’s best approach is to build a secure platform around existing models rather than train everything from scratch. Retrieval augmented generation, embeddings, and citations materially improve answer accuracy and reliability. Human-in-the-loop oversight remains necessary because AI is probabilistic and can hallucinate. Regulation is not just a constraint; it helps force discipline and governance around AI deployment. Open-source models are becoming strong enough that banks can use them selectively to reduce vendor lock-in. AI is delivering tangible productivity gains, especially in software development and document-heavy workflows. As AI handles repetitive tasks, developers will need stronger business context and clearer communication skills. The long-term chip picture may split: GPUs for training, specialized hardware for inference. Prompt quality matters; careful, empathetic prompting can improve outputs and even help elicit “I don’t know” when appropriate.

Data Points: Length of Stock Movers promos: Five minutes or less - Repeated promotional description of Bloomberg’s short audio news format. Goldman developers using AI tools: 12,000 - Argenti says virtually every Goldman developer is equipped with generative AI coding tools, except some using the proprietary Slang language. Typical productivity uplift in development: 10% to 40% - Range of productivity increases seen from AI tools across development workflows. Average productivity uplift in development: 20% - Argenti’s estimate of the current average increase from AI tools for developers. Net productivity gain after accounting for non-coding time: 10% - Because developers spend only about half their time coding, the effective overall gain is lower than raw coding productivity gains. Model context growth: Thousands to tens of thousands to millions - Argenti describes rapid expansion in context window sizes as models evolve. Llama 3.1 model size: 405 billion parameters - Example of a large open-source model Goldman finds compelling. Career span in the U.S.: 40 years - Mentioned in a sponsor ad for real estate investing. Potential real estate investing timeline: 15 years - Sponsor ad claims real estate investing can shorten the path to retirement. Bloomberg journalism footprint: 3,000 journalists and analysts - Mentioned in promotional segments for Bloomberg audio products.

Pivotal Quotes: "If you look at the consumption of resources today, those who consume more resources are people that actually do the training of their own models." — Marco Argenti: Explaining why Goldman favors fine-tuning and RAG over training models from scratch. "Our rule is that there always needs to be a human in the loop." — Marco Argenti: Describing Goldman’s approach to AI oversight and risk management. "You need to take the AI literally by the hand and take it where you want to go." — Marco Argenti: On what makes a good prompt and why prompting style matters.

Implications: Enterprise AI adoption will likely be shaped by governance, security, and workflow design more than flashy demos. Banks and other regulated firms may rely on secure platforms, open-source models, and human oversight to turn AI into measurable productivity gains.