Episode Summary
Executive Summary: Bloomberg’s David Rosenberg explains how Bloomberg GPT was built to specialize a general-purpose LLM for finance. The team trained a 50B-parameter model on a 50/50 mix of general web data and Bloomberg’s FinPile financial corpus, adjusted tokenization for numbers, and spent months debugging training instability. The model performed competitively on general benchmarks and substantially better on finance-specific tasks, while remaining a research prototype rather than a production system.
Main Topics: Why Bloomberg built its own LLM (Priority: 5/5): Bloomberg decided in late 2020 to invest in a GPT-3-style model after seeing the breakthrough potential of large language models and recognizing the need for an in-house system that could handle sensitive financial data. Data strategy: general text plus FinPile (Priority: 5/5): Bloomberg GPT was trained on a roughly even split between standard large-language-model data and Bloomberg-curated financial text (news, filings, press releases, transcripts), aiming to combine broad capability with domain expertise. Training challenges and model engineering (Priority: 5/5): The team closely followed BLOOM/OPT-style architecture but encountered gradient instability, optimizer issues, and mixed-precision quirks, leading to multiple training restarts and fixes such as better weight handling and added normalization. Financial-aware tokenization and numeracy (Priority: 4/5): Because finance text is numerically dense, Bloomberg changed tokenization to represent numbers digit-by-digit and expanded vocabulary size, hoping to improve the model’s handling of financial quantities. Evaluation and benchmark performance (Priority: 5/5): The model was assessed on standard benchmarks like MMLU and BBH, plus finance-specific tasks such as FinQA-like reasoning, sentiment, named entity disambiguation, and natural-language-to-BQL translation. Practical deployment and future direction (Priority: 4/5): Bloomberg GPT is still a research model, not yet in production; the team is exploring smaller models, instruction tuning, better data mixtures, and safer internal use cases before any broader deployment.
Key Arguments: Building from scratch made sense because there were few commercially usable open-source models at the needed scale when the project began, and Bloomberg needed to keep data in-house. A 50/50 blend of general and finance-specific data allowed the model to remain broadly capable while materially improving performance on financial tasks. Handling numbers carefully matters in finance, so digit-level tokenization was a deliberate adaptation to the domain. Training huge models remains unstable and empirical; many issues were resolved by following published logs from OPT and BLOOM and by debugging layer norm, optimizer, and precision details. Bloomberg GPT was competitive with comparable open models on general tasks and notably stronger on finance benchmarks, especially numeric reasoning. Future progress likely depends less on raw scale alone and more on disciplined experiments with data quality, tokenization, instruction tuning, and smaller, easier-to-serve models. The model’s main value today is research and experimentation, not production, because hallucinations and reliability remain unresolved.
Data Points: Model size: 50 billion parameters - Bloomberg GPT size, compared with cited open models like OPT-66B. Training data mix: ~50% general data / ~50% Bloomberg FinPile - Core training split used to balance general ability and finance specialization. FinPile start year: 2007 - Bloomberg’s curated financial corpus spans data collected over many years starting in 2007. Training duration: 42 days - Version 2 training ran for 42 days before leveling off. Additional training after leveling off: ~1 week - After reaching diminishing returns, they continued briefly before stopping. GPU count: 512 NVIDIA A100 GPUs - Hardware used for training on Amazon SageMaker. GPU memory: 40 GB per GPU - A100 configuration used in the training cluster. Estimated compute: ~1.3 million GPU hours - Approximate amount of pre-bought compute capacity mentioned for training. Context length: 2048 tokens - Model context window size. Vocabulary size: ~150,000 tokens - Larger than the typical ~50,000-token English-only vocabulary. Team size: ~9 people - Total team involved in building the model. Core implementation staff: ~4 people - People doing hands-on coding and implementation. Gradient/numerical issue timing: ~8 to 10 days into training - When gradient norm spikes and validation degradation appeared in early runs. FinPile scale: 350 billion tokens - Approximate size of the finance-specific dataset discussed later in the interview.
Pivotal Quotes: "When GPT-3 came out, it was about three years ago. That was a moment of, wow, this is different." — David Rosenberg: Explaining why Bloomberg decided to pursue its own large language model. "We call FinPile, all the documents are timestamped." — David Rosenberg: Describing the Bloomberg-specific financial training corpus and why they considered time-based curriculum learning. "We wanted to control our risk by making the model as close as we could to something that we knew worked before." — David Rosenberg: Explaining the decision to stay close to BLOOM architecture during training.
Implications: The episode shows that domain LLMs can outperform general models on specialized tasks, but success depends on data quality, numeracy, and careful training engineering. It also suggests that smaller, well-trained, in-house models may be more practical and safer for regulated industries than relying entirely on external APIs.