E323: How Billionaires Build Their Portfolios

Episode Summary

Executive Summary: Jonathan of Defiant Capital argues that family offices need advice built around family needs, not product sales. He explains why independent, tax-aware, liquidity-first portfolio construction matters most for first-gen wealth, generational transitions, and families entering liquidity events. He also highlights lower middle market PE, operational value creation, and practical AI use in diligence and reporting.

Main Topics: Why family offices need independent advice (Priority: 5/5): Jonathan says large institutions often mix advice with product manufacturing, creating conflicts that can lead to suboptimal recommendations for families. First-gen vs. inherited wealth behavior (Priority: 5/5): He contrasts operator-founders with Gen 2 heirs, noting different risk tolerances, motivations, and approaches to preservation versus growth. Pre-liquidity estate planning and structure (Priority: 5/5): He stresses proactive planning around trusts, FLPs, QSBS, and gifting shares before an exit to capture tax and control advantages. Portfolio construction for large taxable family offices (Priority: 5/5): He argues that once families cross roughly $50 million, governance, asset location, liquidity, and alternative-heavy portfolio design become central. Tax-aware investing and liquidity management (Priority: 5/5): Jonathan says taxable investors care about net-of-tax returns, K-1 efficiency, drawdowns, and liquidity matching far more than traditional institutional allocators. Why lower middle market PE is attractive (Priority: 4/5): He favors lower middle market manufacturing and industrial businesses due to inefficiency, operational improvement potential, and favorable exit dynamics. AI as an operational tool, not an investment decision-maker (Priority: 3/5): He uses AI for research, data extraction, and client communications, but not for judgment-based investment selection.

Key Arguments: Sell-side firms are consensus-driven and product-conflicted, so they often optimize around what their factory produces instead of what a family truly needs. Independent advisors can evaluate the whole market and recommend the right product without being incentivized by internal product sales or fee waivers. First-generation wealth tends to be growth-oriented and operationally engaged, while second-generation wealth is more preservation-focused and sometimes less disciplined about risk. Families should begin estate and liquidity planning years before an exit because low valuations, QSBS, trusts, and gifting can create major tax savings. Above roughly $50 million, wealth management shifts from scrappy investing to governance, documentation, trust/estate coordination, and institutionalized processes. For billion-dollar family offices, alternative investments typically dominate the portfolio, but the mix must be diversified and aligned with income needs and cash flow. Taxable family offices should prioritize net-of-tax, net-of-fee, absolute-value portfolio construction rather than siloed allocation labels. Liquidity management is foundational; overcommitting to drawdown funds can force families to borrow against operating businesses to meet capital calls. Families are increasingly skeptical of long-duration venture funds unless the liquidity premium and expected return clearly justify the lockup. Lower middle market PE is attractive because mega-fund scale has left a large, inefficient segment under-served, especially in manufacturing and industrials. Independent sponsors can be good access points, but families should demand meaningful personal skin in the game and long-term conviction. AI is best used to accelerate data processing, memo extraction, and communication workflows, not to replace human judgment in private markets.

Data Points: Client wealth scale at Defiant Capital: a couple of billion dollars - Jonathan describes the families his firm works with. Liquidity threshold for generational wealth: more than $50 million - He says family office priorities change materially above this level. Typical large family office alt allocation: well over 50% - Jonathan says alternatives usually dominate a billion-dollar family office portfolio. Private investment concentration risk: 30 different private equity funds / 22 different managers - He warns smaller families against over-diversification into too many managers. Capital call stress example: 20% to 30% - Some families redeploy this share of liquidity back into another business after a sale. Illustrative company valuation: $100 million vs. $50 million paper value - Used to explain gifting at a lower 409A valuation. Potential estate tax exclusion example: 50% of estate tax - Transcript frames the benefit of gifting under low valuation conditions. Venture lockup horizon: 10 to 15 years - Jonathan says families scrutinize long-dated venture commitments heavily. Projected venture outcome example: 3x return over 15 years - Used to question whether long lockups are worth it. Older patriarch example: 72 years old - Used to show how age changes willingness to wait for venture outcomes. Projected early-stage venture return: 8x over 15 years - Example of a fund that may be unattractive to an older family due to horizon mismatch. Mag 7 comparison: 5 years / 10 years - He notes family frustration that large tech stocks have outperformed venture over recent periods. Typical PE check size excluded by mega-funds: $30 million to $40 million equity checks - This creates space for lower middle market firms. Possible fund fee structure: 2% fee on committed capital - Jonathan contrasts fees with the long wait for realized returns.

Pivotal Quotes: "it's only helpful if it helps" — Jonathan: His wife’s guiding principle for building Defiant around family needs first. "the answer is always no if you don't ask" — Jonathan: His career lesson about being proactive and not letting fear prevent opportunities. "I don't invest in funds, we invest in people" — Jonathan: He explains why human judgment and manager quality matter more than AI or paper processes in private markets.

Implications: Family offices are moving toward holistic, tax-aware, liquidity-sensitive portfolios with stronger governance and less tolerance for product conflicts or illiquid capital traps. Managers who can show net-of-tax value, operational edge, and real alignment will win.