Reduce Taxes with Private Investments?

Lately I’ve been studying family offices: how they’re structured, how they invest, how they support the family’s spending, how they minimize taxes, and so on. I’ve studied the endowment model and based my portfolio on it (heavy in real estate and other private assets, in addition to public equities). Now I’m looking to improve my understanding about how to handle withdrawals and the so-called “decumulation phase”. I don’t even like the name of this phase – I always hope to keep growing! 😅

One of the things that I see repeated regularly in this space is the idea that you’re going to be paying significant income taxes in retirement**. That’s why you should pursue private investments with pass-through depreciation (to shield cash flow), return of capital (non-taxable), and similar characteristics.

I’m going to bust the idea that early retirees pay significant taxes in retirement.

For example, I was just watching this YouTube video by “Managing Tech Millions” – his content is interesting, but he mis-estimates how much most retirees would pay in taxes. Note: both he and I are coming at this problem in the context of early retirement in your 30s/40s/50s with taxable portfolios (without retirement account withdrawals, social security, pensions, or similar retirement income).

In this video, the creator uses an example of pulling $200k from a taxable portfolio, and claimed most people would actually receive $140k to $160k in their bank account. He incorrectly asserts that most people are paying ordinary income tax rates on portfolio withdrawals. (Some of it might be ordinary income from bonds or money market funds or REITs or not-yet-qualified dividend stocks, sure.) But in drawdown-style portfolios withdrawals you should be paying long-term capital gains rates on most of it.

A 5.2% tax rate isn’t bad.

However, even this isn’t the true story. When you withdraw $200k from your portfolio, not all $200k of that is gains. You had to actually contribute money to buy the assets in the first place. It’s not uncommon for 40% to 60% of retirees’ portfolio to be the original cost basis that you paid to acquire the assets.

(For reference, one of my brokerage accounts is at 68% cost basis and another is at 72%. However I’m fairly young so my stock hasn’t had decades to compound, and the second account has been tax-loss harvested to increase basis.)

Let’s see what happens if we account for this – assuming a 50% cost basis.

Yes, you can withdraw $200k and pay $0 — zilch, nada — in federal taxes.

As you can see, with a bit more tax planning, you can sell high-basis stock and/or do tax-loss harvesting to minimize this last bit of taxes too.

Astute readers will note that I’m ignoring state taxes here. This is both because rates differ dramatically per state, and because I personally live in a tax-free state. It is important to note that many states do tax LTCG at ordinary income rates. A rate around 3% to 6% is pretty typical. As of publishing, Oregon has the highest rate of 8.75% at this level of taxable income ($68k); even California is only 4% at this level.

Next, I’d expected to highlight who this author’s strategies are actually for: high spenders with large, low-basis portfolios. Unfortunately, I think I disproved this idea… so I’m not sure who its for. Alas, I’ve long-since adopted this portfolio myself.

Let’s take an example where you want to withdraw $600k annually for spending and have a low cost basis of 20% from a tech windfall, since its the YouTube author’s audience managingtechmillions.com.

Now their effective tax rate has jumped to 10.2%. Still, 10.2% ain’t bad … and they’re living large on $600k/year. (Note: this would require a $15M portfolio to be sustainable with a 4% safe withdraw rate.)

Realistically, many people who receive a windfall would buy a house and switch from the standard deduction to itemized deductions. Then they can deduct their mortgage interest (subject to caps for jumbo mortgages), plus state and local taxes (SALT, larger caps for 2025-2030) from their federal taxable income. So their taxes would be even lower than this.

That’s the power of living on long-term capital gains rather than ordinary income.

Does this defeat the purpose of using the strategies mentioned in the YouTube video?

Ironically, I started writing this post because I wanted to show when constructing your portfolio like this makes sense and when it doesn’t (largely because I’ve built my portfolio like this over the past decade). But as I’ve been learning more about withdrawal strategies and decumulation (e.g., from Ari Taublieb), I’m second guessing whether the extra complexity is worth it. Contrary to the claims, it doesn’t appear to be a significant tax optimization (saving 0.4% of net worth) and it comes with significant real costs (transactions in real estate; hefty fees for private investments; entity/legal/accounting costs). With all this, why not just “VTSAX and chill“?

These are the main reasons that I can think of –

1. Behavioral – does receiving income feels better than selling shares?
   
   Behavioral finance is interesting. Mathematically, it makes sense to pay down loans with the largest interest rate first; psychologically, it feels better to pay down loans with the smallest balance first (the so-called “debt snowfall”), which is what I personally did with my student loans (even though I understood the math).
   
   Similarly, I think it feels better to receive income (dividends, rents, royalties, interest) than to sell shares, especially for folks who were aggressive savers and investors during the accumulation phase; mathematically, I’m starting to wonder whether pursuing a total return portfolio and selling shares is actually a more optimal approach (and significantly less complex to build and manage).
   
2. Optimizers – are you an optimizer or satisficer? (Is good enough good enough for you?)
   
   There are so many things to optimize in the wealth game. Folks who achieve a certain level of wealth may not find it easy to sit back and relax.
   
   The above example shows a 10% tax rate, but that’s still $60k+ annually in taxes; maybe they want to drive that to $0 while maintaining a high spending rate for their lifestyle.
   
   Or maybe they think that they can “beat the market” (on a risk-adjusted basis) through investing in private assets (leverage, illiquidity premiums, etc) to grow their income and/or net worth.
   
3. Risk/Loss Aversion – are you uncomfortable with volatility / want to reduce your downside?
   
   VTSAX and chill comes with a lot of volatility. Even with a healthy bond allocation, the correlations with stocks often “go to 1” at the worst possible moment (hello, 2022!) so you can’t really cap the drawdown risk in your portfolio.
   
   Conversely, private assets often aren’t marked-to-market so may be psychologically easier. Plus, they’re often tangible which provides a lot of comfort (e.g., “real estate can’t go to zero”). For many people who have a strong desire for your control, you can even buy and operate your own real estate to further feel like you’re reduce risk. It’s hard to say how many of these benefits are real or perceived; hello again, behavioral finance!
   
   Maybe, like me, you don’t want to lose out on sweet tax-subsidized healthcare benefits. So why not start a “family office” (or other company) that employs you so that it can pay for your healthcare and other pre-tax benefits? Or does the idea of running your family like a business appeal to you?
   
4. Meaning – where do you find your sense of purpose?
   
   Many men especially find their sense of purpose in being a good provider for the family, and later that often evolves into being a good steward for family wealth. This is the zeitgeist behind much of the “generational wealth” ideas (and the micro-family offices that the YouTube creator endorses.)

All of these apply to me to varying degrees, which is probably why I went down this path originally. The question that I’m exploring is whether this remains the correct course (for me) to continue on.

For those who have gone through this before, I’d love to hear how you have thought about portfolio construction and lifestyle optimization.

** you will be paying significant taxes if you have a large pre-tax retirement accounts when Required Minimum Distributions (RMDs) start. These RMDs can force you into significantly higher tax brackets, whether or not you even need the income. Ideally, you take advantage of low-income years and/or down markets to perform Roth conversions to reduce future RMDs and their massive tax impact.