Macro factors explaining S&P 500 equity valuation

Is the S&P 500 earnings yield at fair value?

Based on the model fit we can see whether the current 12-month earnings yield on the S&P 500 makes sense given the US economic backdrop. The current 12-month forward earnings yield on the S&P 500 is 5.15% or 19.4 in P/E terms and the model says the earnings yield should be 5.31% or 18.8 in P/E, based on the historical relationship between the earnings yield and the features, which is a small mispricing of 3%. In other words, given the current macro indicators the S&P 500 is correctly priced as you would expect is an efficient market.

What happens if we make a new prediction given that we change the most important features to some new values that are mostly in a worse direction?

• National Association of Home Builders Market Index (from 79 to 70)
• Conference Board Consumer Confidence (from 107.2 to 100)
• S&P CoreLogic Case-Shiller US National Home Price Index YoY (from 19.2 to 10)
• US Avg Hourly Earnings Private NFP YoY (from 6.7 to 5)
• Fed M2 YoY (from 11 to 7)
• US U-6 Unemployment Rate (from 6.9 to 6.5)
• Net % of Domestic Respondents Tightening Standards on loans for Large companies (from -14.5 to 0)
• US 2-year yield (unchanged at 2.5)

Based on the above changes holding the other variables constant, the model would predict an earnings yield of 18.2 or 6% below the current earnings yield.

Overreactions in the equity market

What is the most interesting about modelling the earnings yield on the S&P 500 with macro variables is that the earnings yield is consistently below the model’s earnings yield during the dot-com bubble period and the recent pandemic. The US equity market during these two periods is reflecting something else than the underlying macro indicators. Both periods were driven by excess investor sentiment and optimism leading to an overreaction relative to the economy. While the difference between the actual earnings yield and the model’s earnings yield is persistently negative over longer periods during sentiment-driven periods of optimism, the difference also exhibits short-lived spikes on the upside (the earnings yield being higher than the predicted value by the model).

There are four distinct periods where the equity market overreacts with a too high earnings yield relative to the macro backdrop and those are in late 2002 to early 2003 before the equity market bottoms out for good post the dot-com collapse, 2008-2009 during the financial crisis, late 2011 during the euro crisis, and again during in late 2018 during the Fed’s policy mistake on rates as the economy was deteriorating. In these periods the equity market is often driven by risk reductions and liquidity constraints accelerating selloffs more than what is warranted by economic data.

The important features are constantly changing

The final model is a random forest with its hyperparameters optimized using cross-validation. The model fit is based on the in-sample data using these optimized parameters. The idea is to explain the variance and not make predictions on future values of the earnings yield. In the cross-validation the model error is considerably larger (three times larger) than the errors seen in the in-sample fit (actual earnings yield vs the model’s earnings yield). What we see in the data is that the model’s ability to explain the variance in the earnings yield rapidly deteriorates after the model fit, which is not surprising.