ChatGPT can predict analyst forecast changes

About once every three to six months I sit through a presentation on what AI can do or read about new developments in the area. And every time I come away with the same ambivalent feeling. On the one hand, I am amazed and excited to learn what additional capabilities AI tools have developed now. These tools start to make a real difference. On the other hand, I can’t help thinking that if this trend continues, these apps will come for my job and the jobs of equity analysts and fund managers everywhere. A case in point is an analysis of earnings calls by chatGPT that can forecast future earnings revisions and recommendation changes by analysts.

A team from the University of Sydney developed a proprietary tool that analyses analyst questions during earnings calls using chatGPT. The tool categorises the questions asked, and the answers given along four key dimensions:

• Growth potential of the company

• Earnings quality

• Quality of management

• Risks to the company

In each category, they defined five sub-categories and key dimensions along which to evaluate the answers.

As an aside, if you are a young up-and-coming equity analyst or an interested retail investor, you should read through Table I in the note (pages 12-14). It’s an excellent checklist on how to think about a company.

Based on the answers given, the researchers asked chatGPT to score each category and sub-category whether it is positive news for the company or negative. The average score across all sub-categories and categories was then calculated.

The resulting Analyst Insight Score (AIS) turns out to be a powerful predictor of analyst actions and share prices in the future. For example, it can predict future recommendation changes by analysts, target price changes, or earnings revisions better than traditional metrics like earnings surprise. It is about two to four times more influential in determining the size of the analyst revisions than these traditional metrics. This shows that with the help of chatGPT, investors can capture more nuance around an earnings release and capture qualitative information that is relevant to investment decisions in a systematic way.

But as we know, analyst revisions can drive share prices, at least in the short term, so if the AIS can forecast future actions by analysts, it should also be able to forecast future share prices. To test this, the researcher created portfolios based on the AIS score of the 200 largest stocks in the US. Every month, they sorted the 200 stocks into five buckets of 40 stocks based on the AIS score in the previous month. Then they repeated the exercise every month between 2015 and 2024.

As always, the big caveat here is that the results are shown without transaction costs and slippage taken into account, and it is unclear how much turnover these portfolios have. But what the tests show is that the qualitative information captured by chatGPT does help to create performance  

Equity performance and AIS

Source: Ming et al. (2024)

Now, the good news about this is that the analysis relies on real analysts asking questions of company executives. But once chatGPT learns how to ask insightful questions, it really is coming for our jobs.