I didn’t know that did you?

Imagine you invest in a bond that is barely investment grade with a credit rating of BBB-. Then, a year later, the credit rating agency comes to you and tells you “Remember the bond we told you last year was investment grade? Turns out it was junk-rated at BB+ already back then.” How would you feel about that? If you’d be mad at them, then you should be mad at some ESG rating agencies as well.

Florian Berg and his colleagues examined the ratings of Refinitiv ESG. They downloaded the E, S, and G, as well as the ESG scores for all the companies in the Refinitiv database in September 2018, September 2020, February 2021, and March 2021. Then they used the data to look for companies that were included in all the datasets and compared the ESG ratings for previous years.

Refinitiv rates companies on a relative scale (like many ESG rating providers do). Hence, companies with the best ESG credentials end up in the top decile or top quartile of the rating scale not because they are great companies on their own, but because they are considered in the top 10% or top 25% of all companies in the world. Fair enough, but in April 2020, Refinitiv announced a change in methodology, where they introduced new metrics and a different weighting scheme that emphasises the differences in materiality for different industries.

But instead of introducing this new methodology in April and using it from then on, they decided to change the ratings of all companies in their database going back to the very first thing ever issued. This was presumably done to ensure that ratings for any given company remain comparable over time. But what it did instead was move companies’ ESG ratings around a lot. The chart below shows that of the companies in the top quartile by ESG ratings typically only 70-80% remained top quartile after the methodology change. The change was particularly harsh in the governance ratings. Less than half the companies that were previously considered top quartile in terms of governance, remained top quartile in governance after the methodology change.

Share of companies that remained top or bottom quartile after methodology change

Source: Berg et al. (2021)

That’s bad, but I might be willing to forgive Refinitiv for some poor choices when changing their methodology. What I am not willing to forgive them for is that even when they don’t change methodology the ratings of past years can change.

The researchers downloaded the same dataset both in February and March 2021. Between these dates, there were no changes in methodology, yet ESG ratings for the years 2011 (10 years before the download date) to 2017 (4 years before) moved as is shown in the chart below.

Ongoing ratings changes in Refinitiv ESG

Source: Berg et al. (2021)

I think this is simply unconscionable. Ratings should not change from one month to another, particularly not ratings from years before. Yet, when checking the ratings changes, the authors of the research paper found something interesting. The old ratings had no predictive power for future equity returns. Whether a company had a good or bad ESG rating did not matter for share price performance. After the methodology change in 2020, however, companies with better ESG ratings showed better future share price performance. And between February and March 2021, when there was supposedly no change in methodology, the ratings downloaded in March were somewhat more predictive of future share price performance than the ratings downloaded in February.

The authors of the study claim that ESG rating providers might mine the data and change their methodology in subtle ways so as to be able to show a better backtest result for the link between ESG ratings and subsequent share price performance. I cannot comment, but if you ask me, Refinitiv has some explaining to do.