[J.P. Consulting Co., Ltd.] Survey of all human capital disclosure cases using ChatGPT! Generative AI x Humans thoroughly reviews human capital disclosure of 4,000 listed companies
*J.P. Consulting Co., Ltd.*
Press release: September 2, 2024
**
All human capital disclosures investigated by ChatGPT! Generative AI x Humans thoroughly reviews human capital disclosure of 4,000 listed companies
*～#3 Human capital disclosure status of Japanese companies revealed by ChatGPT～* Overall summary
–
Approximately 47% of the approximately 4,000 companies disclosed only the minimum amount required, but companies in the Nikkei 225 and prime markets tended to be more proactive in their disclosure. However, even among companies in the Nikkei 225 and prime markets, 27-28% have minimal disclosure, so there is still room for improvement in human capital disclosure. In the future, as the collection of information by generative AI becomes more common, disclosing human capital information that is easy for AI to understand will lead to disclosure that is easy to attract the attention of humans.
Main story
In recent years, the importance of human capital management has increased for companies to achieve sustainable growth and strengthen their competitiveness, and disclosure of human capital information in securities reports has become necessary. Therefore, we used ChatGPT to analyze the disclosure status of human capital management in the securities reports of approximately 4,000 companies, and analyzed model companies and industry trends.
This report introduces the analysis results of approximately 4,000 companies, explains the causes of the difference in scores between AI and humans seen in the first and second reports, and explains how to score securities reports using ChatGPT. I will continue to do so. 1st report: 2nd report:
* Why did you decide to use ChatGPT? *
In today’s information-overloaded society, information collection and processing through generative AI is becoming the new standard. For example, at the annual developer conference held on May 14, 2024, Google announced the introduction of an “AI overview function” that organizes information using generated AI.
In this way, instead of humans directly reading and deciphering documents containing huge amounts of information such as securities reports, it is expected that the trend of utilizing generative AI to summarize and efficiently collect information is expected to accelerate.
It is extremely difficult for humans to fairly and comprehensively review the securities reports of thousands of companies, even if human forces are used. On the other hand, by using generative AI, it becomes possible to homogeneously analyze information from thousands of companies. Against this background, we decided to conduct a
large-scale review using ChatGPT, verify its effectiveness, and take on the challenge of understanding the current state of human capital information disclosure by Japanese companies. .
* Overall results of analysis of approximately 4,000 companies *
As a result of analyzing the securities reports of 3,613 companies published from January 1 to July 31, 2024 using ChatGPT, the overall score distribution was as follows.
– Perfect score (companies that show an intention to proactively disclose information): Approximately 21%
– Scores above average or below perfect (companies that disclose human capital in addition to the minimum necessary disclosure information and exhibit a relatively positive attitude): approximately 32%
– Scores below the average (companies whose disclosures are kept to the minimum necessary or who are undergoing trial and error with disclosure content and do not include sufficient content):
Approximately 47%
In addition, the score distribution when narrowing down to Nikkei 225 was as follows (the average score was 77.03 points of the overall average)
– Perfect score: approximately 34%
– Score is above the overall average but less than the perfect score: Approximately 39%
– Score below the overall average: Approximately 27%
When looking at the overall number of companies, 47% of companies keep the minimum amount of disclosure required, but the percentage of companies listed in the Nikkei 225 decreases. There is a trend in which the percentage of companies that are willing to proactively disclose information has increased to 34%.
In addition, when we looked at the distribution of scores by TSE market, we found the following results.
1. Prime market (1537 companies)
・Perfect score: approximately 32%
・Score is above the overall average but less than the perfect score: Approximately 40%
・Score below the overall average: Approximately 28%
2. Standard market (1376 companies)
・Perfect score: approximately 12%
・Score is above the overall average but less than the perfect score: Approximately 29%
・Score below the overall average: Approximately 59%
3. Growth market (408 companies)
・Perfect score: Approximately 11%
・Score is above the overall average but less than the perfect score: Approximately 26%
・Score below the overall average: Approximately 63%
Compared to growth and standard companies, companies in the prime market tend to have more complete disclosures.
We also investigated the correlation coefficient with the score for capital, market capitalization, number of employees, and PBR (data as of August 14, 2024), and found the following results.
– Correlation with capital: 0.057
– Correlation with market capitalization: 0.12
– Correlation with number of employees: 0.16
– Correlation with PBR: -0.082
The correlation coefficients were all less than 0.2, and not much correlation was observed. There appears to be no difference in commitment or enthusiasm for human capital based on company size. About the causes of differences between AI and humans
During the research process, differences between AI and human ratings were observed. In particular, among the five evaluation criteria, there was a tendency for the correlation to be low for scores related to “specificity” and “linkage with strategy.”
The following two points are thought to be the main reasons for this difference.
1. Differences in how information is interpreted: AI only evaluates explicitly stated information, while humans have the ability to “read between the lines” using context and background knowledge.
2. Unique human bias: Human evaluators tend to be influenced by external factors such as a company’s reputation and past performance.
As information gathering by AI becomes more common in the future, the ability to create sentences that are easy for AI to understand without the need to “read between the lines” is expected to become important. I think it’s safe to say that this doesn’t just mean writing texts for AI, but rather providing clearer, more specific, and more structured information.
In fact, this type of writing leads to information disclosure that is easy to understand not only for AI but also for human stakeholders. As a result, it is expected that communication with investors and business partners will improve, leading to a more appropriate evaluation of corporate value.
*The second report provides a detailed introduction to the
similarities and differences between human and AI evaluations in the Nikkei 225 securities report.
2nd report: How to score using ChatGPT?
This time, we use ChatGPT to score securities reports so that we can quantitatively analyze the review results. Specifically, we specified five evaluation criteria for the command statement (prompt) to ChatGPT, and designed it to be scored on a 100-point scale.
When creating the prompts, we paid particular attention to the following points:
1. Specify specific criteria for scores in the evaluation criteria to minimize fluctuations in evaluations.
2. With reference to ISO30414, 11 aspects of human capital management are incorporated into the prompts to enable evaluation of appropriate human capital disclosure.
3. Prompts include bullet points of common human capital indicators to help distinguish between common and unique indicators.
4. Use markdown notation to structure information and create easy-to-understand prompts.
As a result of these efforts, the content of the prompt became very detailed and comprehensive, and the final word file was about 10 pages long. The use of such detailed prompts is based on the concept of “megaprompts” proposed in recent research. Megaprompts are large-scale prompts containing hundreds to thousands of examples and instructions, and are known to increase the accuracy of AI output.
In addition, in this study, we adopted the ChatGPT-4o mini model to directly process PDF files of securities reports, and ChatGPT I generated the answer using the API. ChatGPT-4o
mini is a model that shows performance that exceeds GPT-4, although its accuracy is slightly inferior to GPT-4. We chose this model because we considered the cost efficiency of using MegaPrompt, and because we needed to process the securities reports of approximately 4,000 companies at high speed.
*Summary*
According to this year’s securities report analysis, companies in the Nikkei 225 and prime markets tended to have higher scores than other companies when evaluated by ChatGPT. However, there is still room for improvement in human capital disclosure, as 27-28% of these companies score below average and disclosure remains minimal. Read between the lines, such as by describing human capital initiatives more
specifically and describing the linkage with business strategy in a way that clearly shows the connection between business strategy and human capital KPIs, rather than abstractly. We need to find ways to prevent this from happening.
Additionally, a negative correlation was found between the human capital disclosure score and PBR (price/book value ratio). At first glance, this result seems to contradict the Yanagi model, which shows a positive correlation between non-financial capital and PBR.
However, according to research by Ryohei Yanagi and Shuhei Sugimori, an empirical analysis based on data from the past 20 years shows that it takes 6 to 11 years for the effects of personnel investment to appear, and that for TOPIX 500 companies, it takes only a short period of time. (0 to 3 years) is known to have a negative correlation.
Given that human capital disclosure mandates are relatively new, the results of the current analysis likely reflect short-term effects. In the future, by continuing scoring using generative AI over the long term, it is expected that the essential relationship between scores and PBR will become clearer.
In this survey of approximately 4,000 companies, we introduced the overall score distribution, but we plan to release a detailed discussion of the characteristics and differences in scores by industry in the future.
Author information
Shinji Yamaoka (Senior Consultant, J.P. Consulting Co., Ltd.) After working in business management at an SNS marketing company, he currently works as a human resources consultant at J.P. Consulting Co., Ltd., where he is engaged in human resources system design and core system construction. Recently, the company has also been focusing on supporting operational efficiency through generative AI.
*Contact information*
J.P. Consulting Co., Ltd.
URL: https://www.jp-cons.com/
Contact: https://www.jp-cons.com/contact