Evaluating ChatGPT for structured information extraction from medical notes

In a latest research printed in npj Digital Drugs, researchers evaluated ChatGPT’s means to extract structured information from unstructured medical notes.

Research: A essential evaluation of utilizing ChatGPT for extracting structured information from medical notes. Picture Credit score: TippaPatt / Shutterstock.com

AI in drugs

Giant-language-based fashions (LLMs), together with Generative Pre-trained Transformer (GPT) synthetic intelligence (AI) fashions like ChatGPT, are utilized in healthcare to enhance patient-clinician communication.

Conventional pure language processing (NLP) approaches like deep studying require problem-specific annotations and mannequin coaching. Nevertheless, the dearth of human-annotated information, mixed with the bills related to these fashions, makes constructing these algorithms tough.

Thus, LLMs like ChatGPT present a viable various by counting on logical reasoning and information to assist language processing.

Concerning the research

Within the current research, researchers create an LLM-based technique for extracting structured information from medical notes and subsequently changing unstructured textual content into structured and analyzable information. To this finish, the ChatGPT 3.50-turbo mannequin was used, as it’s related to particular Synthetic Basic Intelligence (AGI) capabilities.

An summary of the method and framework of utilizing ChatGPT for structured information extraction from pathology studies. a Illustration of the usage of OpenAI API for batch queries of ChatGPT service, utilized to a considerable quantity of medical notes — pathology studies in our research. b A normal framework for integrating ChatGPT into real-world functions.

A complete of 1,026 lung tumor pathology studies and 191 pediatric osteosarcoma studies from the Most cancers Digital Slide Archive (CDSA), which served because the coaching set, in addition to the Most cancers Genome Atlas (TCGA), which served because the testing set, had been remodeled to textual content utilizing R program. Textual content information was subsequently analyzed utilizing the OpenAI API, which extracted structured information based mostly on particular prompts.

ChatGPT API was used to carry out batch queries, adopted by immediate engineering to name the GPT service. Put up-processing concerned parsing and cleansing GPT output, evaluating GPT outcomes towards reference information, and acquiring suggestions from area consultants. These processes aimed to extract data on TNM staging and histology kind as structured attributes from unstructured pathology studies. Duties assigned to ChatGPT included estimating focused attributes, evaluating certainty ranges, figuring out key proof, and producing a abstract.

From the 99 studies acquired from the CDSA database, 21 had been excluded because of low scanning high quality, near-empty information content material, or lacking studies. This led to a complete of 78 real pathology studies used to coach the prompts. To evaluate mannequin efficiency, 1,024 pathology studies had been obtained from cBioPortal, 97 of which had been eradicated because of overlapping with coaching information.

ChatGPT was directed to make the most of the seventh version of the American Joint Committee on Most cancers (AJCC) Most cancers Staging Guide for reference. Information analyzed included major tumor (pT) and lymph node (pN) staging, histological kind, and tumor stage. The efficiency of ChatGPT was in comparison with that of a key phrase search algorithm and deep learning-based Named Entity Recognition (NER) strategy.

An in depth error evaluation was carried out to determine the kinds and potential causes for misclassifications. The efficiency of GPT model 3.50-Turbos and GPT-4 had been additionally in contrast.

Research findings

ChatGPT model 3.50 achieved 89% accuracy in extracting pathological classifications from the lung tumor dataset, thus outperforming the key phrase algorithm and NER Categorised, which had accuracies of 0.9, 0.5, and 0.8, respectively. ChatGPT additionally precisely labeled grades and margin standing in osteosarcoma studies, with an accuracy fee of 98.6%.

Mannequin efficiency was affected by the educational immediate design, with most misclassifications because of an absence of particular pathology terminologies and improper TNM staging guideline interpretations. ChatGPT precisely extracted tumor data and used AJCC staging tips to estimate tumor stage; nevertheless, it usually used incorrect guidelines to differentiate pT classes, reminiscent of decoding a most tumor dimension of two centimeters as T2.

Within the osteosarcoma dataset, ChatGPT model 3.50 exactly labeled margin standing and grades with an accuracy of 100% and 98.6%, respectively. ChatGPT-3.50 additionally carried out constantly over time in pediatric osteosarcoma datasets; nevertheless, it often misclassified pT, pN, histological kind, and tumor stage.

Tumor stage classification efficiency was assessed utilizing 744 cases with correct studies and reference information, 22 of which had been because of error propagation, whereas 34 had been because of improper rules. Assessing the classification efficiency of histological analysis utilizing 762 cases confirmed that 17 instances had been unknown or had no output, thereby yielding a protection fee of 0.96.

The preliminary mannequin analysis and prompt-response evaluate recognized uncommon cases, reminiscent of clean, improperly scanned, or lacking report types, which ChatGPT didn’t detect typically. GPT-4-turbo outperformed the earlier mannequin in virtually each class, thereby bettering this mannequin’s efficiency by over 5%.

Conclusions

ChatGPT seems to be able to dealing with large medical be aware volumes to extract structured information with out requiring appreciable task-based human annotation or mannequin information coaching. Taken collectively, the research findings spotlight the potential of LLMs to transform unstructured-type healthcare data into organized representations, which may finally facilitate analysis and medical choices sooner or later.