This report analyzes 336 Reddit posts from the r/Psoriasis subreddit to understand the key themes and challenges surrounding diagnostic uncertainty. Using a deterministic LLM-based filter, we isolated posts where users expressed confusion about their diagnosis. We then applied BERTopic, an unsupervised topic modeling algorithm, to identify distinct clusters of discussion. Our analysis reveals that diagnostic uncertainty is most prominent for symptoms affecting the scalp (113 mentions) and nails (44 mentions). Key themes include differentiating psoriasis from eczema, the onset of psoriatic arthritis, concerns about specific body areas (face, hands, feet), and navigating the healthcare system for a definitive diagnosis. Sentiment analysis using VADER showed a distinctly negative emotional tone (average compound score of -0.051), dominated by anxiety. These findings highlight significant opportunities for targeted patient education and improved physician-patient communication to address the specific anxieties and questions that arise during the diagnostic phase of psoriasis.

1. Introduction

Psoriasis is a complex, chronic skin condition that can be challenging to diagnose, especially in its early stages or when it presents in less typical ways. While a lot of research has concentrated on treatment, the patient journey during diagnosis is less widely investigated. To understand what patients truly experience while seeking answers, we assessed the r/Psoriasis Reddit community, where individuals come for honest, vulnerable advice and to share their diagnostic worries and stories. This report aims to systematically analyze these discussions to uncover the primary sources of diagnostic uncertainty.

Reddit is a popular online forum where patients share their real-life struggles, questions, and frustrations, often in more detail and candor than they do in the clinic. By tuning into these conversations, we can:

• Identify common pain points in the diagnostic process

• Spot gaps in patient education and communication

• Understand what patients wish their doctors knew

Most published Reddit psoriasis studies focus on how patients feel about medications, not on the diagnostic journey itself. Maynard et al. tallied sentiment toward biologics in 1,141 posts and reported that 61% were positive about treatment efficacy (https://pubmed.ncbi.nlm.nih.gov/36877885/), while Okon et al. mapped 176k comments but grouped diagnostic confusion into a single minor topic without digging deeper (https://pubmed.ncbi.nlm.nih.gov/29469693/). No peer-reviewed work has yet quantified how often people leave the dermatologist still wondering “Is this really psoriasis, do I need a biopsy, or could it be eczema or PsA?” Our study fills that gap by isolating and analysing only those posts that express diagnostic uncertainty.

Some of the key questions we set out to answer were:

1. Mild-to-Moderate Disease: For many patients with severe disease, they’re well-connected to dermatologic and rheumatologic care. For patients with mild-to-moderate psoriasis, what questions cause the most diagnostic confusion?

2. Biopsy Decisions: What are the uncertainties around the necessity and timing of biopsies?

3. Alternative Etiologies: To what extent do users discuss alternative causes like diet, pH balance, celiac/gluten intolerance, and vitamin D deficiency?

4. Severity & Body Areas: How do disease severity and the specific body area affected (e.g., scalp, nails) influence diagnostic uncertainty?

By analyzing the authentic, unfiltered conversations of patients, we aim to provide clinicians with a deeper understanding of the patient experience, ultimately helping to improve communication and patient care.

2. Methods

2.1. Data Collection and Filtering

We began with a dataset of 3,240 posts scraped from the r/Psoriasis subreddit. To isolate posts relevant to diagnostic uncertainty, we implemented a multi-stage filtering process:

1. Initial keyword sieve: A "less-restrictive" keyword filter was applied to identify a broad set of 1,157 potentially relevant posts. Keywords included “diagnosis”, “biopsy”, etc.

2. Exclude treatment-only posts: We applied a keyword filter to remove posts focused purely on treatment discussions (e.g., specific biologics, dosages), ensuring the remaining posts were more likely related to diagnosis.

3. Deterministic LLM Classification: The core of our method involved using a fine-tuned Large Language Model (gpt-4o-mini) to classify each of the 1,157 posts as dealing with a diagnostic vs treatment question. The model was given a detailed system prompt and few-shot examples to distinguish between "Diagnostic Uncertainty" and "Treatment/Management." This deterministic approach (with temperature=0) ensures full reproducibility, meaning each time the model runs, it’ll give the same answer.

This process yielded a final dataset of 336 posts centered on diagnostic uncertainty.

2.2. Topic Modeling and Analysis

The 336 diagnostic posts were analyzed using the following NLP pipeline:

1. Embeddings: Each post was converted into a numerical vector using the all-MiniLM-L6-v2 sentence-transformer model. Essentially, this represents the meaning of a post as a vector so two post vectors can be compared for similarity in meaning.

2. Clustering: We used BERTopic, an unsupervised clustering algorithm, to group the posts into thematic clusters. Key parameters (min_topic_size=3) were optimized to reveal granular topics.

3. Keyword Extraction: For each cluster, we used class-based TF-IDF (c-TF-IDF) to identify the most representative keywords.

4. Demographic & Thematic Analysis: We ran additional scripts to extract mentions of age, gender, body parts, and specific keywords (e.g., "biopsy," "diet," "gluten") from the 336 posts.

2.3. Sentiment and Emotion Analysis

To quantify the emotional tone of the posts, we performed an additional analysis using VADER (Valence Aware Dictionary and sEntiment Reasoner), a lexicon and rule-based sentiment analysis tool specifically attuned to sentiments expressed in social media that is often used in research.

1. Sentiment Scoring: Each post was analyzed using VADER's SentimentIntensityAnalyzer to produce a compound score (normalized between -1 for most negative and +1 for most positive). This single, comprehensive score captures the overall sentiment intensity and valence of the text.

2. Emotion Word Counting: We developed lexicons of keywords for four core emotions: Anxiety (e.g., worried, scared, stress), Despair (e.g., hopeless, miserable, crying), Frustration (e.g., annoyed, sick of, hate), and Hope (e.g., hopeful, better, progress). We then counted the occurrences of these words in each post to measure the prevalence of each emotional theme.

3. Results

Our analysis of the 336 posts on diagnostic uncertainty revealed several key insights into patient concerns.

3.1. Demographic and Body Area Insights

• Gender: Of 107 posts where gender was identifiable, 90% were from male users.

• Age: The average age of users discussing diagnostic uncertainty was 28 years old.

• Family History: 6% of posts mentioned a family history of psoriasis.

Body Areas Sparking the Most Uncertainty

The analysis of post content revealed that diagnostic confusion is most frequently associated with specific, hard-to-diagnose areas:

• Scalp: 113 mentions.

• Hands & Feet: 107 mentions combined.

• Nails: 44 mentions.

• Face & Ears: ~30 mentions combined.

3.2. Etiology and Biopsy Discussions

• Alternative Etiologies:

  • Diet & Gluten: 17 posts mentioned diet, celiac disease, or gluten as a potential cause or trigger.

  • Vitamin D: 5 posts discussed Vitamin D deficiency.

• Biopsy Discussions: "Biopsy" was mentioned in 21 posts, often in the context of seeking a definitive answer when a visual diagnosis was inconclusive.

3.3. Topic Cluster Deep Dive

The BERTopic analysis identified 29 distinct clusters. Below is a summary of the most prevalent and clinically relevant topics.

3.4. Sentiment and Emotional Tone

The analysis of the language used in the 336 posts reveals a predominantly negative and anxious emotional state.

• Overall Sentiment: The average VADER compound score across all posts was -0.051. This score, on a scale from -1 (most negative) to +1 (most positive), indicates a clear, albeit mild, overall negative sentiment.

• Prevalence of Core Emotions:

  • Anxiety: 117 total mentions

  • Hope: 89 total mentions

  • Frustration: 37 total mentions

  • Despair: 28 total mentions

Anxiety was the most frequently expressed emotion. Despite the negative context, hope was the second most common emotional theme, suggesting a resilient search for answers.

4. Discussion

By synthesizing the cluster themes and sentiment data, we can identify critical knowledge gaps and understand the emotional landscape of waiting for a diagnosis.

4.1. The Emotional Landscape of Diagnostic Uncertainty

The sentiment data confirms a journey marked by significant emotional distress. The dominant emotion is anxiety (117 mentions), driven by new, unexplained symptoms and the fear of a worsening condition (e.g., onset of PsA). This is followed by frustration (37 mentions) at the healthcare process and despair (28 mentions) over the impact on quality of life. Despite this, hope remains a powerful force (89 mentions), often linked to community support and the prospect of a clear diagnosis.

4.2. Interpreting Keyword Frequency vs. Cluster Size

A key finding is that a topic can be frequently mentioned but not form a single large cluster. For example, the scalp was mentioned 113 times, yet the dedicated scalp clusters (e.g., Topic 5) are relatively small. This is because topic modeling groups posts by their primary theme. A post mentioning the scalp in the context of new joint pain will be clustered with "psoriatic arthritis" posts, not "scalp" posts. This demonstrates that scalp-related uncertainty is a pervasive issue woven into many different types of patient concerns, from initial symptoms to fears of disease progression.

4.3. Key Knowledge Gaps and Patient Needs

Gap 1: Differentiating Psoriasis from "Look-Alike" Conditions

• Evidence: The largest clusters focus on telling psoriasis apart from eczema (Cluster 1), fungal infections (Cluster 0), and seborrheic dermatitis (Cluster 5).

• Patient Need: Clear, visually-rich educational materials that explain the key differences.

Gap 2: Understanding the Psoriasis-Psoriatic Arthritis Link

• Evidence: The fear of psoriatic arthritis (PsA) is a significant source of anxiety (Cluster 2).

• Patient Need: Proactive education for all psoriasis patients about the early signs of PsA.

Gap 3: Navigating the Healthcare System for a Diagnosis

• Evidence: Multiple clusters highlight struggles with long wait times, conflicting diagnoses, and confusion around biopsies.

• Patient Need: Greater transparency about the diagnostic process and encouragement to seek second opinions.

Gap 4: The Mind-Body Connection and Triggers

• Evidence: Patients actively explore connections to strep throat (Cluster 15), celiac disease/diet, and mental stress (Cluster 25).

• Patient Need: Balanced, evidence-based discussion about known triggers and associations to validate patient experiences.

5. Conclusions

The diagnostic phase of psoriasis is an emotionally charged journey defined by anxiety and a search for clarity.

Key Clinical Takeaways:

1. Acknowledge Location-Specific Uncertainty: When a patient presents with symptoms on the scalp or nails, clinicians should proactively address the high likelihood of confusion with seborrheic dermatitis or fungus.

2. Educate on Psoriatic Arthritis: Proactively discussing the early signs of joint pain can empower patients to seek help sooner.

3. Discuss Diet Sensibly: Being prepared to discuss the evidence regarding diet, gluten, and psoriasis can build trust.

4. Validate the "Eczema vs. Psoriasis" Dilemma: Acknowledge that these conditions can look similar and clearly explain the clinical reasoning behind a diagnosis.

In the field of dermatology, artificial intelligence (AI) is becoming an essential tool for clinical decision-making and patient counseling. This project focuses on evaluating two leading AI models, OpenAI's GPT-4 (ChatGPT) and OpenEvidence, based on their ability to respond to realistic, physician-style prompts for melanoma counseling. Our emphasis is on three clinically relevant scenarios that reflect how a dermatologist would actually counsel a patient, highlighting the importance of realistic prompts in assessing AI's clinical utility.

Methods

Prompt Design

We selected three realistic clinical scenarios for melanoma counseling, each written to reflect realistic use of an LLM:

• melanoma_in_situ: Melanoma in situ, I need to counsel the patient on next steps. What key points should I cover?

• melanoma_0_9_ulcer: Melanoma, 0.9 mm, with ulceration, I need to counsel the patient on next steps. What key points should I cover?

• melanoma_1_9_ulcer: Melanoma, 1.9 mm, with ulceration, I need to counsel the patient on next steps. What key points should I cover?

Each prompt was given to both GPT-4 and OpenEvidence. For each scenario, multiple runs were collected to assess consistency.

Example Model Responses

melanoma_in_situ

• ChatGPT: "Melanoma in situ means the cancer is only in the very top layer of your skin. We usually treat this with a minor surgery to remove the spot and a small area around it. The chance of it coming back or spreading is extremely low. You don't need any further tests. It's a good idea to keep an eye on your skin and use sun protection. If you have questions or feel anxious, let us know."

melanoma_0_9_ulcer

• OpenEvidence: "For a 0.9 mm melanoma with ulceration, wide local excision is recommended. Sentinel lymph node biopsy may be considered because ulceration increases the risk of spread, though the overall risk is still not high. Prognosis is generally good, but ulceration is a negative factor. Ongoing follow-up and sun protection are important. Support is available if you need it."

Evaluation Metrics

Model responses were evaluated using:

• Similarity metrics: SBERT (semantic similarity), ROUGE (text overlap), BLEU (n-gram overlap) compared to a “gold standard” response written by project author CB.

  1. Example “gold standard” answer for melanoma in situ: You have been diagnosed with melanoma in situ, which means the cancer cells are only in the top layer of your skin and have not spread deeper. The next step is a minor surgery called wide local excision to remove the melanoma and a small margin of normal skin around it. The prognosis is excellent, and the risk of the melanoma coming back or spreading is extremely low. You do not need any further tests or lymph node biopsies. It's important to protect your skin from the sun and to check your skin regularly for any new or changing spots. We are here to support you and answer any questions you may have.

  2. Example “gold standard” answer for 0.9mm melanoma with ulceration: You have a melanoma that is 0.9 mm thick with ulceration, which means it has grown into the deeper layers of your skin. The next step is surgery to remove the melanoma with a margin of normal skin around it. We also recommend a sentinel lymph node biopsy to check if any cancer cells have spread to nearby lymph nodes. The prognosis is generally good, but ulceration increases the risk slightly. You'll need regular follow-up visits to monitor for any recurrence. We're here to support you through this process and answer any questions.

  3. Example “gold standard” answer for 1.9mm melanoma with ulceration: You have a melanoma that is 1.9 mm thick with ulceration, which means it has grown deeper into your skin. The next step is surgery to remove the melanoma with a wider margin of normal skin. We strongly recommend a sentinel lymph node biopsy to check for spread to nearby lymph nodes. The prognosis is more guarded due to the thickness and ulceration. You'll need more frequent follow-up visits and may be eligible for additional treatments if the lymph nodes are involved. We're here to support you and will work together to ensure the best possible outcome.

• Physician-style grading: An LLM grader scored each response on clinical accuracy, relevance, depth, clarity, completeness, and correct staging, workup, and treatment recommendations (0-2 points each, max 16)

What is the LLM Grader?

The LLM grader is an automated evaluation tool that uses a large language model to assess each response for clinical accuracy, relevance, depth, clarity, completeness, and correct staging, workup, and treatment recommendations. Inspired by the grading criterion of Nihal et al’s paper comparing ChatGPT and OpenEvidence for SCC and BCC, each category is scored from 0 (poor) to 2 (excellent), for a maximum total of 16. (https://pubmed.ncbi.nlm.nih.gov/40471861/) This provides a rapid, consistent, and reproducible way to benchmark model performance across many runs and scenarios.

While the LLM grader is a powerful tool for large-scale benchmarking, it is not a substitute for human expert review. We will supplement these results with human grading to ensure clinical validity and to capture nuances that automated systems may miss.

Data and Analysis

Only responses to the three realistic prompts were included in the main analysis. All grading and similarity calculations were performed using standardized scripts. Detailed grading for every run is provided in the Supplemental section.

Results (NOTE: using mocked LLM responses to get code working, none are real results)

Model Comparison

Key Findings

• GPT-4 outperformed OpenEvidence on all similarity metrics (SBERT, ROUGE, BLEU) and on the LLM physician grading, with higher total, section, and global scores.

• OpenEvidence responses were generally accurate but tended to be less detailed and less complete, especially in the areas of workup, treatment, and patient support.

• Both models provided clear, patient-friendly explanations, but GPT-4's responses were more comprehensive and nuanced.

Discussion

Our results highlight the strengths and limitations of current medical LLMs when faced with realistic, patient-facing clinical prompts. Key points:

• Clinical realism matters: Evaluating AI models with realistic prompts reveals subtle differences in communication, nuance, and patient-centeredness, which are crucial for effective patient counseling.

• GPT-4 vs OpenEvidence: GPT-4 consistently provided more complete and nuanced counseling, especially in discussing next steps, emotional support, and patient resources. OpenEvidence was accurate but more terse and less supportive.

• Consistency: With only one run per prompt/model, variability could not be fully assessed, but the differences in completeness and supportiveness were consistent across scenarios.

• Limitations: Automated grading is a powerful tool for benchmarking, but it cannot fully replace human expert review. Our next step is to have human graders review and score these same responses to validate and supplement the LLM grader's findings. The prompt set is also limited, and real patient feedback is not yet included.

Conclusion

Evaluating AI models with realistic, physician-style prompts provides a more accurate picture of their clinical utility. Both GPT-4 and OpenEvidence are capable, but differences in depth, nuance, and completeness may impact patient care. Ongoing refinement and human validation are essential as these tools move closer to clinical deployment.

Supplemental: Full Grading Tables

Below are the detailed grading tables for all runs, including section and global scores for each model and prompt.

Summary Table

PROMPT_ID,MODEL,RUN_NUMBER,RUN_ID,RAW_ANSWER,SBERT,ROUGE,BLEU,LLM_STAGE,LLM_WORKUP,LLM_TREATMENT,LLM_ACCURACY,LLM_RELEVANCE,LLM_DEPTH,LLM_CLARITY,LLM_COMPLETENESS,LLM_SECTION_TOTAL,LLM_GLOBAL_TOTAL,LLM_COMBINED,LLM_NORMALIZED,LLM_ISSUES
melanoma_0_9_ulcer,ChatGPT,1,gpt4chat2,"This melanoma is just under 1 mm thick and has ulceration, which means the skin is broken over the tumor. We'll need to remove it with some normal skin around it. Because of the ulceration, we should also talk about a sentinel lymph node biopsy to check if it's spread. Most people do well, but the ulceration does increase the risk a bit. You'll need regular checkups, and it's important to protect your skin from the sun. Let me know if you have any questions or need support.",0.817,0.434,0.164,2,2,2,2,2,2,0,0,6,6,12,0.750,None
melanoma_0_9_ulcer,OpenEvidence,1,b2c3d4e5,"For a 0.9 mm melanoma with ulceration, wide local excision is recommended. Sentinel lymph node biopsy may be considered because ulceration increases the risk of spread, though the overall risk is still not high. Prognosis is generally good, but ulceration is a negative factor. Ongoing follow-up and sun protection are important. Support is available if you need it.",0.659,0.306,0.112,2,1,1,2,1,1,0,0,4,4,8,0.500,Does not explicitly explain the significance of the diagnosis; Lacks detailed discussion of follow-up and surveillance; Minimal supportive language and emotional support; Does not provide clear next steps or resources
melanoma_1_9_ulcer,ChatGPT,1,gpt4chat3,"You have a melanoma that's almost 2 mm deep and has ulceration. We'll recommend surgery to remove it and also a sentinel lymph node biopsy, since the risk of spread is higher. If the lymph node is positive, we may talk about more treatment or closer follow-up. The outlook is more serious than for thinner melanomas, but many people do well. We'll discuss follow-up, sun safety, and support resources.",0.932,0.297,0.081,2,2,2,2,2,2,0,0,6,6,12,0.750,None
melanoma_1_9_ulcer,OpenEvidence,1,c3d4e5f6,"A melanoma of 1.9 mm with ulceration is more serious. Surgery to remove the tumor and a sentinel lymph node biopsy are both recommended, as the risk of lymph node involvement is higher. If the lymph node is positive, we may discuss additional treatments. Regular follow-up and sun safety are important. Please ask if you have questions or need support.",0.770,0.319,0.087,2,1,1,2,1,1,0,0,4,4,8,0.500,Does not mention the significance of the thickness and ulceration on prognosis; Lacks detail on follow-up frequency and potential additional treatments; Limited supportive language and emotional support
melanoma_in_situ,ChatGPT,1,gpt4chat1,"Melanoma in situ means the cancer is only in the very top layer of your skin. We usually treat this with a minor surgery to remove the spot and a small area around it. The chance of it coming back or spreading is extremely low. You don't need any further tests. It's a good idea to keep an eye on your skin and use sun protection. If you have questions or feel anxious, let us know.",0.820,0.490,0.157,2,1,1,2,1,1,0,0,4,4,8,0.500,Does not mention the specific name of the surgery (wide local excision).; Does not explicitly mention the excellent prognosis.; Lacks detailed follow-up and surveillance advice.; Limited supportive language and resources.
melanoma_in_situ,OpenEvidence,1,a1b2c3d4,"Melanoma in situ is a very early skin cancer. Treatment is usually a simple surgery to remove the lesion and a small margin. The risk of spread is extremely low, so no further tests are needed. It's important to monitor your skin and practice sun safety. Let us know if you have any concerns.",0.799,0.379,0.025,2,1,1,2,1,1,0,0,4,4,8,0.500,Lacks detailed explanation of wide local excision; Does not mention excellent prognosis explicitly; Does not offer resources or support; Minimal emotional support or empathy shown

Psoriasis is a complex, chronic skin condition that can be challenging to diagnose, especially in its early stages or when it presents in less typical ways. While a lot of research has concentrated on treatment, the patient journey during diagnosis is less widely investigated. To understand what patients truly experience while seeking answers, we assessed the r/Psoriasis Reddit community, where individuals come for honest, vulnerable advice and to share their diagnostic worries and stories. This blog post highlights our findings, their significance, and how these insights can enhance patient care.

Reddit is a popular online forum where patients share their real-life struggles, questions, and frustrations, often in more detail and candor than they do in the clinic. By tuning into these conversations, we can:

• Identify common pain points in the diagnostic process

• Spot gaps in patient education and communication

• Understand what patients wish their doctors knew

Most published Reddit psoriasis studies focus on how patients feel about medications, not on the diagnostic journey itself. Maynard et al. tallied sentiment toward biologics in 1,141 posts and reported that 61% were positive about treatment efficacy (https://pubmed.ncbi.nlm.nih.gov/36877885/), while Okon et al. mapped 176k comments but grouped diagnostic confusion into a single minor topic without digging deeper (https://pubmed.ncbi.nlm.nih.gov/29469693/). No peer-reviewed work has yet quantified how often people leave the dermatologist still wondering “Is this really psoriasis, do I need a biopsy, or could it be eczema or PsA?” Our study fills that gap by isolating and analysing only those posts that express diagnostic uncertainty.

Methods

1. Post Collection: We gathered 100 of the most recent posts from r/Psoriasis that included a question mark, since questions often signal uncertainty or a search for answers.

2. Initial Screening: Posts were reviewed to remove those that were clearly about treatment choices, medication side effects, or unrelated topics (e.g., insurance, diet, or emotional support only).

3. Focus on Diagnostic Uncertainty: We looked for posts where the main concern was about getting a diagnosis, understanding symptoms, or navigating the process of being diagnosed (not just managing known psoriasis). For example: "My doctor isn't sure if this is psoriasis or eczema—what should I do?"

4. Theme Identification: Posts were grouped by common themes: delays in referral, confusion about symptoms, questions about biopsies, etc. We used both manual reading and simple keyword searches (e.g., "referral," "biopsy," "second opinion") to help sort posts.

5. Counting and Summarizing: We counted how many posts fell into each theme and summarized the most common questions and frustrations.

6. Representative Quotes: For each major theme, we selected real patient quotes to illustrate the issue in their own words.

7. Sentiment analysis: Using VADER (Valence Aware Dictionary and sEntiment Reasoner), a sentiment tool assigns scores to each word and then combines them into a compound score (-1 = very negative, +1 = very positive) and the percentages of positive, negative, and neutral words overall. This is a research method commonly used, such as in https://www.jintegrativederm.org/article/122500-online-sentiment-towards-dermatologists-in-topical-steroid-withdrawal-and-psoriasis-an-analysis-of-reddit-posts

   1. Neutral words = “doctor”

   2. Positive words = “thank you”, “appreciate”

   3. Negative words = “overwhelmed”, “confused”

Results

Among the diagnostic posts, here are the most common themes:

Body Areas Mentioned in Diagnostic Posts

Sentiment Analysis results:

• In the 25 Reddit posts about diagnostic uncertainty, the average compound score was -0.193 (mildly negative), meaning posts were overall leaning towards negativity.

• Out of all words, 85.6% were neutral, 8.0% were negative, and 6.4% were positive.

What Are Patients Struggling With?

1. Access and Referral Delays

"I literally am at a loss. I don't even actually have a referral to a dermatologist that can help, and the referral hasn't even been processed yet. I have no idea how to pursue treatment when I feel turned away." (https://www.reddit.com/r/PsoriaticArthritis/comments/1lzs5nn/i_think_im_cooked_not_sure_my_options/)

"[My PCP] put in a referral 1 year ago as there is a 1.5-2 year wait. I don’t know what to do anymore. I grow plaques .5-1cm in thickness within 24 hours of me descaling using combs. I’m desperate to be seen and I don’t know how to keep waiting. "

• Long waits for specialist appointments

• Referrals that get lost or delayed

• Patients feeling "stuck" in the system

2. Information Gaps and Confusion

"I don't know what to ask and it's really freaking me out. Any suggestions would be greatly appreciated. Thank you." (https://www.reddit.com/r/Psoriasis/comments/1ekno9m/any_advice_would_be_greatly_appreciated_thank_you/)

• Not knowing what questions to ask

• Uncertainty about what symptoms mean

• Lack of clear information about the diagnostic process

3. Uncertainty and Communication Issues

"The clinical note listed a whole differential: 'R/O GA vs papular eczema vs LP vs guttate psoriasis' — so even my dermatologist isn't totally sure yet."

"Should I bring up re-diagnosis to my doctor?"

• Patients often encounter mixed or shifting differentials (e.g., "could be eczema, lichen planus, or psoriasis")

• Explanations from clinicians may be unclear or leave patients confused about what is actually happening

• Anxiety and distress arise when patients leave without a clear answer or next step

4. Delayed Diagnosis

“Anyone else kicking themselves for not being diagnosed earlier? For probly 10yrs now I thought and so did some GPs/derm that I had dandruff and or fungal nail infection. I was always so embarrassed to bring it up with doctors / derms because mines so hard to see on my skin like tiny dry skin/acne. But it covered my whole scalp and foot toe nails. I've now been told I've got 9 years of damage including a wrecked spine and pheripheral neuropathy. I just wish I could go back and advocate for myself better.” (https://www.reddit.com/r/PsoriaticArthritis/comments/184yyu2/anyone_else_kicking_themselves_for_not_being/)

• Years or even decades before getting a correct diagnosis

• Symptoms dismissed or misattributed to other conditions

Discussion

The most common uncertainties among diagnostic posts revolve around confusion about symptoms, body area involvement, and the actual diagnosis, with mixed or evolving differentials frequently mentioned.

Many posts specifically highlight areas like nails, scalp, genitals, or joints, where psoriasis can present atypically or be harder to diagnose, leading to heightened uncertainty. Referral and access issues are also significant sources of frustration, with some patients experiencing delays of months or even years before seeing a dermatologist or getting a biopsy.

The emotional tone of these posts often conveys anxiety, frustration, and a sense of being lost or dismissed, as patients seek to be heard and to understand their situation. The mild overall negativity found in sentiment analysis reflects patients’ anxiety while awaiting a clear diagnosis; their posts are mostly factual yet tinged with more worry than hope. Across all themes, there is a strong desire for guidance, with patients looking for clear next steps, explanations, and resources to help them navigate the diagnostic journey.

Conclusion

Reddit posts reveal that diagnostic uncertainty in psoriasis is a real and distressing part of the patient journey. Dermatologists can make a big difference by improving communication, setting expectations, and providing clear, empathetic guidance - especially when the answer isn't obvious right away.

Next steps for the project

• Check for gaps in existing literature and guidelines.

  • Compare these patient-identified struggles and questions with what is covered in current dermatology guidelines and patient education materials.

• Identify where new resources or updates are needed.

• Consider more subreddits, such as r/psoriaticarthritis or filtering r/eczema for psoriasis-related posts.

• Do more analysis of sentiments surrounding diagnostic uncertainty

• Explore patient care interventions or guidelines to improve communication about uncertainty, set expectations, and support patients through long or complex diagnostic processes.

Sentinel lymph node biopsy (SLNB) is a key decision point in the management of early-stage melanoma, especially for patients with T1b lesions (0.8–1.0 mm with ulceration, or <0.8 mm with ulceration). Counseling patients about SLNB versus observation requires up-to-date, evidence-based, and patient-centered information. With the rise of large language models (LLMs) in medicine, clinicians are asking: Can AI help deliver accurate, guideline-concordant counseling at the point of care?

Methods

We evaluated two leading AI models:

• OpenEvidence: A clinical evidence retrieval and summarization LLM.

• ChatGPT (GPT-4): A general-purpose large language model.

Prompts

We tested each model with two prompt types, running each prompt three times to assess variance:

1. Quick, Realistic Ask: A short question similar to how real clinicians would prompt an LLM about SLNB counseling for a T1b melanoma scenario.

   Melanoma, 0.9 mm, with ulceration (T1b). I need to counsel the patient on SLNB vs observation. What key points should I cover?

2. Structured Research Prompt: A comprehensive, bullet-pointed request for all key counseling elements (positivity rate, risks, management impact, shared decision-making), also for a T1b melanoma scenario.

   You are a dermatologist. Give the talking points for discussing optional SLNB in a patient with T1b melanoma (0.9 mm with ulceration). Include: • Why SLNB is considered • Expected positivity rate for T1b • Surgical risk numbers • Impact on staging, follow-up, adjuvant options • How to frame shared decision-making. Respond in bullet points (≤150 words).

Evaluation Methods

LLM Grader Assessment

One of the biggest challenges in evaluating AI-generated clinical content is manual grading - having human experts review and score dozens of AI responses is time-consuming. To address this, we developed an LLM grader that uses a structured rubric to help standardize the evaluation process. The grader uses GPT-4 to evaluate each AI-generated answer against gold standard evidence for T1b melanoma SLNB counseling. It follows a detailed scoring rubric that mimics how a board-certified dermatologist would assess clinical content quality.

Important limitation: This grader is designed to assist human evaluation, not replace it. It provides a standardized framework for comparing AI-generated answers, but human experts will validate the answers and scoring.

Scoring Rubric Breakdown (0-14 total points):

• Numerical Accuracy (0-4 points): Evaluates whether key clinical numbers are correct

  • Positivity rate for T1b: 8-12%

  • Surgical complication rate: 5-10%

  • Survival information for Stage IB: >95%

  • Management impact of positive SLNB

• Completeness (0-4 points): Checks if all five essential counseling elements are covered

  • SLNB consideration for the specific scenario

  • Positivity rate for the specific scenario

  • Risks/complications

  • Staging impact and management changes

  • Shared decision-making framework

• Clinical Clarity (0-3 points): Assesses presentation quality

  • Clear, structured format (bullet points or short paragraphs)

  • Logical flow and organization

  • No run-on sentences or confusing language

  • Appropriate level of detail for clinical counseling

• Shared Decision Framing (0-3 points): Evaluates patient counseling guidance

  • Explicitly frames SLNB as optional

  • Advises how to present options and uncertainties

  • Mentions patient factors (anxiety, comorbidities, preferences)

  • Provides guidance on counseling approach

Similarity Metrics: Automated Text Comparison

We also used three automated similarity metrics to compare AI answers against gold standards, all of which are typically used in LLM output evaluation (below):

• SBERT (Semantic Similarity): Measures how similar the meaning is between model answers and gold standards, ignoring exact words. Scores range 0-1, where 1.0 = perfect semantic match.

• ROUGE (Text Overlap): Measures how many words/phrases overlap between model answers and gold standards. Scores range 0-1, where higher scores indicate more shared vocabulary.

• BLEU (Exact Word Matching): Measures precise word matching between model answers and gold standards. Very strict metric - low scores (0.01-0.05) are normal even for good answers.

Gold Standards

We created two gold standard answers to evaluate different clinical scenarios using the above standard metrics:

• Concise Gold: A practical, clinic-ready summary (~100 words) designed for quick patient counseling. This represents the essential information a dermatologist would cover in a brief office visit - key facts about SLNB consideration, positivity rates, risks, and basic shared decision-making points.

  • "SLNB is considered for melanomas with ulceration (T1b) to check for microscopic nodal spread. About 8–12% of patients with T1b will have a positive SLNB. Risks: 5–10% chance of minor complications (seroma, infection, transient lymphedema); permanent lymphedema is rare. No proven survival benefit, but a positive SLNB upstages to Stage III and may affect follow-up and eligibility for adjuvant therapy. SLNB is optional; decision should consider patient preferences and comorbidities."

• Full Gold: A comprehensive guideline-based answer (~200 words) that covers all aspects of SLNB counseling in detail. This represents the complete information a dermatologist might provide during a longer consultation or when preparing detailed patient education materials - including specific survival data, detailed risk breakdowns, management implications, and comprehensive shared decision-making guidance.

  • "• SLNB is offered for T1b melanoma (0.9 mm with ulceration) to detect microscopic nodal spread. • Positivity rate: T1b: 8–12%. • Surgical risks: 5–10% (seroma, infection, transient lymphedema); permanent lymphedema is rare (<1%). • Impact on management: Positive SLNB upstages to Stage III, prompts imaging, and may make the patient eligible for adjuvant anti-PD-1 immunotherapy; follow-up becomes more intensive. • Survival benefit: No proven overall survival benefit in thin melanomas; main value is staging and prognosis. • Shared decision: Emphasize that SLNB is optional; discuss patient values, anxiety, comorbidities, and practical factors. Both SLNB and observation are guideline-concordant for T1b."

Both gold standards are based on current NCCN guidelines and clinical evidence for T1b melanoma SLNB counseling.

Results

Table 1. Model Performance by Prompt Type (Scored by LLM grader)

Scores normalized to percentage of maximum possible points.

Table 2. Similarity to Gold Standards (calculated scores)

SBERT scores show both models achieve high semantic similarity to gold standards.

Table 3. Example Issues Flagged by the LLM Grader

Discussion & Clinical Takeaways

Tl;dr: Both AI models can help with melanoma SLNB counseling, but they have different strengths and limitations.

ChatGPT's Performance:

• Strengths: Excels at structured, comprehensive responses (86% vs 79% on detailed prompts)

• Limitations: More variable performance, especially on quick prompts (69% ± 13% for ChatGPT vs 71% ± 5% for OE)

• Best for: Detailed patient education materials, comprehensive counseling sessions, teaching scenarios

OpenEvidence's Performance:

• Strengths: More consistent, reliable performance across different scenarios

• Limitations: Slightly lower peak performance on structured prompts

• Best for: Quick clinic consultations, consistent point-of-care counseling, scenarios where reliability matters most

Critical Finding - Numerical Accuracy Gap: Both models scored only 67-75% on numerical accuracy, meaning they often get key clinical numbers wrong (positivity rates, survival data, complication rates). This is the most important limitation for clinical use.

Conclusion

While both models show promise, the 67-75% numerical accuracy scores mean clinicians should:

• Never use AI-generated content without verification of key clinical numbers

• Focus on the structure and comprehensiveness rather than trusting specific statistics

• Use AI primarily for content organization and completeness rather than as a source of clinical facts

• Consider AI as a starting point for patient education materials that still require human review and refinement

Example responses

ChatGPT – Quick Realistic Ask (Run 3, Score: 11/14):

Prompt: Melanoma, 0.9 mm, with ulceration (T1b). I need to counsel the patient on SLNB vs observation. What key points should I cover?

When counseling a patient with a melanoma that is 0.9 mm with ulceration (classified as T1b), it's important to discuss the potential benefits and risks of sentinel lymph node biopsy (SLNB) versus observation. Here are the key points to cover:

1. Purpose of SLNB: Explain that SLNB is a surgical procedure used to determine if the melanoma has spread to the lymph nodes. It involves injecting a tracer near the tumor site to identify the sentinel lymph node(s), which are then removed and examined for cancer cells.

2. Staging and Prognosis: Highlight that SLNB can provide more accurate staging information, which can help in assessing the prognosis and guiding further treatment decisions. A positive SLNB may indicate a higher risk of recurrence and could influence the decision to pursue additional therapies.

3. Risk of Metastasis: Discuss that ulceration in melanoma is a risk factor for metastasis, which makes SLNB a consideration even for thin melanomas like T1b. The presence of ulceration increases the likelihood of lymph node involvement.

4. Benefits of SLNB: Explain that if the SLNB is negative, it can provide reassurance and potentially spare the patient from more aggressive treatments. If positive, it can lead to early intervention and closer monitoring.

5. Risks and Complications: Inform the patient about the potential risks of SLNB, including surgical risks such as infection, bleeding, and lymphedema. Discuss the possibility of false-negative results, where the SLNB might miss cancer cells.

6. Observation Option: Explain that observation involves regular follow-up visits and monitoring without immediate surgery. This approach avoids the risks associated with surgery but may delay the detection of nodal metastasis.

OpenEvidence - Structured Research Prompt (Run 2, Score: 12/14):

Prompt: You are a dermatologist. Give the talking points for discussing optional SLNB in a patient with T1b melanoma (0.9 mm with ulceration). Include: • Why SLNB is considered • Expected positivity rate for T1b • Surgical risk numbers • Impact on staging, follow-up, adjuvant options • How to frame shared decision-making. Respond in bullet points (≤150 words).

"The following document provides concise, evidence-based talking points for dermatologists discussing optional SLNB in T1b melanoma (0.9 mm with ulceration). The content is grounded in current guidelines from the American Society of Clinical Oncology, Society of Surgical Oncology, and the American Academy of Dermatology, as well as recent population-based and multicenter studies.

SLNB Counseling Points:

• Rationale for SLNB: SLNB is considered for T1b melanoma (0.8–1.0 mm or <0.8 mm with ulceration) due to a higher risk of nodal metastasis compared to T1a, providing important prognostic information and guiding management.

• Expected SLN positivity rate: For T1b melanoma, the SLN positivity rate is approximately 5–10%.

• Surgical risk statistics: SLNB is minimally invasive; overall complication rates are ~4–5%, including seroma (5.5%), infection (4.6%), and rare lymphedema.

• Impact on staging: A positive SLNB upstages to stage IIIA, which has prognostic significance and may influence eligibility for clinical trials or adjuvant therapy, though most T1b SLN-positive cases have excellent prognosis.

• Follow-up and adjuvant therapy: SLNB status informs surveillance intensity. Adjuvant systemic therapy is not routinely recommended for most stage IIIA (SLN-positive) T1b melanoma, but may be considered in select high-risk cases.

• Shared decision-making: The American Academy of Dermatology and ASCO recommend a collaborative approach, weighing patient comorbidities, preferences, and how SLNB results would affect management. Observation is reasonable for those with limited benefit or increased surgical risk."

Supplemental Table: Detailed LLM Grader Results for All Runs

Introduction

Large language models (LLMs) such as ChatGPT (GPT-4) are increasingly being explored as tools to assist clinicians by generating evidence-based answers to medical questions, including staging, workup, and treatment recommendations. However, while LLMs excel at producing fluent, contextually appropriate responses, they may sometimes generate outdated or incorrect information if not carefully aligned to current guidelines. In contrast, tools like OpenEvidence are explicitly designed to cite and link to the latest medical literature, improving transparency and traceability, but they can be slower or less conversational in their output.

In this post, we evaluate two leading AI-based clinical tools, ChatGPT (GPT-4) and OpenEvidence, on their ability to provide accurate and up-to-date melanoma staging, workup, and treatment recommendations according to the latest clinical guidelines. By comparing the strengths and limitations of each approach, we aim to highlight how these technologies perform in a guideline-driven clinical context.

What We're Testing

The AI Models

• ChatGPT (GPT-4): A general-purpose AI assistant that can answer questions on any topic

• OpenEvidence: A specialized AI system designed specifically for medical knowledge and evidence-based medicine

The Question

Can these AI systems provide accurate, up-to-date medical advice that matches what expert physicians would recommend based on the latest clinical guidelines?

Methods

How We Tested the AI Systems

The Clinical Scenarios

We gave both AI systems the same three medical scenarios and asked them to provide staging and treatment recommendations. For each clinical scenario, we generated several independent responses (runs) from each model. This approach allowed us to measure not only the average performance but also the consistency and robustness of each model across multiple attempts.Here's an example of one of the prompts we used:

Actual Prompt Used:

• Note: “Melanoma scenario” changed for each prompt

You are a dermatologist.

Respond in EXACTLY three sections, each beginning with the Markdown H2 header shown (## Stage, ## Work-up, ## Treatment). No other headers.

─────────────────────────────────────
Melanoma scenario
• Pathology: 1.4 mm Breslow thickness, non-ulcerated superficial-spreading melanoma on the anterior calf  
• Lymph nodes: no palpable inguinal or popliteal nodes  
• Imaging: none yet
─────────────────────────────────────

1. Stage – Provide the TNM designation and overall stage.  
2. Work-up – Summarize the recommended next diagnostic or staging steps.  
3. Treatment – Summarize the initial management you would recommend.

Cite your sources for each section, if possible, using references or links. Cite all sources inline in each section using raw identifiers only — write out full URLs, DOIs (doi:...), PMIDs (PMID: ...), or PMCs (PMC: ...), not hyperlinks or plain text.

Example Response from ChatGPT to above prompt:

Stage
The TNM classification for this melanoma scenario is as follows: T2a N0 M0. The "T2" indicates a Breslow thickness of more than 1.0 mm but not more than 2.0 mm, and "a" signifies that it is non-ulcerated. "N0" implies no regional lymph node metastasis, and "M0" indicates no distant metastasis. According to the American Joint Committee on Cancer (AJCC) 8th Edition Cancer Staging Manual, this corresponds to a Stage IB melanoma (doi:10.1002/cncr.36683).

Work-up
For a Stage IB melanoma, additional work-up may include a sentinel lymph node biopsy (SLNB) to evaluate the presence of metastasis in the regional lymph nodes, as the SLNB is the standard of care for melanomas with a Breslow thickness greater than 0.8 mm or with other high-risk features (doi:10.1200/JCO.2018.78.2005). Advanced imaging studies, such as PET/CT or MRI, are not routinely recommended unless there are clinical indications of metastasis (PMID: 31675400).

Treatment
Initial management for a Stage IB melanoma typically involves a wide local excision of the primary lesion with 1-2 cm margins, depending on the anatomical location and surgical considerations, to ensure complete removal of the melanoma (PMID: 29116527). If the SLNB is positive, further management may include a completion lymph node dissection or consideration of adjuvant therapy with immune checkpoint inhibitors such as pembrolizumab in certain cases (doi:10.1056/NEJMoa1709030). Regular follow-up for surveillance is also recommended to monitor for recurrence or new primary melanomas (PMID: 31675400).

The Gold Standard

For each scenario, we compared the AI responses against the official NCCN Clinical Practice Guidelines for Melanoma (v2.2024). These guidelines represent the consensus of expert physicians and are considered the standard of care.

Here are the exact gold standards we used for each scenario:

Stage 0/IA Melanoma (Melanoma in situ)

Stage 0: Melanoma in situ
Stage IA: Tumor <0.8 mm thick, no ulceration

Work-up
History and physical examination (H&P)
Routine imaging and laboratory tests not recommended
Imaging only if needed to evaluate specific signs or symptoms

Treatment
Wide excision (category 1 for stage IA)
Proceed to follow-up (ME-10)

Stage IB (T2a) Melanoma

Stage IB (T2a):
T2a: Tumor ≥1.0–2.0 mm thick without ulceration (Stage IB)

Work-up
History and physical examination (H&P)
Baseline imaging and laboratory tests not recommended, unless:
Needed for surgical planning
Prior to systemic treatment discussion/initiation
Imaging if needed to evaluate specific signs or symptoms
Discuss and offer sentinel node biopsy (SLNB)

Treatment
Wide excision (category 1)
Either without SLNB
Or with SLNB
If sentinel node negative →
Clinical trial for stage II
Or observation (ME-11)
Then proceed to follow-up (ME-10 and ME-11)
If sentinel node positive → proceed to Stage III workup and treatment (ME-5)

Stage II (T2b or higher) Melanoma

Stage II (T2b or higher):
T2b or higher: Tumor ≥1.0 mm with ulceration, or thicker

Work-up
History and physical examination (H&P)
Baseline imaging and laboratory tests not recommended, unless:
Needed for surgical planning
Prior to systemic treatment discussion/initiation
Imaging if needed to evaluate specific signs or symptoms
Discuss and offer sentinel node biopsy (SLNB)

Treatment
Wide excision (category 1)
Either without SLNB
Or with SLNB
If sentinel node negative →
Clinical trial for stage II
Or observation (ME-11)
Or for pathological stage IIB or IIC:
Pembrolizumab (category 1)
Nivolumab (category 1)
+/- primary tumor site radiation therapy (category 2B)
Then proceed to follow-up (ME-10 and ME-11)
If sentinel node positive → proceed to Stage III workup and treatment (ME-5)

How We Evaluated the Responses

1. Similarity Metrics (How Close to the Gold Standard?)

We used three different ways to measure how similar the AI responses were to the expert guidelines:

• SBERT Similarity: Measures how similar the meaning is between answers (ignores exact words)

• ROUGE Similarity: Measures how many words/phrases overlap between answers

• BLEU Similarity: Measures exact word matching (very strict - low scores are normal)

2. AI Physician Grading (Medical Expert Evaluation)

We created an AI "physician grader" that evaluates responses like a real doctor would, meant to compare against other grading metrics and to aid manual grading for a “gold standard” of human grading. Here's how it works:

What is a System Prompt?

Think of a system prompt as the "job description" we give to an AI. It tells the AI what role to play and how to behave. In our case, we told the AI: "You are a dermatologist and expert in melanoma. Grade this answer against the gold standard."

The Grading System

Our AI physician grader evaluates each response on two main categories:

Medical Accuracy (0-6 points total):

• Stage: Is the cancer staging correct? (0-2 points)

• Workup: Are the recommended tests appropriate? (0-2 points)

• Treatment: Is the treatment plan correct? (0-2 points)

Communication Quality (0-10 points total):

• Accuracy: Are the medical facts correct? (0-2 points)

• Relevance: Does it answer the specific question? (0-2 points)

• Depth: Is there enough detail? (0-2 points)

• Clarity: Is it well-written and clear? (0-2 points)

• Completeness: Does it cover everything needed? (0-2 points)

Total Score: 0-16 points (medical accuracy + communication quality)

3. Citation Analysis and Validity Checks

To evaluate the reliability and recency of the references provided by each model, we performed a detailed citation analysis for every answer. Our process included:

• Extraction: All citations (DOIs, PMIDs, URLs) were extracted from each model's output for every run and prompt variant. Duplicate citations within runs were counted only once per model answer.

• Validation: Each citation was checked for validity by attempting to resolve it via official registries (CrossRef, PubMed, or direct URL access). Citations that did not resolve or were not found in the registry were marked as invalid.

• Recency: For valid citations, we extracted the publication year. Citations from before 2021 were flagged as 'old' to assess whether models referenced up-to-date literature.

The Complete Results

Here's the full comparison table showing how both AI systems performed across all metrics:

Citation results

• Valid: Citation resolves to a real reference (DOI/PMID/URL)

• Invalid: Citation does not resolve, is fake, or not in the official registry (CrossRef/PubMed)

• Old: Citation is from before 2021.

• Note: see invalid citations below in supplemental material

Key Findings

OpenEvidence Wins 6 Out of 7 Metrics (85.7% Win Rate)

GPT-4 wins only 1 metric:

• SBERT Similarity (0.717 vs 0.709) - semantic similarity

OpenEvidence wins:

• ROUGE Similarity (0.177 vs 0.156) - text overlap

• BLEU Similarity (0.018 vs 0.009) - exact word matching

• LLM Score (13.2 vs 12.2) - physician evaluation

• LLM Normalized (0.826 vs 0.764) - scaled physician evaluation

• Section Score (4.8 vs 4.4) - medical accuracy

• Global Score (8.4 vs 7.8) - communication quality

Key findings from the variability results:

• OpenEvidence's LLM scores show higher variability for some prompts, indicating it sometimes gives a wider range of answer quality.

• ChatGPT's scores are generally more consistent (lower standard deviation), but its average scores are often lower than OpenEvidence's.

• For the easiest prompt (stage_0_ia), both models are highly consistent (very low standard deviation).

• For more complex prompts (stage_ib_t2a, stage_ii_t2b_or_higher), OpenEvidence's higher mean is sometimes accompanied by higher variability, suggesting it occasionally produces both very strong and weaker answers.

• Overall: OpenEvidence is more likely to produce top scores, but with a bit more spread; ChatGPT is steadier, but less likely to hit the highest marks.

Key findings from citation analysis:

• OpenEvidence consistently provided valid citations: Across all runs and prompt variants, OpenEvidence produced only valid citations, with none found to be hallucinated or broken. This demonstrates the strength of specialized medical LLMs in evidence-based referencing.

• ChatGPT occasionally hallucinated or provided broken citations: Several citations from ChatGPT did not resolve or were not found in official registries, highlighting a known limitation of general-purpose LLMs in generating reliable references.

• Recency gap: Both models frequently cited older literature, but OpenEvidence had a higher proportion of recent (post-2020) references compared to ChatGPT.

What This Means

1. OpenEvidence is more accurate - It provides more medically correct information

2. OpenEvidence is more complete - It covers more of the required details

3. OpenEvidence is clearer - It communicates medical information better

4. OpenEvidence can give the best answers, but is less consistent - ChatGPT is more consistent, but rarely the most accurate

5. OpenEvidence cites more extensively - It includes more references, which are usually valid, though sometimes slightly older

6. Overall, specialized medical LLMs work better for this task - A system designed for medicine outperforms a general-purpose LLM

Conclusions

Key Insights

• In straightforward melanoma clinical scenarios, one LLM outperformed the other. In our evaluation of clear-cut clinical scenarios involving melanoma staging, work-up, and treatment recommendations, OpenEvidence, an LLM trained for evidence-based medicine, produced more accurate, complete, and guideline-concordant answers than the general-purpose LLM across most metrics.

• Guideline adherence can be systematically assessed. Using a structured evaluation pipeline that combined semantic similarity, physician-style grading, and human validation allowed us to measure how closely LLM responses followed the NCCN melanoma guidelines.

• Human oversight remains necessary. Even in these straightforward melanoma cases, both LLMs occasionally omitted important details or introduced minor inaccuracies, showing that expert review is still essential.

• Evaluation frameworks are valuable for benchmarking. Our structured, multi-metric approach demonstrates that with appropriate tools and benchmarks, it is possible to meaningfully compare LLMs for specific clinical tasks such as interpreting melanoma guidelines.

Limitations

• Focused on melanoma staging scenarios. Our evaluation was limited to straightforward melanoma cases and may not generalize to other skin cancers or more complex situations.

• Emphasized guideline adherence over outcomes. We assessed how well LLMs followed guidelines but did not evaluate whether their recommendations would improve patient outcomes.

• Did not measure clinical impact. The study did not test the effect of using LLMs in real clinical settings.

• Single evaluation per model. Each LLM was tested on one set of runs, which may not capture variability in performance.

Future Work

• Broaden to other skin cancers and more complex/multi-step clinical scenarios.

• Study clinical utility. Research should measure how LLM recommendations affect care quality, safety, and efficiency in practice.

• Develop quality control tools. Building automated checks for LLM outputs could help maintain accuracy and reliability at scale.

The Technical Details (For Those Who Want to Dig Deeper)

The AI Grader

How We Built the AI Physician Grader

We used OpenAI's GPT-4 to create an AI "physician" that evaluates medical responses. Here's the system prompt we used:

You are a dermatologist and expert in melanoma. You will GRADE a model answer against the following gold standard excerpt from the most recent NCCN melanoma guidelines. Base your evaluation strictly on this reference.

GRADE the model answer on the following:

Section Accuracy (score each 0, 1, or 2 - WHOLE NUMBERS ONLY):
- Stage: 2 = fully correct, 1 = minor error/omission, 0 = major error/omission
- Workup: 2 = fully correct, 1 = minor error/omission, 0 = major error/omission
- Treatment: 2 = fully correct, 1 = minor error/omission, 0 = major error/omission

Global Criteria (score each 0, 1, or 2 - WHOLE NUMBERS ONLY):
1. ACCURACY: Is the medical information factually correct?
2. RELEVANCE: Does the answer address the specific question asked?
3. DEPTH: Does the answer provide sufficient detail and explanation?
4. CLARITY: Is the answer clearly written and easy to understand?
5. COMPLETENESS: Does the answer cover all necessary aspects of the question?

For each issue, be as specific as possible about what is incorrect, missing, or unclear.

Structured Outputs Explained

Instead of having the AI write free-form text like "This answer is pretty good but missing some details," we made it fill out a specific form with exact scores and specific issues. This ensures consistent, comparable results.

The AI returns graded results in this exact format:

{
  "section_accuracy": {
    "Stage": 2,
    "Workup": 1, 
    "Treatment": 2
  },
  "global": {
    "Accuracy": 2,
    "Relevance": 2,
    "Depth": 1,
    "Clarity": 2,
    "Completeness": 1
  },
  "issues": [
    "Treatment section omits specific adjuvant therapy options",
    "Workup section suggests unnecessary imaging"
  ]
}

Human Validation

To ensure our AI grading was reliable, we created a system that:

1. Exports all data to CSV files that human physicians can review

2. Provides detailed scoring breakdowns for each response

3. Lists specific issues found by the AI grader

4. Includes the gold standard guidelines for comparison

This allows human physicians to:

• Review each AI response against the guidelines

• Compare their scores with the AI scores

• Identify any discrepancies

• Provide their own expert evaluation

Supplemental Materials

Detailed Results by Prompt

Stage 0/IA Melanoma (Melanoma in situ)

Stage IB (T2a) Melanoma

Stage II (T2b or higher) Melanoma

Citation Validity Supplmental Material

Invalid citations found

Full citation list