Abraham A. Brody

Abstract

Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/ false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Abstract

Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Abstract

The purpose of this article was to analyze the concept of "the fighter in the intensive care unit (ICU)"per the scientific literature and the impact this mentality has on care administered in the ICU. A literature review and a concept analysis based on Rodger's evolutionary method were performed to identify surrogate terms, antecedents, attributes, and consequences pertaining to the "fighter"in the ICU. Thirteen articles with a focus on "the fighter"were included in this analysis. There is a strong desire to remain optimistic and maintain high spirits as a coping mechanism in the face of extreme prognostic uncertainty. Themes that emerged from the literature were the need to find inner strength and persist in the face of adversity. The concept of "the fighter in the ICU"can serve as either adaptive or maladaptive coping, depending on the larger clinical picture. Patient experiences in the ICU are fraught with physical and psychological distress. How the patient and family unit cope during this anxiety-provoking time is based on the individual. Maintaining optimism and identifying as a fighter can be healthy ways to adapt to the circumstances. This concept analysis highlights the importance of holistic care and instilling hope particularly as patients may be nearing the end of life.

Abstract

Objectives: To describe the implementation of a mentored staff-delivered sleep program in nursing facilities. Design: Modified stepped-wedge unit-level intervention. Setting and Participants: This program was implemented in 2 New York City nursing facilities, with partial implementation (due to COVID-19) in a third facility. Methods: Expert mentors provided staff webinars, in-person workshops, and weekly sleep pearls via text messaging. We used the integrated Promoting Action on Research Implementation in Health Services (i-PARiHS) framework as a post hoc approach to describe key elements of the SLUMBER implementation. We measured staff participation in unit-level procedures and noted their commentary during unit workshops. Results: We completed SLUMBER within 5 units across 2 facilities and held 15 leadership meetings before and during program implementation. Sessions on each unit included 3 virtual webinar presentations and 4 in-person workshops for each nursing shift, held over a period of 3 to 4 months. Staff attendance averaged >3 sessions per individual staff member. Approximately 65% of staff present on each unit participated in any given session. Text messaging was useful for engagement, educational reinforcement, and encouraging attendance. We elevated staff as experts in the care of their residents as a strategy for staff engagement and behavior change and solicited challenging cases from staff during workshops to provide strategies to address resident behavior and encourage adoption when successful. Conclusions and Implications: Engaging staff, leadership, residents, and family of nursing facilities in implementing a multicomponent sleep quality improvement program is feasible for improving nursing facilities’ sleep environment. The program required gaining trust at multiple levels through presence and empathy, and reinforcement mechanisms (primarily text messages). To improve scalability, SLUMBER could evolve from an interdisciplinary investigator-based approach to internal coaches in a train-the-trainer model to effectively and sustainably implement this program to improve sleep quality for facility residents.

Abstract

To describe the implementation of a mentored staff-delivered sleep program in nursing facilities.

Abstract

Objectives: To evaluate the impact of a mentoring program to encourage staff-delivered sleep-promoting strategies on sleep, function, depression, and anxiety among skilled nursing facility (SNF) residents. Design: Modified stepped-wedge unit-level intervention. Setting and Participants: Seventy-two residents (mean age 75 ± 15 years; 61.5% female, 41% non-Hispanic white, 35% Black, 20% Hispanic, 3% Asian) of 2 New York City urban SNFs. Methods: Expert mentors provided SNF staff webinars, in-person workshops, and weekly sleep pearls via text messaging. Resident data were collected at baseline, post-intervention (V1), and 3-month follow-up (V2), including wrist actigraphy, resident behavioral observations, Pittsburgh Sleep Quality Index (PSQI), Patient Health Questionnaire-9 (PHQ-9) depression scale, Brief Anxiety and Depression Scale (BADS), Brief Cognitive Assessment Tool (BCAT), and select Minimum Data Set 3.0 (MDS 3.0) measures. Linear mixed models were fit for continuous outcomes and mixed-effects logistic models for binary outcomes. Outcomes were modeled as a function of time. Planned contrasts compared baseline to V1 and V2. Results: There was significant improvement in PSQI scores from baseline to V1 (P = .009), and from baseline to V2 (P = .008). Other significant changes between baseline and V1 included decreased depression (PHQ-9) (P = .028), increased daytime observed out of bed (P ≤ .001), and increased daytime observed being awake (P < .001). At V2 (vs baseline) being observed out of bed decreased (P < .001). Daytime sleeping by actigraphy increased from baseline to V1 (P = .004), but not V2. MDS 3.0 activities of daily living and pain showed improvements by the second quarter following implementation of SLUMBER (P's ≤ .034). There were no significant changes in BADS or BCAT between baseline and V1 or V2. Conclusions and Implications: SNF residents had improvements in sleep quality and depression with intervention, but improvements were not sustained at 3-month follow-up. The COVID-19 pandemic led to premature study termination, so full impacts remain unknown.

Abstract

To evaluate the impact of a mentoring program to encourage staff-delivered sleep-promoting strategies on sleep, function, depression, and anxiety among skilled nursing facility (SNF) residents.

Abstract

Background and Objectives: The coronavirus disease 2019 (COVID-19) pandemic severely disrupted hospice care, yet there is little research regarding how widespread disruptions affected clinician and family decision-making. We aimed to understand how the pandemic affected structures, processes, and outcomes of end-of-life care. Research Design and Methods: Retrospective narrative chart review of electronic health records of 61 patients referred and admitted to hospice from 3 New York City geriatrics practices who died between March 1, 2020, and March 31, 2021. We linked longitudinal, unstructured medical, and hospice electronic health record notes to create a real-time, multiperspective trajectory of patients' interactions with providers using directed content analysis. Results: Most patients had dementia and were enrolled in hospice for 11 days. Care processes were shaped by structural factors (staffing, supplies, and governmental/institutional policies), and outcomes were prioritized by care teams and families (protecting safety, maintaining high-touch care, honoring patient values, and supporting patients emotionally and spiritually). Processes used to achieve these outcomes were decision-making, care delivery, supporting a "good death,"and emotional and spiritual support. Discussion and Implications: Care processes were negotiated throughout the end of life, with clinicians and families making in-the-moment decisions. Some adaptations were effective but also placed extraordinary pressure on paid and family caregivers. Healthcare teams' and families' goals to meet patients' end-of-life priorities can be supported by ongoing assessment of patient goals and process changes needed to support them, stronger structural supports for paid and family caregivers, incentivizing relationships across primary care and hospice teams, and extending social work and spiritual care.

Abstract

The coronavirus disease 2019 (COVID-19) pandemic severely disrupted hospice care, yet there is little research regarding how widespread disruptions affected clinician and family decision-making. We aimed to understand how the pandemic affected structures, processes, and outcomes of end-of-life care.

Abstract

Background: Aortic Stenosis (AS) is a common syndrome in older adults wherein the narrowing of the aortic valve impedes blood flow, resulting in advanced heart failure.1 AS is associated with a high mortality rate (50 % at 6 months if left untreated), substantial symptom burden, and reduced quality of life.1-3 Transcatheter aortic valve replacement (TAVR) was approved in 2012 as a less invasive alternative to surgical valve repair, offering a treatment for older frail patients. Although objective outcomes have been widely reported,4 the perspectives of older adults undergoing the TAVR process have never been synthesized. Objectives: To contextualize the perspectives and experiences of older adults undergoing TAVR. Methods: An integrative review was conducted using Whittemore and Knafl's five-stage methodology.5 Four electronic databases were searched in April 2023. Articles were included if a qualitative methodology was used to assess the perceptions of older adults (>65 years old) undergoing or recovering from TAVR. Results: Out of 4619 articles screened, 12 articles met the criteria, representing 353 individuals from 10 countries. Relevant themes included the need for an individualized care plan, caregiver and family support, communication and education, persistent psychosocial and physical symptoms, and the unique recovery journey. Conclusion: Older adults with AS undergoing TAVR generally perceive their procedure positively. Improved interdisciplinary and holistic management, open communication, symptom assessment, support, and education is needed.