Benchmark Evaluation of a Tool-Augmented Large Language Model Agent Using Traditional Asian Medicine Metadata
- 문서 유형
- 연구출판물 · 학술지 논문
- 등재
- SCIEScopus
- 저널 순위
- JCR Q2Scopus Q1
- 저널
- APPLIED SCIENCES-BASEL16(7) · 2026
- 피인용
- Web of Science 0Scopus 0
- FWCI (Scopus)
- 0
- JCR IF
- JCR 2024 2.5
- ISSN
- 2076-3417
저자
- Lee, Won-Yung제1저자Wonkwang Univ, Coll Korean Med, Iksan 54538, South Korea
- 김지환Pusan Natl Univ, Sch Korean Med, Dept Sasang Constitut Med, Div Clin Med, Pusan 46241, South Korea
- Leem, JungtaeWonkwang Univ, Res Ctr Tradit Korean Med, Iksan 54538, South Korea
- Lee, Byung-WookDongguk Univ, Coll Korean Med, Gyeongju 38066, South Korea
- Lee, Seungho교신저자Woosuk Univ, Coll Korean Med, Jeonju 54987, South Korea
- Kim, Young Woo교신저자Dongguk Univ, Coll Korean Med, Gyeongju 38066, South Korea
초록
This study evaluated a tool-augmented large language model (LLM) agent system utilizing a traditional Asian medicine (TAM) metadata. A structured information comprising 4780 entries across five entity types (herbs, syndromes, TCM symptoms, modern medicine symptoms, and acupoints) was constructed. Four LLMs (GPT-5.2, GPT-5-mini, Claude Sonnet 4.6, and Claude Haiku 4.5) were evaluated under baseline, non-agentic retrieval (RAG), and tool-augmented (agent) conditions using two public benchmarks: TCMBench (1300 items) and TCMEval-SDT (600 items). McNemar’s test and bootstrap confidence intervals were applied to examine performance differences. Tool augmentation showed a nominally significant improvement for GPT-5-mini on TCMBench term-related items (+4.5 percentage points on paired items, McNemar p = 0.034, 95% CI [+0.9, +8.5] pp), while Claude Sonnet 4.6 showed a nominally significant decline on SDT pathogenesis (−0.038, bootstrap p = 0.006). The agent condition outperformed the non-agentic RAG baseline for GPT-5-mini on TCMBench (agent 85.6% vs. RAG 77.7% vs. baseline 75.4%), suggesting that selective, autonomous tool invocation is more effective than fixed retrieval. Tool usage rates varied substantially across models (2–87%), with moderate usage (30–40%) associated with the most consistent gains. These findings provide empirical evidence on the potential and limitations of metadata-based tool augmentation for LLMs in the TAM domain.
키워드
agent systembenchmarkfunction callinglarge language modelmetadataretrieval-augmented generationtool augmentationtraditional Asian medicine
식별자
- Scopus EID
- 2-s2.0-105035504794
- WoS UID
- 001738498900001



로그인
연구물 목록으로