본문 바로가기

Benchmark Evaluation of a Tool-Augmented Large Language Model Agent Using Traditional Asian Medicine Metadata

문서 유형
연구출판물 · 학술지 논문
등재
SCIEScopus
저널 순위
JCR Q2Scopus Q1
저널
APPLIED SCIENCES-BASEL16(7) · 2026
피인용
Web of Science 0Scopus 0
FWCI (Scopus)
0
JCR IF
JCR 2024 2.5
ISSN
2076-3417

저자

  1. Lee, Won-Yung제1저자Wonkwang Univ, Coll Korean Med, Iksan 54538, South Korea
  2. 김지환Pusan Natl Univ, Sch Korean Med, Dept Sasang Constitut Med, Div Clin Med, Pusan 46241, South Korea
  3. Leem, JungtaeWonkwang Univ, Res Ctr Tradit Korean Med, Iksan 54538, South Korea
  4. Lee, Byung-WookDongguk Univ, Coll Korean Med, Gyeongju 38066, South Korea
  5. Lee, Seungho교신저자Woosuk Univ, Coll Korean Med, Jeonju 54987, South Korea
  6. Kim, Young Woo교신저자Dongguk Univ, Coll Korean Med, Gyeongju 38066, South Korea

초록

This study evaluated a tool-augmented large language model (LLM) agent system utilizing a traditional Asian medicine (TAM) metadata. A structured information comprising 4780 entries across five entity types (herbs, syndromes, TCM symptoms, modern medicine symptoms, and acupoints) was constructed. Four LLMs (GPT-5.2, GPT-5-mini, Claude Sonnet 4.6, and Claude Haiku 4.5) were evaluated under baseline, non-agentic retrieval (RAG), and tool-augmented (agent) conditions using two public benchmarks: TCMBench (1300 items) and TCMEval-SDT (600 items). McNemar’s test and bootstrap confidence intervals were applied to examine performance differences. Tool augmentation showed a nominally significant improvement for GPT-5-mini on TCMBench term-related items (+4.5 percentage points on paired items, McNemar p = 0.034, 95% CI [+0.9, +8.5] pp), while Claude Sonnet 4.6 showed a nominally significant decline on SDT pathogenesis (−0.038, bootstrap p = 0.006). The agent condition outperformed the non-agentic RAG baseline for GPT-5-mini on TCMBench (agent 85.6% vs. RAG 77.7% vs. baseline 75.4%), suggesting that selective, autonomous tool invocation is more effective than fixed retrieval. Tool usage rates varied substantially across models (2–87%), with moderate usage (30–40%) associated with the most consistent gains. These findings provide empirical evidence on the potential and limitations of metadata-based tool augmentation for LLMs in the TAM domain.

키워드

agent systembenchmarkfunction callinglarge language modelmetadataretrieval-augmented generationtool augmentationtraditional Asian medicine

식별자

Scopus EID
2-s2.0-105035504794
WoS UID
001738498900001