본문 바로가기

Distilling Privileged Knowledge From Transformers to Lightweight CNNs for On-Device Time Series Forecasting

문서 유형
연구출판물 · 학술지 논문
등재
SCIEScopus
저널 순위
JCR Q2Scopus Q1
저널
IEEE ACCESS14 · pp.30373 · 2026
피인용
Web of Science 0Scopus 0
FWCI (Scopus)
0
JCR IF
JCR 2024 3.6
ISSN
2169-3536

저자

  1. 나상진제1저자Pusan Natl Univ, Sch Comp Sci & Engn, Pusan 46241, South Korea
  2. 조용훈Pusan Natl Univ, Sch Comp Sci & Engn, Pusan 46241, South Korea
  3. 백윤주교신저자Pusan Natl Univ, Sch Comp Sci & Engn, Pusan 46241, South Korea

초록

Time Series Forecasting (TSF) is a pivotal capability for critical applications in industrial automation, energy management, and smart infrastructure. While Large Language Models (LLMs) and Transformer-based architectures have set new benchmarks in forecasting accuracy, their exorbitant computational costs and memory requirements prohibit their deployment on resource-constrained Microcontroller Units (MCUs). Lightweight models offer an alternative but often fail to capture complex long-term dependencies. This paper introduces a comprehensive framework to address these interconnected challenges of accuracy and efficiency. We propose a highly efficient Convolutional Neural Network (CNN) architecture, enhanced through a novel heterogeneous knowledge distillation methodology. In our framework, a powerful LLM-based teacher model, utilizing self-attention mechanisms, guides the training of a compact student model that operates solely on lightweight convolutions during inference. This approach is augmented by the Gated Dilated depthwise separable Convolution (GDC) block, which efficiently captures long-range dependencies and multi-scale patterns while drastically minimizing parameter count to meet strict MCU constraints. The robustness and generalizability of our framework are validated through extensive experiments on six long-term and five short-term forecasting benchmarks. Critically, our framework establishes a new state-of-the-art among lightweight architectures, demonstrating consistent performance gains across diverse forecasting horizons. Experimental results confirm that our method reduces Mean Squared Error (MSE) by 11.5% in long-term and 12.4% in short-term forecasting tasks compared to massive LLM-based baselines, all while utilizing less than 0.01% of the parameters. The final, optimized model is successfully deployed on a commercial ESP32-S3 MCU, demonstrating its practical viability for real-time, accurate, and efficient forecasting on edge devices with an inference latency of just 256 ms.

키워드

edge AIheterogeneous knowledge distillationknowledge distillationlightweight convolutional neural networksmodel compressionTime series forecastingTinyML

식별자

Scopus EID
2-s2.0-105031525377
WoS UID
001706374500042