Distilling Privileged Knowledge From Transformers to Lightweight CNNs for On-Device Time Series Forecasting
- 문서 유형
- 연구출판물 · 학술지 논문
- 등재
- SCIEScopus
- 저널 순위
- JCR Q2Scopus Q1
- 저널
- IEEE ACCESS14 · pp.30373 · 2026
- 피인용
- Web of Science 0Scopus 0
- FWCI (Scopus)
- 0
- JCR IF
- JCR 2024 3.6
- ISSN
- 2169-3536
저자
초록
Time Series Forecasting (TSF) is a pivotal capability for critical applications in industrial automation, energy management, and smart infrastructure. While Large Language Models (LLMs) and Transformer-based architectures have set new benchmarks in forecasting accuracy, their exorbitant computational costs and memory requirements prohibit their deployment on resource-constrained Microcontroller Units (MCUs). Lightweight models offer an alternative but often fail to capture complex long-term dependencies. This paper introduces a comprehensive framework to address these interconnected challenges of accuracy and efficiency. We propose a highly efficient Convolutional Neural Network (CNN) architecture, enhanced through a novel heterogeneous knowledge distillation methodology. In our framework, a powerful LLM-based teacher model, utilizing self-attention mechanisms, guides the training of a compact student model that operates solely on lightweight convolutions during inference. This approach is augmented by the Gated Dilated depthwise separable Convolution (GDC) block, which efficiently captures long-range dependencies and multi-scale patterns while drastically minimizing parameter count to meet strict MCU constraints. The robustness and generalizability of our framework are validated through extensive experiments on six long-term and five short-term forecasting benchmarks. Critically, our framework establishes a new state-of-the-art among lightweight architectures, demonstrating consistent performance gains across diverse forecasting horizons. Experimental results confirm that our method reduces Mean Squared Error (MSE) by 11.5% in long-term and 12.4% in short-term forecasting tasks compared to massive LLM-based baselines, all while utilizing less than 0.01% of the parameters. The final, optimized model is successfully deployed on a commercial ESP32-S3 MCU, demonstrating its practical viability for real-time, accurate, and efficient forecasting on edge devices with an inference latency of just 256 ms.
키워드
edge AIheterogeneous knowledge distillationknowledge distillationlightweight convolutional neural networksmodel compressionTime series forecastingTinyML
식별자
- Scopus EID
- 2-s2.0-105031525377
- WoS UID
- 001706374500042



로그인
연구물 목록으로