Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation

Improving specialized-domain LLMs typically requires costly human-labeled data or real-world feedback, which is often unavailable. This paper proposes Neuron-OPSD, a method that uses the model's own internal neuron activation patterns—rather than external labels—to select useful training data and build a "teacher" context for self-distillation via majority-vote pseudo-labels. Compared to existing annotation-free approaches, which suffer from either poor out-of-domain generalization or inflated calibration error, Neuron-OPSD improves in-domain performance while avoiding these pitfalls. This is especially valuable for specialized fields like medicine or law, where labeled training data is scarce or expensive to produce. Authors: Zhuowei Chen, Xiang Lorraine Li Paper: https://arxiv.org/abs/2607.02460v1