LLM-guided Semantic Guidance: A New Interpretable Text Classification Method Enabling Tsetlin Machines to Have BERT-level Comprehension

This article proposes an innovative semantic guidance framework that transfers LLM knowledge to the symbolic model Tsetlin Machine (TM), solving the dilemma where pre-trained language models (such as BERT) have strong semantic capabilities but lack interpretability, while symbolic models are interpretable but have weak semantic generalization. This framework achieves BERT-level text classification performance while maintaining full symbolization and efficiency, making it suitable for high-risk fields like healthcare and law, and providing a new paradigm for interpretable AI.

The field of natural language processing has long faced a trade-off: pre-trained models (like BERT) have strong semantics but are not interpretable, while symbolic models (like TM) are transparent and interpretable but have weak semantic generalization. High-risk fields (healthcare, law) require model decisions to be accurate and auditable, but traditional symbolic models struggle to capture semantic relationships.

Advantages of Tsetlin Machine: Clause-level transparency, full interpretability, and multi-task applicability; Limitations: Based on boolean bag-of-words representation, it is difficult to generalize across semantically related terms (e.g., having only learned "excellent" cannot associate with "outstanding").

Core Idea: Use LLM's semantic understanding to guide symbolic model learning, and remain independent of LLM during deployment. Steps: Sub-intent discovery (LLM decomposes categories into sub-intents), structured data generation (three-stage curriculum), semantic clue extraction (NTM learns high-confidence literals from synthetic samples), data augmentation (inject clues into real data).

Three-stage Curriculum:

1. Seed Stage: LLM generates domain-standard samples as anchors;
2. Core Stage: Generate samples with structural changes but stable vocabulary to help TM learn across syntax;
3. Enrichment Stage: Introduce synonyms/modifiers to expand vocabulary and promote semantic generalization.

Technical Implementation: Non-negated TM (NTM) extracts clues and injects them into the bag-of-words of real data; deployment does not require LLM or embedding layers, maintaining symbolic efficiency.

In multiple text classification tasks, this method improves accuracy and interpretability compared to the original TM, reaching performance equivalent to BERT. Key advantages:

• No runtime LLM calls, independent deployment;
• No embedding vectors, pure symbolic representation;
• Data-efficient, reducing the need for large-scale annotation;
• Strong domain adaptability, general prompt templates apply to any labeled dataset.

Medical Document Analysis: Interpretability allows doctors to understand diagnostic basis, and semantic guidance helps understand relationships between medical terms; Legal Document Review: The transparency of symbolic models meets the traceability of decisions, suitable for contract review/case retrieval; Financial Compliance Detection: High performance while providing clear decision-making basis, meeting regulatory interpretability requirements.

Current Limitations: Training relies on LLM-generated synthetic data; the quality of sub-intent discovery depends on prompt design; highly specialized fields require additional knowledge injection.

Future Directions: Automated prompt optimization; multi-language expansion; integration with other symbolic models; dynamic semantic updates (updating knowledge after deployment).

This framework successfully bridges the semantic capabilities of neural networks with the transparency and efficiency of symbolic models, providing an ideal solution for high-risk applications. It proves that high-performance text classification can be achieved without sacrificing interpretability, offering a reference for TM applications and neuro-symbolic integration research. In today's era where AI is deeply involved in key decision-making fields, this innovative architecture that balances performance and interpretability has important practical significance.