TAG-Head: A Lightweight Graph Neural Network Head for Fine-Grained Action Recognition Using Only RGB Videos

A paper accepted by ICPR 2026 introduces TAG-Head, a plug-and-play spatiotemporal graph head module. It upgrades standard 3D backbone networks into powerful tools for fine-grained action recognition without additional modalities, outperforming multimodal methods on multiple benchmarks. This module is lightweight and efficient, seamlessly integrable into mainstream architectures like SlowFast and R(2+1)D-34, providing a new solution for fine-grained action recognition.

Fine-Grained Human Action Recognition (FHAR) requires distinguishing visually similar actions (e.g., gymnastics flips, diving twists) and relies on subtle spatiotemporal cues. Traditional solutions depend on multimodal information (posture, optical flow, text) to improve accuracy but face issues like high additional annotation costs, complex computation, and bloated systems.

TAG-Head is a lightweight spatiotemporal graph head that can be plug-and-play integrated into 3D backbone networks. Its two-stage process:

1. Transformer Global Encoding: Uses learnable 3D positional encoding to capture long-range spatiotemporal dependencies, laying the foundation for global context;
2. Graph Neural Network Refinement: Includes two edge types—intra-frame fully connected edges (to distinguish subtle appearance differences) and time-aligned edges (to stabilize motion cues without over-smoothing).

TAG-Head has multiple advantages:

• High parameter efficiency: Introduces minimal parameters and computational overhead, suitable for resource-constrained environments;
• Plug-and-play: Seamlessly integrates into mainstream 3D backbones without modifying the original structure;
• End-to-end training: Trains together with the backbone network, simplifying the process;
• Low latency: Minimal additional overhead, meeting real-time application requirements.

On FineGym (Gym99/Gym288) and HAA500 datasets, TAG-Head achieves state-of-the-art performance among RGB-only models and outperforms many multimodal methods. Ablation experiments show that the Transformer encoder provides global context, and the combination of the two edge types is crucial for performance improvement—using either type alone cannot achieve the effect of the complete model.

TAG-Head relies only on RGB videos and can be applied to:

• Sports analysis: Assisting referee scoring and athlete technique analysis;
• Fitness guidance: Real-time action quality analysis and personalized recommendations;
• Human-computer interaction: Natural interaction in VR/AR;
• Video surveillance: Enhancing the intelligence level of security systems. No additional sensors are needed, reducing deployment barriers.

TAG-Head improves the performance of RGB-only fine-grained recognition without additional modalities through a lightweight spatiotemporal graph head module, outperforming multimodal competitors. Its design principles (balance between global and local, spatiotemporal fusion) provide new ideas for video understanding. The research team commits to releasing the code on GitHub to promote technology dissemination and application.