Verification of Neural Network Semantic Alignment: Cross-Combination Experiments of Encoder-Decoder Reveal Key Bottlenecks in Model Interoperability

By independently training two encoder-decoder models with the same architecture and conducting cross-splicing experiments, we quantitatively analyze the performance degradation caused by semantic misalignment in the latent space, proving the necessity of semantic consistency between model components for system integration.

• Original Author/Maintainer: qinyishcn
• Source Platform: GitHub
• Original Title: semantic_alignment
• Original Link: https://github.com/qinyishcn/semantic_alignment
• Publication Date: 2026-05-25

In the engineering practice of deep learning systems, a long-neglected but crucial question is: when we combine neural network components from different sources and training processes, can they collaborate seamlessly? This issue is particularly prominent in scenarios such as modular AI systems, Model-as-a-Service (MaaS), and federated learning.

The traditional view holds that as long as the model architectures are the same, components should be interchangeable. However, through carefully designed controlled experiments, this study uses data to prove a counterintuitive conclusion: even with identical architectures, independently trained encoders and decoders cannot be directly interchanged, because there is a problem of semantic misalignment in their latent space representations.

The research is based on the following key hypothesis: the low-dimensional bottleneck layer of a neural network is not a universal semantic interface; its specific meaning is jointly determined by random initialization, optimization path, and data order during training. If two models are trained independently, even with the same architecture, the latent space representations they learn may have systematic differences.

The experiment uses a classic autoencoder classifier architecture, with the MNIST handwritten digit dataset as the benchmark:

Encoder Structure

• Input layer: 784 dimensions (28×28 image flattened)
• Hidden layer 1: 256 dimensions, ReLU activation, Dropout(0.2)
• Hidden layer 2: 128 dimensions, ReLU activation, Dropout(0.2)
• Bottleneck layer: 32 dimensions, ReLU activation (latent space)

Decoder Structure

• Input layer: 32 dimensions (receives latent space vector)
• Hidden layer 1: 128 dimensions, ReLU activation, Dropout(0.2)
• Hidden layer 2: 256 dimensions, ReLU activation, Dropout(0.2)
• Output layer: 10 dimensions (corresponding to classification of digits 0-9)

1. Independent Training Phase: Train Model A and Model B separately using the same network architecture but different random seeds
2. Baseline Performance Evaluation: Record the classification accuracy of each model on the test set (expected to reach 97-98%)
3. Cross-Splicing Test:
   • Combination 1: Encoder of Model A + Decoder of Model B
   • Combination 2: Encoder of Model B + Decoder of Model A
4. Performance Difference Analysis: Quantify the accuracy loss caused by model mismatch
5. Latent Space Analysis: Compare the statistical feature distributions of the latent spaces of the two models

The experimental results are shocking: when the encoder and decoder come from different training processes, the system performance drops off a cliff.

Model Combination	Expected Accuracy	Performance Status
Original Model A	97-98%	Normal
Original Model B	97-98%	Normal
Encoder A + Decoder B	10-30%	Severe Degradation
Encoder B + Decoder A	10-30%	Severe Degradation

This result clearly shows that semantic misalignment in the latent space leads to almost complete system failure. The decoder of Model B expects to receive a specific latent space representation (z_B) that co-evolved with its encoder during training, while the encoder of Model A produces a completely different representation (z_A). The semantic gap between the two makes it impossible for the decoder to correctly interpret the input.

Further latent space visualization analysis reveals significant differences in the distribution of the two models in the 32-dimensional latent space. These differences are reflected in:

• Different clustering structures: Samples of the same digit category have different cluster centers and distribution ranges in the latent space
• Differences in manifold geometry: The low-dimensional manifolds formed by data in the latent space have different curvatures and topological structures
• Uneven variance distribution: There are systematic deviations in the importance ranking of each dimension and the variance contribution rate

These statistical differences explain why simple component splicing cannot achieve the expected function—the latent space vector faced by the decoder does not match the mapping relationship it learned during training at all.