Explore how neural networks can represent more features than their explicit dimensions through superposition. In this simplified model, we have several distinct "features" (colors) that are projected into a limited "embedding space" (two dimensions). Observe how features are combined and the resulting "reconstruction error" when trying to recover the original features from the compressed representation.
This is the compressed representation where active features are combined.
The model attempts to reconstruct original features from the embedding space. A higher error indicates more interference or loss of information.