Superposition in Embeddings: An Interactive Demo

Explore how neural networks can represent more features than their explicit dimensions through superposition. In this simplified model, we have several distinct "features" (colors) that are projected into a limited "embedding space" (two dimensions). Observe how features are combined and the resulting "reconstruction error" when trying to recover the original features from the compressed representation.

Features

Embedding Space (2 Dimensions)

This is the compressed representation where active features are combined.

Dim 1
Dimension 1
Dim 2
Dimension 2

Reconstructed Features

The model attempts to reconstruct original features from the embedding space. A higher error indicates more interference or loss of information.

Global Controls & Metrics

Total Reconstruction Loss: 0.00