Shared Geometry
as a Rosetta Stone

Cross-Modal Alignment Without Paired Data

1TU Munich 2MCML 3Ulm University 4ETH Zurich 5MIT †Equal advising

caption wordsimages

FOSCTTM

FOSCTTM stands for the fraction of samples closer than the true match. Take an image and its true caption. Count how many of the other captions are closer to the image than the true one, and divide by the number of captions. Averaged over all images, this gives one number between 0 and 1.

A value of 0 means the true caption is always the nearest. A value of 0.5 is what random guessing gives. We report it on the 40,504 pairs of the COCO validation set, which the method never sees during training.

CKA

Centered kernel alignment compares two embeddings of the same samples by the pairwise similarities within each space. It does not need the spaces to share dimensions or axes, and it does not change under orthogonal transformations or a uniform scaling of either space. We compute it on paired data only as an analysis tool. The method itself never uses pairs.

A value of 1 means both spaces arrange the samples in the same way, up to an orthogonal transformation and a scale. A value of 0 means the two arrangements are unrelated.

Gromov-Wasserstein

Gromov-Wasserstein matching compares two sets of points that lie in different spaces. It looks for a matching between the two sets under which the distances between points in one set agree as well as possible with the distances between the matched points in the other set.

Because it only compares distances within each set, it needs no shared coordinates. Our read-out takes one step of this optimization, which turns the coarse correspondence of the initialization into a one-to-one matching.

Orthogonal Procrustes

Orthogonal Procrustes finds the orthogonal map that moves one set of points closest to a matched second set, measured by the sum of squared distances. An orthogonal map keeps all distances and angles between points, so it can only rotate and reflect the space.

The best map has a closed-form solution from a singular value decomposition, so it is fast to compute.

Hungarian matching

The Hungarian algorithm finds the one-to-one matching between two sets with the highest total score, given a score for every pair. We use the similarity between the mapped images and the captions as the score, and we run the algorithm on random batches of samples.

RAE diffusion model

A representation autoencoder (RAE) uses the frozen DINOv2 B/14 image encoder as its encoder, and a decoder turns the encoder's features back into an image. A diffusion model generates images in this feature space.

We condition the diffusion model on the mean DINOv2 embedding of an image, and we train it on ImageNet images only, so it never sees a caption. A caption can therefore only change the generated image through our map.

R²

R², the coefficient of determination, is the share of the variation in FOSCTTM that a straight line in CKA explains. A value of 1 means that all points lie on the line, and a value of 0 means that the line explains none of the variation.

Spearman correlation

The Spearman correlation ρ compares the ranks of two quantities instead of their values. It is 1 when a higher CKA always comes with a higher FOSCTTM, and it is −1 when a higher CKA always comes with a lower FOSCTTM.

A value close to −1 means that sorting the combinations by CKA almost sorts them by how well they align.

Paper, Code, and Citation

@article{schnaus2026rosetta,
  title   = {Shared Geometry as a Rosetta Stone: Cross-Modal Alignment Without Paired Data},
  author  = {Schnaus, Dominik and Dag{\`e}s, Thomas and Cremers, Daniel and Wang, Xi and Isola, Phillip},
  journal = {arXiv preprint arXiv:2610.09411},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.09411}
}