We show that models trained independently in different modalities (e.g., image and text models) are similar enough that they can be coarsely aligned without pairs using a single, orthogonal map. This is challenging because the models have no shared coordinates or known correspondences and potentially have different numbers of samples and dimensions. Our method uses that coarse distances are often similar in both modalities.
The plot shows 1,000 images and 1,000 captions from two separate halves of MS COCO. At the start, the images (circles) and the captions (triangles) lie in their own spaces. The animation then applies the learned map, and the colours show that each image ends up near captions about the same kind of content.
Main findings
- Vision and language models can be aligned without pairs, even across different datasets.
- When few pairs (≤1000) are available, our method outperforms existing few-pair methods by a large margin.
- predicts how well the unpaired alignment works, also for scientific and medical modalities.
- Our generic method is competitive with or better than other domain-specific methods on their respective unpaired benchmarks.
Models trained independently in different modalities can often be coarsely aligned.
We take images from one half of the dataset and captions from the other half, so no image is paired with its original caption. We embed them with independently trained vision and language models. Our method then learns one global orthogonal map, a single matrix that moves every image embedding into the caption space.
Circles are image embeddings from DINOv2 B/14, and triangles are caption embeddings from Qwen3-8B. The two sets come from separate, non-overlapping subsets of MS COCO, and they start out in different spaces with different numbers of dimensions. There is no known correspondence between the dimensions of the two spaces or between the samples in the two sets. Hover over or tap any point to see its image or caption.
Geometric initialization
We cluster the images and the captions separately. We then match the image clusters to the caption clusters so that the similarities between cluster centers agree as well as possible, which is a small quadratic assignment problem. Repeating this matching on random batches and averaging the results gives a coarse correspondence between the two sets.
The plot shows 8 clusters. The method uses 30 clusters and 30 repetitions.
Read-out
We turn the coarse correspondence into a one-to-one matching between individual images and captions. The matching is one optimization step on the objective. then finds the one orthogonal matrix that best fits all matched pairs at once.
Validation after the read-out…
Refinement
We alternate between updating the correspondence between images and captions with and updating the map with . The map stays a single orthogonal matrix that is shared by all samples.
Validation after the refinement…
Our Wasserstein Procrustes method with geometric initialization finds one global map between image and caption embeddings without any pairs.
Explore 1,000 image and caption pairs from the MS COCO validation set, which the method never sees during training. Every point is moved by the same orthogonal map. Hover over or tap a point to see its three closest matches and its true image or caption. The label on the line gives the fraction of captions that are closer to the image than its true caption. is the average of this fraction, where 0 means perfect alignment and 0.5 means chance. The option MS COCO → SPC uses images from MS COCO and captions from SPC. The map shown is the median of 5 random seeds.
Image and text embeddings from independently trained models often share enough structure to be aligned coarsely without any image-caption pairs. The alignment works for many model combinations, and it works even when the images and captions come from different datasets. Its quality varies with the model combination and the caption style. For example, Qwen3 gen stays close to chance on the long, detailed captions of SPC, DCI, and DOCCI, and we analyze this difference further in the paper.
Image and text embeddings are similar enough for a coarse alignment without pairs, even across different datasets.
Use the slider to add known image-caption pairs and see how they change the alignment. In the plot, each known pair is an outlined image and its caption, joined by a line. The known pairs come from MS COCO, while the two embedding sets are still taken from separate, non-overlapping subsets of the dataset. Our method uses these pairs in each of its three steps. The plot shows the same unseen MS COCO validation set as before.
With 20 or fewer known pairs, our is 14 to 28 times lower than the best of seven methods that learn from pairs. Six of these methods need 100 to 500 pairs to reach the that our method reaches with no pairs at all, and SUE never reaches it.
Shared structure also improves the alignment when some pairs are available.
Generating Images From Captions
The Map Made Visible
We use image generation to make the alignment visible. We embed a caption with the MPNet language model, and our map moves this embedding into the image space of DINOv2 B/14. A small , which was trained only on ImageNet images and never on captions, then generates an image from the mapped embedding. The caption can therefore only influence the image through our map. The goal is to show what the map preserves, and not to compete with text-to-image models that are trained on millions of image-caption pairs.
Without any pairs, the generated images already show a reasonable scene, but they often miss fine details. Adding a few known pairs sharpens individual objects and scene attributes.
The slider goes through all 84 combinations of vision model, language model, and dataset, sorted by their . measures how similarly the two spaces arrange the same samples. It needs paired data, which we use only for this analysis. The thin lines connect 80 images to their true captions. For combinations with a low many lines are long, and for combinations with a high most lines are short. You can also click a point in the chart.
Unpaired alignment works best when the two spaces arrange the samples in a similar way. Across all model combinations, explains 73% of the variation in alignment quality ( = 0.73), with a of ρ = −0.90. So , measured on a small set of paired examples, can estimate whether unpaired alignment is likely to work well.
The between the two spaces predicts how well unpaired alignment works.
We ran the same method with the same settings on fourteen pairs of modalities from seven fields. Each point in the plot is one pair of modalities, and the gray dots are the image and caption combinations from the previous chapter.
Lower is better. The line is a least-squares fit, and the band is a bootstrap interval. = 0.69, = −0.82.
The relation between and alignment quality also holds for these other pairs of modalities.
On three benchmarks, methods were built specifically for the domain. We run the same method with the same settings and compare the mean over seeds.
Mean ± standard deviation over seeds. Lower is better, and the best value per benchmark is bold. NQ averages ten pairs of text encoders and NSD fMRI averages 56 pairs of subjects.
Our generic method is competitive with mini-vec2vec on NQ, and it has a lower mean than SCOT+ on PBMC and than Platonic Brain on NSD fMRI.