Cross-Lingual Alignment

Train a word-embedding model on English and another on Spanish, and you get two spaces that encode the same concepts in completely different coordinates — yet with nearly the same shape. The geometry of "king is to queen as man is to woman" survives translation. Cross-lingual alignment exploits that near-isometry: find one transformation that lines the spaces up, and you get translation, cross-lingual search, and zero-shot transfer for free.

Interactive Alignment Explorer

The explorer below is real: the source words are genuine English GloVe vectors, and pressing Apply Procrustes learns the orthogonal map from your chosen anchor pairs and rotates the source onto the target. Watch the held-out translation accuracy jump from zero — and watch it refuse to reach 100% for the distant pair, no matter how big the dictionary.

Why two languages line up at all

A monolingual embedding space has no canonical orientation — rotate every vector by the same matrix and all distances, angles, and analogies are unchanged. Two independently-trained spaces therefore differ by (approximately) a rotation. If that were exactly true the spaces would be isometric, and a single orthogonal matrix would map one onto the other perfectly. Real languages are only approximately isometric, and the gap grows with linguistic distance — which is precisely what makes some pairs easy and others hard.

Procrustes: the supervised baseline

Given a seed dictionary of matched pairs — source rows X, target rows Y — find the orthogonal map that best aligns them:

The constraint W^\top W = I keeps the map a pure rotation/reflection — no stretching, so neither space is distorted. Remarkably, this has a closed form: take the SVD of the cross-covariance X^\top Y and discard the singular values.

def procrustes_align(X_src, X_tgt):
    """Closed-form orthogonal map W (WᵀW = I) sending source → target."""
    X = X_src - X_src.mean(0)
    Y = X_tgt - X_tgt.mean(0)
    U, _, Vt = np.linalg.svd(X.T @ Y)   # SVD of the cross-covariance
    W = U @ Vt
    return W                            # aligned_source = X_src @ W

One SVD over a few hundred anchor pairs is enough to align two related languages — exactly what the explorer shows when two anchors already snap Spanish onto English.

Aligning without a dictionary

A seed dictionary is a luxury for low-resource languages. Two families remove it:

VecMap bootstraps from the spaces' own structure: induce a dictionary from the current mapping, re-solve Procrustes, repeat. Self-learning converges from a tiny or even numeral-only seed.

def vecmap_refine(W, X, Y, n_iter=10):
    """Unsupervised self-learning — no human dictionary required."""
    for _ in range(n_iter):
        pairs = mutual_nearest_neighbors(X @ W, Y)  # induce dictionary
        W = procrustes_align(X[pairs.src], Y[pairs.tgt])  # re-solve
    return W

MUSE aligns adversarially: a discriminator tries to tell mapped-source from target, and the mapping W is trained to fool it. When the discriminator can no longer separate the languages, the spaces are aligned.

class Aligner(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.W = nn.Linear(d, d, bias=False)            # the mapping
        self.D = nn.Sequential(nn.Linear(d, 128),
                               nn.ReLU(), nn.Linear(128, 1))

    def losses(self, x, y):
        d_loss = bce(self.D(self.W(x)), 0) + bce(self.D(y), 1)
        w_loss = bce(self.D(self.W(x)), 1)              # fool D
        return d_loss, w_loss

Supervised or unsupervised?

Measuring alignment

The standard yardstick is Bilingual Lexicon Induction — translate each source word by its nearest target neighbor after mapping, and score precision@k on a held-out set (never the anchors):

def bli_precision_at_k(X_src, Y_tgt, gold, W, k=1):
    """Held-out word-translation accuracy after alignment."""
    sims = cosine_similarity(X_src @ W, Y_tgt)
    topk = np.argsort(-sims, axis=1)[:, :k]
    return np.mean([g in topk[i] for i, g in enumerate(gold)])

This is the number the explorer reports live — and it is scored on words the dictionary never saw, so it measures genuine generalization, not memorization.

Where it breaks

The orthogonal-map assumption is a modelling choice, and it has a failure mode: typologically distant pairs. English↔Spanish are close enough that one rotation aligns them; English↔Chinese differ by more than a rotation (script, morphology, structure), so orthogonal Procrustes leaves a residual gap and BLI plateaus well below the near pairs — exactly the cap you hit in the explorer.

When mapping-based alignment is not enough, jointly-trained multilingual models (mBERT, XLM-R) sidestep it: train one transformer on dozens of languages with a shared sub-word vocabulary, and a common space emerges during pre-training — no post-hoc map required. The cost is that you must train the shared model from the start.

Applications

• Zero-shot transfer — fine-tune a task model in English, run it on a language it never saw labels for.
• Cross-lingual retrieval — map every document into one space and search across languages with a single index.
• Low-resource MT — bootstrap translation from comparable (not parallel) corpora.

Best practices

• Anchor on the unambiguous. Named entities, numerals, and cognates make the most reliable seed pairs.
• Iterate. Start with a small dictionary, induce a larger one, re-solve — VecMap-style refinement compounds.
• Pivot through a hub. For a low-resource pair, align both languages to English rather than to each other.
• Sanity-check symmetry. A good alignment should translate well in both directions; large asymmetry signals a degenerate map.

Conclusion

Cross-lingual alignment turns a hard problem — making models work across languages — into a clean geometric one: find the rotation that lines two spaces up. A single SVD handles related languages from a few hundred anchor pairs; unsupervised methods drop the dictionary; and where the isometry assumption breaks, jointly-trained multilingual models take over. The explorer above makes the core fact tangible — separately-trained languages really are the same shape, just spun apart.

Related concepts