Recursive functions
Computation through composition, recursion, and unbounded minimization. It foregrounds which numerical functions can be built from simpler ones.
Exploratory 3D NCD atlas
Programming-Language Atlas
Explore a three-dimensional ordinal map of compression-transfer distance: an LZMA-based approximation to normalized information distance across 200 sampled source-code corpora.
200 corpora LZMA NCD 4 repository-disjoint folds 64 KiB per object
Choose a corpus to inspect its full-space density and nearest neighbours.
This is an ordinal display of the full normalized compression distance matrix. Larger points have larger five-neighbour NCD radius; rings identify the newest cohort of 25. Screen-space whitespace is not direct evidence of an information gap.
Method
From universal distance to this atlas
The theory begins with the shortest effective description of an object and asks how much information is needed to transform either member of a pair into the other. Normalization makes that distance relative to the information contained in the larger object.
Information
Kolmogorov complexity K(x) is the length of the shortest self-delimiting program that produces x. Conditional complexity K(x|y) is the shortest such program when y is already available. Up to a logarithmic term, information distance is the larger transformation cost:
E(x,y) = max(K(x|y), K(y|x))
Normalize
Dividing by the larger object's complexity distinguishes a small change to a large object from the same absolute change to a small one. NID lies between zero and one and is a metric up to negligible discrepancies:
NID(x,y) = max(K(x|y), K(y|x)) / max(K(x), K(y))
Using symmetry of information, its numerator can be rewritten, up to logarithmic terms, as K(xy) - min(K(x), K(y)). This removes the conditional terms and opens the way to compression.
Universal
NID is universal in a precise theoretical sense: among effective normalized distances satisfying a density requirement, it captures the dominant computable similarity between each pair. This theorem belongs to the uncomputable NID. A distance built from a particular real compressor is only a practical approximation and inherits that compressor's field of view.
NCD
Because K cannot be computed, normalized compression distance substitutes the compressed length C. LZMA is order-sensitive, so this experiment conservatively takes the larger concatenation length:
NCD_C(x,y) = [max(C(xy), C(yx)) - min(C(x), C(y))] / max(C(x), C(y))
LZMA
LZMA does not parse or execute code. It reads bytes in sequence through a finite sliding dictionary, replacing repeated substrings with backward references. It also adapts probabilities for literals, match lengths, and match distances, then range-codes more expected events with fewer bits.
- MatchFind an exact byte sequence already present within the dictionary.
- PredictEstimate the next literal or match event from recent byte-level context.
- CodeSpend fewer bits on events that the evolving model considers more probable.
Regularity
The compressor has direct access to repeated identifiers, keywords, punctuation, indentation, boilerplate, and recurring byte contexts. Grammar and common idioms can affect those patterns, so LZMA may detect their textual consequences indirectly. It has no direct representation of an abstract syntax tree, a type rule, an evaluation strategy, or whether two differently written programs compute the same result.
- Direct
- Exact substrings, local byte statistics, repetition within the finite dictionary.
- Indirect
- Syntax and idioms only where they repeatedly constrain the emitted source text.
- Absent
- Parsing, execution, semantic equivalence, runtime behaviour, and intent.
Objects
For this demonstration, each language label contributes four repository-disjoint source objects of exactly 64 KiB. The matrix averages all cross-fold NCD comparisons. Labels are GitHub Linguist programming types represented in a frozen Stack v2 revision. The objects supplied to the theory are therefore prepared source corpora.
- Equalized
- UTF-8 byte length, fold count, repository disjointness, and the maximum contribution from one repository.
- Not equalized
- Files, functions, programs, tokens, statement count, unit length, or the number of independent semantic examples inside 64 KiB.
- Inherited
- Stack coverage and Linguist classification, including possible extension collisions, generated forms, and heterogeneous repository contexts.
Cohorts
The first 100 corpora are frozen. Remaining programming-type labels were ordered by deduplicated Stack corpus bytes, then file count, before extension NCD was measured; the first 100 passing the same corpus gate form four groups of 25. Moving the slider takes cumulative induced submatrices at 100, 125, 150, 175, and 200 labels. The groups are release batches, not clusters.
Map
Each cumulative NCD matrix receives its own three-dimensional ordinal embedding. Later stages are aligned to the shared first 100 anchors, removing arbitrary rotation, reflection, translation, and global scale. Earlier points may still move because adding corpora changes a global fit. Thin lines are the minimum spanning tree of the raw stage matrix.
Projection
This display preserves much, but not all, of the full NCD ordering. NCD remains compressor-dependent, and screen-space whitespace is not direct evidence of an information gap.
The object
A language has more than one description
Information distance is defined between discrete objects. The decisive experimental move is therefore choosing what string will stand for a programming language. Source code is one such description, but not the only one.
| Object | What it encodes | Distance asks |
|---|---|---|
| Source corpus | Notation, idioms, libraries, domains, and author practice | This atlasWhich corpora transfer useful compression regularities? |
| Tokens or trees | Lexical and grammatical structure after some surface variation is removed | SyntacticWhich languages generate similarly structured program forms? |
| Specification | Grammar, binding, types, evaluation, effects, and memory rules | DefinitionalHow compactly can one formal language definition describe another? |
| Behavioural suite | Observable execution across matched programs and tasks | OperationalWhich languages produce related behaviours under controlled probes? |
| Latent distribution | A trained model's representation of many code fragments | LearnedWhich corpora occupy similar regions under that model? |
Equal length is necessary, not sufficient
NCD is sensitive to object length, so exact 64 KiB folds remove a basic size asymmetry. They do not make those folds compositionally equivalent. Terse languages can place many small units into the same byte budget; verbose languages fewer large ones; some paradigms do not have a language-neutral analogue of a function at all.
| Corpus label | Raw files | Raw repositories | Minimum repositories in a 64 KiB fold |
|---|---|---|---|
| Brainfuck | 3,360 | 1,777 | 95 |
| PostScript | 414 | 290 | 25 |
| Solidity | 1,656 | 503 | 48 |
The table shows substantial support for every displayed fold, but also very different sampling regimes. It does not certify that every detected file is a canonical example of the named language. That requires a separate label-purity audit.
What a stronger standardization would compare
No single normalization should replace the byte-controlled view. A stronger design would retain it and add parallel objects: fixed token budgets; matched distributions of whole-program or parser-defined top-level-unit lengths; normalized token and syntax-tree views; and task-aligned implementations of the same behaviours. “Function length” can be one stratum where the language supplies functions, but it cannot be the universal unit for Brainfuck programs, PostScript procedures, SQL queries, assembly regions, and proof terms alike.
The atlas as a demonstration
The map makes the NID-to-NCD move visible. A point is not located by a chosen list of language features. Its pairwise position emerges from the extra compressed description length of joining corpus objects. Nearness therefore means compression transfer under this representation and compressor.
Towards a geometry of language structure
- SurfaceRepeat NCD after comments, identifiers, and literals are separately controlled.
- SyntaxConstruct comparable token, parse-tree, grammar, and specification objects.
- SemanticsAdd behavioural probes and compare distributions of code embeddings as their own distance instrument.
- Common structureAsk which geometry predicts held-out representations, and which relationships remain specific to one description.
Theoretical lineage
Li, Chen, Li, Ma, and Vitányi, “The Similarity Metric” develops NID and its universality; Cilibrasi and Vitányi, “Clustering by Compression” develops NCD as the practical analogue; Vitányi, Balbach, Cilibrasi, and Li, “Normalized Information Distance” gives the broader derivation and applications.
Prediction
A compressor is also a model of what comes next
Prediction and lossless compression are two views of the same coding problem. A sequential model that assigns probability p to the next symbol implies an ideal code cost of -log₂ p bits. Better prediction yields a shorter description.
01Predict
Estimate the next symbol from the available context.
02Code
Turn those probabilities into a lossless bitstream.
03Transfer
Ask how much context from x reduces the code length of y.
04Distance
Normalize that shared descriptive advantage across the pair.
From unsupervised learning to NCD
In his 2023 talk “An Observation on Generalization”, Ilya Sutskever develops a view of unsupervised learning through compression: structure learned from one body of data is useful when it shortens the description of another. The connection is older than neural language models, but it makes the relevance to this atlas vivid. LZMA is a small fixed predictor over byte events; a language model is a much richer learned predictor over a chosen symbolization. Both can be evaluated by code length. “Language Modeling Is Compression” demonstrates this prediction-compression correspondence directly for modern language models.
This does not mean that prediction necessarily recovers the unique causal or semantic structure of its data. It means that any regularity which improves calibrated prediction can reduce code length. The learned representation is judged by transfer, not by a claim that every internal coordinate has an intrinsic interpretation.
Invariance and the plurality of machines
The invariance theorem says that two universal description machines differ in Kolmogorov complexity by at most an additive constant fixed by the machines, not by the object. This gives K its asymptotic machine independence. It does not say that finite objects compressed by LZMA, PPMd, or a neural predictor will have the same practical geometry. At this scale, model class, context, tokenization, and implementation are visible parts of the measurement.
Turing machines
A finite controller reads and rewrites symbols on an unbounded tape. It foregrounds local sequential operations and explicit storage.
KU machines
Kolmogorov-Uspensky machines transform a finite, locally navigated graph rather than a linear tape. They foreground evolving relational storage.
λ-calculus
Computation proceeds through abstraction, application, substitution, and reduction. It foregrounds functions and binding rather than machine state.
These classical formalisms agree on computability, but they make different organization cheap, local, or explicit. That is the opening for a computational analogue of linguistic relativity: not that a programming language changes what is computable, but that its primitives and notation alter what is natural to express, inspect, predict, and reuse. Kenneth Iverson's “Notation as a Tool of Thought” is a more precise precedent than a strong Sapir-Whorf claim.
Two geometries
Compression space and embedding space answer different questions
An embedding does bring discrete code fragments into a continuous coordinate system. But distance there is not NCD, and it is not the continuity of the loss landscape.
A / code length
Compression distance
Dcode(x,y) ≈ normalized excess description length
Measures how much regularity transfers when a particular coder describes the pair. In this atlas, that coder sees finite byte histories and exact substring recurrence.
B / representation
Embedding distance
Dembed(X,Y) = distance between E(X) and E(Y)
Measures separation under a trained encoder's learned features, tokenizer, pooling rule, and objective. Cosine compares direction; Euclidean distance also retains scale. A corpus mean erases multimodality, so distributions of chunk embeddings should be compared directly.
Not the loss landscape
The loss landscape is a function over model parameters, θ ↦ L(θ). Embedding space contains activations E(x) produced by one fixed θ. Its apparent continuity comes from the network's continuous operations, while its inputs remain discrete token sequences and its coordinates remain conditional on the trained model. Nearby vectors mean the model has learned to treat the inputs similarly for its objective; they do not by themselves establish semantic or computational equivalence.
Why appending embeddings is not the answer
If E is fixed and computable, then E(x) can be generated from x. In algorithmic terms, appending it adds at most the constant description of E:
K(x, E(x)) = K(x) + O(1)
For a practical compressor, serialized floating-point vectors may change the result, but mainly because quantization, precision, coordinate order, and model-specific numerical patterns have been exposed as extra bytes. Ordinary LZMA cannot reliably interpret geometric nearness between floats. If the decoder does not already know the encoder, its tokenizer and weights must also be encoded or declared as shared side information.
Tokenization is therefore not fatal, but it must be explicit. A reversible tokenizer followed by an exact probabilistic coder remains lossless; code lengths should be reported in bits per original byte. A lossy normalization or pooled embedding defines a different object.
Proposed experiment
Do both, but do not collapse them prematurely
- Byte NCDRetain the present LZMA matrix as the transparent surface-regularity instrument.
- Neural code lengthUse a fixed code model as a predictor and measure symmetric normalized cross-corpus excess code length, including exact tokenizer accounting.
- Latent distributionsEmbed repository-disjoint chunks and compare language distributions with energy distance, sliced Wasserstein distance, or MMD rather than compressing raw floats.
- Held-out synthesisFit a common geometry on some instruments and require it to predict withheld distance matrices. Inspect the residuals instead of choosing arbitrary weights for a combined score.
Demonstration
Information distance made visible
The cumulative LZMA NCD panel strongly distinguishes repeated samples of the same corpus label from samples of different labels.
The result shows that a simple compression-based realization of information distance extracts a strong global organization without being given a language taxonomy. It also shows why NCD is a family of empirical distances rather than NID itself: LZMA missed one synthetic Markov-relation check, while the PPMd alternative badly violated self-identity. The representation and compressor are part of the scientific result.