Correlation dimension of natural language in a statistical manifold

Letter
Open Access

Correlation dimension of natural language in a statistical manifold

Xin Du and Kumiko Tanaka-Ishii

Phys. Rev. Research 6, L022028 – Published 2 May 2024

Abstract

The correlation dimension of natural language is measured by applying the Grassberger-Procaccia algorithm to high-dimensional sequences produced by a large-scale language model. This method, previously studied only in a Euclidean space, is reformulated in a statistical manifold via the Fisher-Rao distance. Language exhibits a multifractal, with global self-similarity and a universal dimension around 6.5, which is smaller than those of simple discrete random sequences and larger than that of a Barabási-Albert process. Long memory is the key to producing self-similarity. Our method is applicable to any probabilistic model of real-world discrete sequences, and we show an application to music data.

Received 25 July 2023
Revised 24 January 2024
Accepted 9 April 2024

DOI:https://doi.org/10.1103/PhysRevResearch.6.L022028

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Physics Subject Headings (PhySH)

Language models

Artificial neural networks Fractal analysis Fractal dimension characterization Information theory Non-Markovian processes Phase space dynamics Time series analysis

Statistical Physics & Thermodynamics

Authors & Affiliations

Xin Du ^1,* and Kumiko Tanaka-Ishii ^2,†

¹Waseda Research Institute for Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan
²Department of Computer Science and Engineering, School of Fundamental Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan