Lacuna Language Learning: Leveraging RNNs for Ranked Text Completion in Digitized Coptic Manuscripts

Coptic belongs to the Afro-Asiatic language family and is the latest stage of the Egyptian language, the longest continuously attested language on Earth. Coptic utilizes the 24 glyphs of the Greek alphabet and adds additional Demotic (Egyptian) glyphs (a minimum of 6 depending on dialect) to represent sounds not found in Greek. In late antiquity, more than a dozen regional dialects were spoken and written Layton (2011). Owing to these dialect variations, the use of superlinear strokes and other diacritical marks, and irregular orthography of Greek loan words, Coptic provides a highly complex dataset.

Coptic manuscripts preserve the diverse textual tradition of late-antique and medieval Egypt. Inscribed on papyrus and other perishable media, many Coptic manuscripts contain small gaps or holes (lacunae), which often cannot be restored on the basis of other extant manuscripts. Scholars use qualitative methods to restore lacunae, chiefly through study of the manuscript’s context and (con)textual parallels. Occasionally, appeal is made to traditional canons of textual criticism, but here too the scholar’s own judgment guides the restoration Wasserman (2013). Initial testing has shown that human methods of textual restoration have a high error rate at both the word level and the character level (Sommerschield et al., 2023, 711–712).

Following early attempts using n-gram models to approach the Indus Valley script Rao et al. (2009), most previous work on reconstructing lacunae in manuscripts, as well as in epigraphic data, has focused on Greek and Latin Novokhatko and Maier (2022); Matsumoto (2022). Early projects included eAQUA Schubert (2011), which pioneered proposing automatic reconstructions of lacunae based on statistical methods from larger datasets (in the context of ancient languages). More recently, studies using neural methods for the reconstruction of Greek Assael et al. (2019) and Latin Brunello et al. (2023) have appeared, with papers in the last three years specifically proposing to leverage transformer based language model architectures for both born-digital and (OCRed) handwritten inputs in a range of languages Vogler et al. (2022).

We are not aware of previous papers applying language models to the reconstruction of Coptic texts, though a recent Web page prepared by the CoptOT project¹¹1https://coptot.manuscriptroom.com/manuscript-speculation-tool provides a ‘Manuscript Speculation Tool’ which helps in laying out missing letters on predefined digitized manuscript spaces. However, in the tool’s operating scenario, a base text to be laid out is known (e.g. a chapter of the Bible), and the question is how many letters of each verse might fit into each missing line or part of a line. To our knowledge, this paper is the first attempt to leverage language modeling for lacuna reconstruction in Coptic.

In 2019, Devlin et al. introduced BERT, a foundational masked language model (MLM), where random tokens in the input were masked, and the model was trained to predict the masked token based on the context. For 15% of the tokens in training, each one is replaced with either [MASK], a random token, or the original token, without change. Masking mimics gaps and teaches the model to fill in missing segments of strings, which makes the MLM approach highly applicable to our lacuna reconstruction task.

In the same paper, Devlin et al. found that a model with only left to right context performed worse than a bidirectional masked language model, which is able to use context from before and after the masked token. They advocate for a bidirectional model that can use left and right context at every layer over concatenating a left to right model and a right to left model, as proposed earlier in ELMo Peters et al. (2018). As we are framing our lacuna reconstruction task as a prediction of masked characters, parallel to the masked token prediction done by models such as BERT, this finding regarding bidirectionality leads us to adopt a bidirectional strategy for our model as well.

As the masked language model strategy was popularized with transformer based models such as BERT, there is not much existing work regarding the implementation of masked language models with an RNN-based architecture. However, in scenarios with relatively small quantities of data and limited long distance dependencies, it can still be preferable to use an RNN-based architecture over a transformer-based architecture Mishra (2021). Considering that we have almost 1.22 million tokens of Coptic data, and we are looking to fill in character gaps at the sentence level, we consider our Coptic lacuna prediction task to be one such scenario, and we opt to use an RNN based architecture in our model.

While we have done some preliminary prototyping with transformer based architectures, such as ELECTRA Clark et al. (2020), so far our experiments with RNN-based architectures have made the most progress. As such, we present those finding here. However, we still believe it would be worthwhile to return to the exploration of various transformer architectures in future work.

We applied three rudimentary heuristic baselines to our three test data sets, the results of which are shown in the bottom half of Table 1. The first baseline selected a random character from the SentencePiece character model vocabulary for each character prediction. The second baseline always predicted the most common letter in the data set (mode character), "{coptic}e". The third baseline is a simple tri-gram language model. Results for "Test Random" and "Test Smart" are the performance of the baselines on the auto-generated random masked test data and smart masked test data respectively, while "Test Reconstructed Lacunae" is the performance on the gold standard data of manually reconstructed lacunae.

We started our model training by doing hyperparameter searches on four different model configurations, using combinations of the masking strategies for masking distribution and re-masking frequency (random-once model, random-dynamic model, smart-once model, smart-dynamic model). After we selected the best performing hyperparameters for each masking configuration with regard to accuracy scoring on the correspondingly masked dev data, we ran the four best performing models (one for each masking configuration) on the three test data sets outlined at the top of this section. The results from these runs are shown in the top section of Table 1.

The random test set has the highest scores on average, while the reconstructed lacuna test set has the lowest scores on average, indicating that the reconstructed lacuna test set is the more difficult scenario. However, it is also the most realistic scenario out of all three test sets, so performance on this test set should be considered the most significant. We observe that all of the tested model configurations outperform the baselines, showing a substantial increase in performance on all test sets. Out of the four different masking strategies we explored, we found that the model utilizing the random-dynamic masking strategy had the highest performance on the random test set and the reconstructed lacuna test set, while the smart-once masking strategy had the highest performance on the smart test set.

It is somewhat surprising that the model utilizing the random strategy outperforms the model using the smart strategy on the reconstructed lacuna set, considering that the smart masking strategy was developed to better reflect the conditions in which actual lacunae occur. This result is likely because the reconstructed lacuna data set is composed of only sentences with fully reconstructed lacunae, and thus is biased towards containing shorter lacunae than we might otherwise expect. As such, in Figure 2 we consider the accuracy of each of our models with respect to the length (in characters) of the lacuna being reconstructed, and we observe that overall performance decreases as lacuna length increases. We also see that while the random-dynamic model has the best performance for lacunae of length 1-2, the smart-once model actually has better performance for lacunae of length 6+. For this reason, we recommend the smart-once model configuration for cases where the lacuna is more than a few characters. For our use case studies in Section 6, we consider outputs from both the random-dynamic model and the smart-once model.

As we saw in the previous section evaluating the quality of our RNN model outputs, performance on the more realistic reconstructed lacuna test set was relatively low, peaking at 37% accuracy. As such, we cannot consider the model by itself to be a definitive means of manuscript reconstruction. The model is better thought of as an additional tool in the toolbox of scholars attempting to reconstruct manuscript lacunae. To this end, we propose to use the RNN model as a means of ranking the likelihood of potential candidates for the lacuna reconstruction.

If a scholar has several candidates for a lacuna from various qualitative methods of reconstruction, in addition to getting the model to predict what it considers to be the most likely reconstruction, we can also extract the probabilities associated with each of the scholar’s potential reconstructions. Once we have these probabilities, we can sort them in descending order to get a ranking of which potential reconstructions the model considers to be more likely.⁶⁶6One limitation of this is that in order for the probabilities to be compatible, the input context for the model must be the same. This means that all candidates being compared must be of the same character length. This will give a sense of which option is statistically the most likely considering the distribution of characters present in the training data of the model. While still not definitive, this ranking gives scholars another piece of evidence to consider when putting forward an argument for a particular reconstruction.

The manuscript P.Duk. inv. 282 comprises four contiguous fragments from a parchment codex and is currently held at Duke University, pictured in Figure 3 Wagner (2022). The manuscript contains portions of Isaiah chapters 36–38 in the Sahidic dialect. The manuscript’s date is unknown. Some lacunae in the manuscript can be restored on the basis of the only other Sahidic manuscript containing these chapters, Morgan Library M 568. For example, at Isaiah 36:16 there are two small lacunae in the Duke manuscript: {coptic}a[..]n\stackunder[-1.5pt]t.e[.]n̄cemoou. We can restore the original reading with confidence from the Morgan manuscript: {coptic}a[uw] n\stackunder[-1.5pt]t.e[t]n̄ce moou (“and you will drink water”).

Other lacunae in the Duke manuscript cannot be restored entirely on the basis of the Morgan manuscript. For example, at Isaiah 37:24 there are four lacunae: {coptic}akno[…….]eic\0a[…..]djeh1m\stackunder[-1.5pt]p̄.[…….]a\stackunder[-1.5pt]r.matj[…….]raïep
{coptic}[….]. The Morgan manuscript helps restore the passage excluding the penultimate lacuna: {coptic}akno[qneq pdjo]eic a[kdjooc] djeh1m\stackunder[-1.5pt]p̄.[ayaï nnh1]a\stackunder[-1.5pt]r.ma tj[….. eh1]raï ep[djice] (“You have reproached the Lord. You said, ‘with the multitude of my chariots […] to the height.”). Where the Duke manuscript has {coptic}tj[, the Morgan manuscript contains the past tense conjugation base verb: {coptic}aïale eh1raï epdjice (“I have gone up to the height.”). Still, as in the Morgan manuscript, the Duke manuscript must contain a verb in the lacuna followed by {coptic}eh1rai “up”. Thus the letter before the lacuna can only be the personal subject prefix {coptic}tj - “I”, which must be followed by a present or future tense verb. Restricting our search to verbs that are both contextually appropriate and appear in high frequency in the database of the Coptic SCRIPTORIUM project, we propose three reconstructions. The models rank each reconstruction as a sequence of consecutive characters, including uninterrupted context following and especially before the gap: {coptic}aknoqneqpdjoeicakdjoocdjeh1mpayainnh1armatj[…..]eh1rai. The three reconstructions from the random-dynamic model are as follows, in order of probability (the log probability⁷⁷7Log probabilities are used to avoid potential numerical underflow that can result from the multiplication of standard probabilities when calculating the likelihood of a sequence. of each sequence is included in parenthesis):

1. 1.
   {coptic}

   mooye (-11.16) $\rightarrow$ {coptic}tj[mooye] eh1rai
   “I am walking up”

2. 2.
   {coptic}

   nabwk (-12.27) $\rightarrow$ {coptic}tj[nabwk] eh1rai
   “I will go up”

3. 3.
   {coptic}

   naale (-12.60) $\rightarrow$ {coptic}tj[naale] eh1rai
   “I will rise up”

The first result, which is in the durative present tense, is less appealing than the other results when considering the other ancient language witnesses to this passage in Isaiah. To the best of our knowledge, all witnesses approximate the Morgan manuscript’s past tense, except for two witnesses that give the future: the Syriac Peshitta (’n’ ’sq, ‘I will go up”) and some manuscripts of Jerome’s Commentary on Isaiah (11.7: ego ascendam, “I will go up”). These two witnesses increase the probability of the second and third result, both of which are in the future tense (signaled by the auxiliary {coptic}na). Although ranked lower by the model, some scholars would surely prefer the third result over the second, since, as we saw above, the same verb ({coptic}ale) appears in this passage in the Morgan manuscript. On the other hand, beyond its higher ranking, {coptic}bwk appears far more often in Old Testament books and especially in Isaiah: in the Morgan manuscript {coptic}bwk is used five times in ch. 37 alone, while {coptic}ale appears only here at the point of disagreement with the Duke manuscript. Thus the most plausible restoration of the passage: {coptic}a[kdjooc] djeh1m\stackunder[-1.5pt]p̄.[ayaï nnh1]arma tj[nabwk eh1]raï epdjice “You have reproached the Lord. You said, ‘with the multitude of my chariots I will go up to the height.”

The Gospel of Philip (GPhilip) is the third composition included in codex II of the Nag Hammadi (NH) library, a collection of thirteen papyrus codices containing a diverse range of ancient Christian texts. Unlike the example discussed in Section 6.1, there are no other surviving manuscript versions of (GPhilip). The codex sustained moderate damage to the top and bottom margins and most of its leaves contain peninsula-shaped lacunae⁸⁸8Archival photo of the manuscript: https://ccdl.claremont.edu/digital/collection/nha/id/2962/rec/182.

The restoration of Saying 55 (63.30-64.5) has been a particular point of scholarly intrigue. While smaller gaps in the Saying can be restored with some confidence, scholars have proposed various readings for one lacuna of 5-6 letters, which contains the object of the verb {coptic}acpaze or “kiss.” The passage, which describes Jesus kissing Mary Magdalene, reads: {coptic}nef1acpaze mmoc atec…nh1ah1 ncop, "He used to kiss her on the ….. many times" (63.35-36).

This case presents an especially challenging reconstruction due to the size of the lacuna. As discussed above (Section 5.2), the accuracy of the model degrades as the size of the lacuna increases. We consider outputs from both the smart-once model, which provides the highest accuracy rates for longer lacunae, and the random-dynamic model, which provides the highest accuracy rates for short lacunae.

Since the model is trained on Sahidic texts, the Saying needs to follow the orthographic conventions of the Sahidic dialect. Thus we changed the prenominal preposition {coptic}a- ‘towards, on’ to {coptic}e- (in the Sahidic dialect {coptic}a is the past tense marker), resulting in the input text: {coptic}acpaze m̈moc etec[…..] n̈h1ah1 n̈cop “kissed her on her … on many occasions”.

The four letters before the lacuna includes an indirect object construction headed with preposition {coptic}a followed by a feminine possessive article {coptic}tec- “hers.” Due to this syntactic environment, reconstructions are limited to feminine nouns, likely a body part in this case. To fill the lacuna, we have the models produce their predictions for either a 5 character gap or a 6 character gap:

1. Smart-Once:

   1. 5 spaces: {coptic}h1huen

   2. 6 spaces: {coptic}h1hueee

2. Random-Dynamic:

   1. 5 spaces: {coptic}h1ooee

   2. 6 spaces: {coptic}h1ooeee

Unfortunately, none of the reconstructions produce an attested Coptic lemma.

However, the models can still be leveraged to compare editorial suggestions and assign greater or lesser probability of editorial reconstructions. In this case, editors Bentley Layton and Hans-Martin Schenke propose several options for a 5 letter feminine body part: Schenke proposes “mouth” ({coptic}tapro). Layton offers multiple readings: “mouth” ({coptic}paiqe or {coptic}tapro), “cheek” ({coptic}ouoqe), “foot” ({coptic}qalodj), and "forehead"({coptic}teh1ne) as possible candidates Schenke (1997); Layton and Isenberg (1989). The editors present these candidates in an unordered manner, not singling out any one as being particularly more likely than the others.

Table 2 compares the output of smart-once model and the random-dynamic model, and contrasts the confidence of each model’s top two predictions (again, not attested Coptic lemmas) with the list of attested feminine nouns supplied by the editors. As the table details, the lemma ({coptic}ouoqe), “cheek” is favored by the smart-once model and the lemma ({coptic}teh1ne), “forehead” is preferred by the random-dynamic model. Both of these results differ from Schenke’s reconstruction of {coptic}tapro, “mouth” Schenke (1997).

Table 2 also compares the effect of normalization on the model reconstructions. As discussed above in Section 3, the Coptic SCRIPTORIUM data utilized to train the model includes both normalized and original (un-normalized) data. We hypothesized that the normalization of dialect differences to conform to Sahidic orthography would greatly impact the results. However, in the end, the normalization had little impact and only slightly modified the ranking orders and confidence as Table 2 documents. Note the slightly different ranking of {coptic}paiqe and {coptic}tapro, two different words meaning "mouth," by the smart-once model.

These models provide quantitative data about reconstructions and offers a relative ranking of the alternatives proposed by text editors. In cases like the one discussed above in GPhilip where editors have provided multiple possible reconstructions to fill the lacuna and comparison to other manuscripts is not possible, this is an especially valuable tool in assisting readers in deciding which reading best fits within their comprehension of the passage.