Testing learning hypotheses using neural networks by manipulating learning data

Neural network language models are systems that learn probability distributions over sequences of words given a text corpus. While they are not tuned or designed to predict linguistic acceptability, they can be used to model acceptability judgments through targeted syntactic evaluations (Lau et al., 2017; Linzen et al., 2016; Marvin and Linzen, 2018; Warstadt et al., 2020). Given a minimal pair such as (1), a neural network language model can be used to assign each sequence of words a probability score. A model that assigns a higher probability to the grammatical sentence (1) than than the ungrammatical (1) shows sensitivity to the underlying syntactic differences between the two sentences. Targeted syntactic evaluations have shown that neural network language models are sensitive to a variety of syntactic and semantic constraints (Linzen et al., 2016; Warstadt et al., 2020).

By using neural network language models as theories of acquisition, we can address an existing methodological limitation in acquisition studies with humans, as these models allowing for complete control of the input provided to the learner: specifically, We can train multiple models on corpora which differ in controlled and targeted ways, and compare how learners with the same initial state and learning goals, but different input, diverge in their behavior at the end of learning.

We present a case study using this method. We train neural network language models on approximately the same amount of linguistic data that humans are exposed to, and use these models to answer two questions: firstly, is a human-scale amount of linguistic input sufficient for a model to learn to make judgments that are similar to human judgments on passive exceptions? Secondly, what kinds of information might a learner be using from the linguistic input to learn to make those judgments?

In Experiment 1, we answer the first question in the affirmative. We show that neural network language models can learn to make acceptability judgments on exceptions to passivization that are similar (though not identical) to those of English speakers. Such behavior suggests that language models can make use of indirect evidence in the linguistic input to learn exceptions.

To answer the second question, we take inspiration from work that uses systematic interventions on a model’s training corpus to test causal links between a model’s input and its eventual inferences (e.g. Misra and Mahowald 2024; Wei et al. 2021). We generate counterfactual training corpora which withhold evidence required by the lexical semantics and entrenchment hypotheses and train language models on both types of corpora. We then compare the acceptability judgments of models trained on these counterfactual datasets against models trained on the unaltered dataset. We find that altering the relative frequency with which a verb appears in the active and passive voice significantly changes models’ judgments of how passivizable a verb is, but altering the lexical semantics associated with a verb does not. We also find that neither of these hypotheses can account fully for a verb’s passivizability, which suggests that additional signals of a verb’s passivizability are present in the linguistic input.

These findings map out a feasible path by which a learner might acquire human-like restrictions on a generalization through statistical indirect evidence in its input. More broadly, they illustrate a method by which researchers interested in understanding the role of linguistic input on learners can examine the effects of large-scale but controlled changes to the learner’s input on the outcome of learning.

In Experiment 1, we compare human judgments of the passivizability of a verb with those of a model which we train. This experiment has two parts. In Experiment 1A, we collect acceptability judgments from English speakers on a set of active and passive sentences containing verbs that are reported in the literature as unacceptable in the passive (Bach, 1980; Levin, 1993; Postal, 2004; Zwicky, 1987). In Experiment 1B, we compare our previously-collected human judgments against the acceptability judgments of a neural network language model which we train on 100M words of English text.

In Experiment 2, we test the causal factors that allowed our model to learn to approximate human judgments. In Experiment 2A, we test whether models’ acceptability judgments about passive sentences containing a highly passivizable verb change if they are trained on a dataset in which that verb occurs much less frequently in the passive than in the original corpus. In Experiment 2B, we train models on a dataset where an unpassivizable verb co-occurs with arguments associated with a canonically passivizable verb. Doing so changes the verb’s expected distribution, which allows us to test whether the lexical semantics of a verb affects its passivizability.

We collected acceptability judgments from English speakers on the active and passive sentences described in the Materials section. Each participant only saw either the active or the passive of any given sentence pair. Specifically, the 140 sentence pairs were divided into two groups of 70 sentence pairs (i.e. 140 sentences) such that each group contained between two to three sentence frames per verb. Each group was then split into two sets of 70 sentences such that the active and passive versions of each item were in different sets. Each set of sentences contained 70 sentences – one quarter of the test and control stimuli.

The presentation order was further counterbalanced by making four ordered lists for each group as follows. Each group was organized into two lists such that an item that appeared in the first half of of one list appeared in the second half of the other list. The order of items was pseudorandomized within those lists to ensure that not more than two active or passive sentences and no two sentences within the same verb class were seen in succession. These lists were then reversed, so that a total of four ordered sentence lists were made per sentence group.

Additionally, every critical sentence was followed by at least one filler sentence. Filler sentences (26 grammatical and 52 ungrammatical) were also used as attention checks. Since the passives of control sentences were expected to be acceptable, we included a larger number of ungrammatical than grammatical fillers to balance the experimental stimuli. The full set of stimuli is available in A.

Participants were instructed to rate each sentence’s acceptability based on their gut reaction, and were told that there were no right or wrong answers. Participants rated sentences by moving a slider from “Completely unacceptable” to “Completely acceptable”, which corresponded to an integer score (invisible to them) between 0 and 100. They were not able to rate a sentence with a score of 50. Two practice sentences (one ungrammatical, one grammatical) were used to familiarize participants with the experimental setup. Figure 1 shows an example of the interface that participants used to rate sentences. The full text of the experiment instructions is given in B.

We estimated the amount of explainable variance across all acceptability judgments, as well as within each verb class, using ten split-half reliability analyses as follows. In each of the ten instances of this analysis, participants were randomly split into two groups. We obtained two point estimates for each item by calculating the mean acceptability judgment score of each item within each half. We then calculated the Pearson correlation coefficient between the item estimates for each half of the results. The mean of these ten correlation coefficients was then entered into the Spearman-Brown prophecy formula (Spearman, 1910) to calculate the corrected reliability coefficient. We use this value to estimate how aligned participants are in their acceptability judgments — participants may differ in the consistency of their judgments about different verb classes, reflecting differences in their grammars. In addition, since any predictor using the same method cannot be expected to show greater correlation with the empirical data than the two halves of the data show with each other, this predicted reliability estimate is an approximation of the highest possible correlation obtainable from the data.

We recruited 84 participants who had IP addresses located in the US and self-reported as native English speakers via the crowdsourcing platform Prolific. Each participant rated 140 sentences (70 test + 70 filler) and was paid US$3.50. The experiment took an average of 12 minutes to complete.

Participants were excluded from analysis if they rated more than 15 filler sentences unexpectedly, either by giving ungrammatical sentences scores above 50 or giving grammatical sentences scores below 50. This resulted in the exclusion of 24 participants. In the analysis that follows, each sentence was rated by at least 13 participants.

We calculated the passive drop of a sentence pair as the difference in mean acceptability ratings between its active and passive version. The results are reported in Figure 2; a steeper downward gradient corresponds with a larger passive drop, or greater degree of unacceptability in the passive than in the active. Since the active and passive sentences in a sentence pair contained the same lexical items except for the auxiliary was/were and by, which are common across all sentences, directly comparing active and passive sentences isolates the effect of passivization from lexical effects that might increase the acceptability of sentences with more common verbs like helped compared to low-frequency verbs like profited.

Across all verb classes, participants gave higher scores on average to active sentences (mean: 88.5 points) than passive sentences (mean: 66.4 points). Although the passive drop was positive for all verbs, its magnitude differed across verb classes. The duration class showed the largest mean passive drop (61.9 points), and the ooze class showed the lowest mean passive drop (8.44 points) among the test verb classes. Sentences in the agent-patient class had an average passive drop of 8.86 points.

To determine whether the difference in passive drop between verb classes was significant, we fit a linear mixed-effects model to predict sentence score from sentence type and verb class as well as their interaction as fixed effects; frame and verb as random intercepts; and by-participant random slopes and intercepts for sentence type. We used the agent-patient verb class as the reference level, since passive sentences with these verbs are canonically passivizable. We found significant interactions between sentence type and verb class in four cases: estimation verbs ($p<0.001$), price verbs ($p<0.001$), duration verbs ($p<0.001$) and experiencer-theme verbs ($p<0.001$). This result indicates that passive sentences containing verbs in these three verb classes were significantly different from the canonically acceptable passive sentences containing agent-patient verbs. On the other hand, there was no significant difference in the sentence scores obtained from agent-patient verbs and ooze verbs ($p=0.754$) or advantage verbs ($p=0.106$).

Within the verb classes that were less passivizable than agent-patient verbs, some verbs were also more passivizable than others. We fit linear mixed-effects models to the data within each verb class to predict sentence score using verb and sentence type as fixed effects with random intercepts for frame and by-participant random slopes for sentence type. We used the verb with the smallest passive drop in the verb class as the reference level. We found that there was a significant interaction between sentence type and verb in some but not all cases. For example, last was significantly less passivizable than required ($p<0.001$), but took was not ($p=0.365$), and cost was less passivizable than earned ($p<0.001$) but fetched was not significantly different from earned ($p=0.257$). These results point to the fact that, even among verbs that can occur in the same sentences, some verbs may nonetheless be more passivizable than others (Zwicky, 1987).

Table 2 reports the results of our split-half reliability analysis, which was conducted by repeatedly splitting the dataset into two and comparing the mean item scores of each item between the two halves. The reliability of the collected acceptability judgment scores was high across all test items as well as within verb classes, suggesting that participants had similar judgments about most items.

Across all verbs except approximated, and even in sentences containing canonically passivizable agent-patient verbs, participants rated active sentences more highly than passive sentences. This difference may be accounted for by pragmatic factors: each sentence in the acceptability judgment task was presented to participants without any surrounding context, although the passive construction is more pragmatically marked than the active (Comrie, 1988). This setting might have caused participants to rate passive sentences as worse than their active counterparts even in the control verb classes.

Notably, although experiencer-theme verbs are often thought of as passivizable, they showed a significant difference in passive drop from agent-patient verbs. Conversely, despite being reported as unpassivizable in the literature, the advantage and ooze verb classes did not have significantly different passive drops than agent-patient verbs. Such nuances in judgments may be difficult to capture in binary judgments that rely on a single linguist’s introspection, and point to the value of crowdsourcing acceptability judgments for complex phenomena (Sprouse and Almeida, 2017), particularly if the judgments are robust across multiple participants, as we showed.

In summary, the human acceptability judgment experiment demonstrated that some verbs in the verb classes being tested are degraded in the passive voice, and that unacceptability was gradient between verbs. For a model to adequately approximate such behaviour, then, it must exhibit the following characteristics:

• •

  Verb class-level exceptionality: some verbs classes (e.g. duration verbs) exhibit passive drops that are significantly higher than the baseline passive drop expected of the canonically passivizable agent-patient verbs.

• •

  Verb-level exceptionality: Some verbs within a verb class (e.g. last and cost) also have passive drops that are significantly different from the passive drops of other verbs in the same class.

• •

  Gradience: (un)acceptability is gradient, with some verbs on average exhibiting higher passive drop than others.

The models we trained are based on the transformer architecture (Vaswani et al., 2017) as implemented in GPT-2 (Radford et al., 2019), and in particular GPT-2 small, which has 117M parameters. We used the Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of 6e-4, and set a maximum input length of 512 tokens and a batch size of 16. The neural network training process is subject to stochasticity under which models might learn probability distributions that make different judgments on our test sentences. We thus trained five different models to verify the reliability of each model’s predictions. Each model was trained for 50 epochs with early stopping if validation loss did not decrease for three consecutive evaluation steps.

We adopted this architecture since there is substantial evidence that GPT-2 models can learn exceptions to passivization: OpenAI’s GPT-2 models produced judgments that correlated well with human acceptability judgments of passive exceptions (Leong and Linzen, 2023). GPT-2 is also sensitive to more general restrictions on English passives; Warstadt et al. (2020) show that GPT-2 gives lower scores to passive sentences containing intransitive verbs (e.g. Jeffrey’s sons are smiled by Tina’s supervisor) than sentences containing transitive verbs (e.g. Jeffrey’s sons are insulted by Tina’s supervisor, showing sensitivity to the fact that intransitive verbs cannot be passivized in English. Finally, GPT-2 also demonstrates sensitivity to other exceptions to general rules, performing well on targeted syntactic evaluations requiring sensitivity to argument structure, such as differentiating between verbs which do and do not participate in dative alternation (Hawkins et al., 2020). However, since we train our models on substantially fewer words than OpenAI’s GPT-2, as we discuss in the next section, it is an empirical question whether models trained on smaller datasets will behave similarly.

GPT-2 was trained on OpenAI’s proprietary WebText corpus, which contains 40GB of data from web content — approximately 8B words, assuming each word contains an average of 5 bytes (characters). By contrast, English-speaking children are exposed to 2–7M words per year (Gilkerson et al., 2017), or 26M–91M words by the age of 13. As our goal is to determine what can be learned from the data available to humans, we trained our models using a significantly smaller training corpus than Radford et al. (2019). Rounding to the nearest order of magnitude, we trained our models on a 100M word subset of the OpenWebText corpus (Gokaslan and Cohen, 2019) to simulate a more plausible model of the linguistic input a human may receive. The OpenWebText corpus is an open-source reproduction of the Web Text corpus and contains web text linked from Reddit with at least three upvotes. This selection method aims to choose a wide range of web text curated by humans.

We used the targeted syntactic evaluation paradigm (Linzen et al., 2016; Lau et al., 2017; Warstadt et al., 2019) to compare between our models and human acceptability judgments. Tasks in this paradigm involve obtaining model judgments on minimal pairs in an analogous fashion to our human subjects study. For each sentence in the test set, we obtained a sentence score by summing the log-probabilities assigned to each token in the sentence, which gives a measure of the likelihood the model assigns to that sentence occurring. The sentence with the higher probability of the two sentences in the pair is deemed to be more acceptable.

Before reporting on the model’s acceptability judgments on passivization, we looked at the model’s judgments on a wide range of basic phenomena as a benchmark of the overall reliability of our models sentence scores. We tested our models’ on BLiMP (Warstadt et al., 2020), a broad-coverage acceptability judgment dataset. BLiMP uses the minimal pair paradigm and evaluates a model’s judgments on syntactic and semantic phenomena ranging from subject-verb agreement to restrictions on the distribution of quantifiers like at least, and including, most pertinently, argument structure restrictions such as the unpassivizability of intransitive verbs.

We adapted the targeted syntactic evaluation paradigm in order to collect numeric scores from the model rather than binary acceptability judgments. After obtaining the scores for each sentence, we calculated the passive drop of each sentence pair by subtracting the active score from the passive score, which normalizes for the effects of lexical items in each sentence.

Broadly, our neural network language models captured some aspects of human judgments of passivizability — particularly, the fact that judgments were gradient depending on the verb used, and that some verbs were less passivizable than other verbs in their verb class. That being said, compared to the split-half reliability measure of 0.92 between human participants, which we use as an upper bound for the amount of variance that can be accounted for in the human judgments, the correlation coefficient of 0.61 between our neural network language models’ judgments and human judgments suggests that our models are still some distance away from being accurate models of human acceptablity judgments on passivizability. For instance, our model systematically overpredicted the passive drop of experiencer-theme verbs and underpredicted the passive drop of advantage verbs. In fact, contrary to human judgments, the model found profit and strengthen more acceptable in the passive than in the active. These findings were unexpected and suggest that the model is sensitive to other information at the verb class level that may not have influenced human acceptability judgments.

Yet, our models nonetheless captured key qualitative aspects of gradient and exceptional judgments on passivization. This finding suggests that the models were sensitive to some evidence for passive exceptions that were present in their training data.

We first used spaCy’s en_core_web_trf pipeline (Honnibal et al., 2020) to obtain dependency parses for each sentence in the training corpus. We then counted the number of times our mutating and target verbs were used in transitive active sentences and passive sentences. Sentences where the mutating or target verb had a dependency edge to a passive auxilliary (auxpass), a passive nominal subject (nsubjpass) or a passive clausal subject (csubjpass) were classified as passive sentences, while sentences with a direct object (dobj) or a clausal complement (ccomp) dependency edge from the mutating or target verb were classified as active. All other sentences were classified as an other sentence type. Figure 5 shows the relative frequencies of occurrence of each mutating and target verb in the active and passive in the training corpus.

Using the corpus statistics and sentence categorization method illustrated above, we created modified corpora for each pair of mutating and target verb. In each dataset, we let the mutating verb appear in the passive only as many times as the target verb does. For example, we intervened to make the distribution of the mutating verb drop, which occurred 3279 times in transitive active sentences and 1146 times in the passive, similar to that of the target verb last, which occurred 683 times in transitive active sentences and four times in the passive. We randomly chose four occurrences of passive drop in the training corpus to keep, and removed all other passive occurrences of drop. Doing so inflates the verb’s relative frequency of occurrence in the active compared to the passive. Figure 6 illustrates the distribution of the training corpora before and after we performed this intervention — the frequencies of all other verbs remained the same, while the frequency of the source verb decreased in the passive and remained constant in the active.

We then trained five models on each corpus following the same training procedure outlined in Experiment 1B. After training these models on the modified corpora, we obtained acceptability judgments from these models and compared them against the judgments of the models we trained on the original corpus in Experiment 1B. If the frequency of a verb in the passive significantly affects that verb’s passivizability, then the mutating verb in a model trained on an modified corpus should be less passivizable relative to models trained on the original data set, while the passive drop of all unaltered verbs should remain the same.

In this experiment, we found that the frequency with which a verb occurs in the passive in the training corpus significantly affects how passivizable the verb is to a model trained on that corpus. This suggests that a verb’s frequency of occurrence in the passive is one source of evidence by which our models learn whether a verb is or is not passivizable.

If frequency of occurrence in the passive were the only driver of unpassivizability in our models, we would expect that each mutating verb would behave exactly like its target verb after our intervention. Contrary to this prediction, although we reduced the passive frequency of mutating verbs to be exactly the frequency of the passive of a target verb, the passive drop of these mutating verbs never increased to the level of unpassivizability of their respective target verbs. This result indicates that passive frequency, at least as operationalized in this experiment, is not the sole driver of unpassivizability in our target verbs.

If either the relative or the absolute frequency of passive occurrences of a verb directly modulated passivizability, then we might expect differences in the magnitude of change in passive drop depending on choice of target verb, since the three target verbs occur with varying frequency in the passive (see Figure 5). Models trained using last as the target verb would demonstrate the highest increase in passive drop since those models were trained on datasets where the mutating verb occurs the most infrequently in the passive. Instead, unexpectedly, each mutating verb showed increases in passive drop of similar magnitudes regardless of target verb. Taken together, these results suggest that frequency alone is unlikely to be the only factor mediating a verb’s passivizability in our models.

For each pair of mutating and target verbs, we converted 30% of the active sentences in our training corpus containing the target verb lemma into sentences containing the mutating verb lemma (e.g. dropping $\rightarrow$ lasting; dropped $\rightarrow$ lasted). Keeping the target verb in the majority of cases allows us to compare the mutating verb against the target verb in models trained on the modified corpus. In addition to the sentences in which the mutating verb occurred originally, the mutating verb then co-occurred with some of the subjects and objects that previously co-occurred with target verb. We randomly chose which sentences to alter; these sentences were interspersed throughout the corpus. Examples of sentences that underwent intervention using the mutating verb last and target verb drop are given in (\nextx):

The BBC’s Geeta Pandey recently droppedlasted in to meet him. \aIf you don’t charge the attack, the arrow won’t fly very far and will droplast toward the ground. \xeSince alterations were performed on strings, the syntax of the original sentence was not a factor in this intervention: both intransitive and transitive sentences could be altered, unlike in Experiment 2A, where only transitive sentences were altered. Phrasal verbs and idiomatic expressions were also affected by this process, as illustrated in (\lastxa). Although the semantics of this new verb was not explicitly specified, the distribution of the arguments of the mutating verb changed through our intervention: the arguments of the mutating verb in the added sentences might be ones which co-occur with other agent-patient verbs, or ones that occur more frequently in the passive than the mutating verb’s original arguments.

After obtaining modified corpora for each pair of mutating and target verbs, five models were trained on each modified corpus and acceptability judgments were obtained from the models following the procedure outlined in Experiment 1B.

Figure 10 reports the change in passive drop of mutating and non-mutating verbs as a result of altering the mutating verb’s arguments in the training data. Verbs that did not undergo any intervention, which we call non-mutating verbs, did not show any significant change, as expected. The mutating verbs did not all show changes in passive drop either.

We found a small decrease in passive drop for the mutating verb last, but no consistent change in passive drop for sentence pairs with the mutating verbs require and take.

We first determined if the effect of the intervention was significant, abstracting away from the specific verb by including the identity of the verb as a random effect in a linear mixed-effects model. Specifically, we modeled the relationship between passive drop and training corpus type, using by-model and by-verb frame random intercepts as well as by-verb random slopes and intercepts for whether the verb is mutating in the training data. We compared this model against a full model that additionally included whether the verb has been mutating in the training corpus as a fixed effect. A likelihood-ratio test indicated that the full model does not provide a better fit for the data than the reduced model, $\chi^{2}(1)=1.32,p=0.34$. Thus, we found that altering the arguments a verb is seen with in the training dataset does not significantly affect the verb’s passivizability when verb-level differences are accounted for.

We next verified that the verb last behaved differently from require and take when mutated. We fit a linear mixed-effects model to model the passive drop of the duration verbs. We used training corpus, verb, whether the verb was mutating, as well as the interaction between each verb and its mutating status as main effects. We included random intercepts for frame and participant, as well as by-frame random slopes for verb. We compared this full model to a reduced model that excluded the interaction between verb and whether the verb was mutating. A likelihood-ratio test indicated that the full model provided a better fit for the data than the reduced model, $\chi^{2}(1)=44.4,p<0.001$, suggesting that our intervention of mutating a verb’s arguments interacted with the specific verb being mutated. Specifically, whereas modifying the training corpus with last as the mutating verb led to an average decrease in passive drop of 3.37 points ($p<0.001$), no significant change was observed from our modification when we used require ($p$ =0.12) or take ($p=0.11$) as the mutating verb.

Changing the arguments which a mutating verb co-occurred with did not consistently affect its passivizability. Instead, while the mutating verb last showed a decrease in passive drop, the mutating verbs require and take showed no significant change in passive drop after our intervention. We attribute this verb-specific difference in behavior to the original distribution of the mutating verbs in the training data. The verbs require and take have relatively commonly-used senses that are more affected and are also compatible with the passive, illustrated in these sentences from the training corpus:

State University was required to take 150 rooms for at least six nights. \aThe scientists said that more research was needed. \xe

Meanwhile, the verb last does not have any commonly-used alternative senses that are compatible with the passive. Our intervention may thus changed the distribution of arguments associated with last, but not to take and require, since these verbs were already compatible with the passive in another sense.

Since neural networks use the same word embedding to represent all different senses of a single word, this finding could indicate that the neural network displays a ‘spillover’ of passivizability from passivizable senses to unpassivizable senses within a given verb. For example, the passivizability of take in (\lastxa) may have increased the passivizability of take in our test sentences even though our test sentences use a different sense of the word. One implication of this account is that the passivizable senses of took and require may be subject to the reverse effect: speakers may judge these senses to be less acceptable due to the presence of other unpassivizable senses. This finding raises the question of whether homonymy and polysemy affect how speakers judgments of passivizability.

We used the English passive as a case study of exceptions to syntactic generalizations. As much of the existing literature on these exceptions relies on informally-collected binary acceptability judgments, we sought to verify these intuitions through an acceptability judgment task in Experiment 1. Instead, we found that although some verbs and classes of verbs are unpassivizable as reported in the literature, this intuition is not borne out for all verb classes reported as unpassivizable — specifically, verbs in the ooze and advantage classes were not significantly different from canonically passivizable verbs. We additionally showed that English speakers’ judgments of passive exceptions are more nuanced than binary acceptability judgments might suggest. Not all verbs which were reported to be unacceptable were equally unacceptable: although last and resembled are both reported as unacceptable, we find that there is a much larger average passive drop for the former.

The exceptions to passivization that we present in this paper complicate theories of passivization that assume passivizability to be a binary feature. The gradience of participants’ judgments on passive sentences provides a window into the complex interplay between syntax, semantics, and pragmatics that is invoked through the acceptability judgment task (Sprouse, 2015; Sprouse et al., 2016). We hope that future theoretical work engages with these findings critically.

What benefits do neural network language models bring to the study of human language acquisition? Existing studies of how children acquire passive restrictions (e.g. Fox and Grodzinsky 1998; Gordon and Chafetz 1990; Maratsos 1985) are limited by the inability to exert full control over a child’s linguistic input: natural language experiments and corpus studies are limited not only by inherent variability in speakers’ input and the intractability of comprehensively tracking what input participants are exposed to, but also by the difficulty of isolating and manipulating features of potential interest (Culbertson and Schuler, 2019). It would be difficult, for instance, to ensure that a child never hears a specific verb in the passive voice, as we implemented in Experiment 2. To some extent these limitations can be addressed in human artificial language learning experiments, which target specific hypotheses of learning through controlled experimentation on constructed languages. However, the test languages in these experiments are necessarily simpler than natural languages — often using vocabularies of under 50 items — and participants are exposed to these languages in an experimental setting where they are actively engaged in learning, unlike in first-language acquisition scenarios (Ettlinger et al., 2016). These differences in paradigm and content may lead to meaningful differences in the outcome of learning.

Using neural networks as a theory of acquisition addresses these existing methodological lacunae in acquisition studies. Neural networks, unlike human learners, are trained on data which the researcher has full control over. Unlike the symbolic models sometimes used in language acquisition research, which are often simplified proof-of-concept systems, neural networks are broad-coverage models that can be trained on a corpus that is as close to the input to human children as possible (Warstadt et al., 2023; Vong et al., 2024). Researcher control is not limited to to the ability to intervene on the data, as we do in this paper: working with neural networks allows for the ability to probe a model’s internal processes to understand which mechanisms are vital to the model’s learning process and form hypotheses about how humans may learn (Baroni, 2022; Lakretz et al., 2021).

That being said, there are a number of methodological limitations to our approach. Firstly, the value of modeling is limited by the interpretability and cognitive plausibility of the models we use (Baroni, 2022). Without a clear understanding of the inductive biases of the particular neural network chosen for comparison, we cannot make a fair comparison between these models and our theories of human cognition. Although we highlighted some similarities between the GPT-2 architecture and human language learning and processing, this architecture is clearly not a perfect model for human language learning (for example, transformers’ working memory constraints are fundamentally different from those of humans; Armeni et al. 2022; Timkey and Linzen 2023), and care should be taken to make fair comparisons between the two.

Secondly, the methodology of intervening on a corpus is highly reliant on the tools available: without the ability to precisely and accurately parse and alter a corpus containing heterogenous data, it is difficult to ensure the feasibility and reliability of any intervention. For instance, the filters we used to make our interventions may have introduced new confounds in the training data. While we were interested in transplanting verbs into agent-patient environments in Experiment 2B, our intervention also targeted sentences containing idiomatic expressions and senses of the verb that were outside our scope. Refining the filtering process would allow us to test hypotheses that are more narrowly defined, and thus to make conclusions at a more granular level.

Finally, the computational cost of training models on each modified corpus is high. We trained a total of 125 models for Experiment 2, each requiring 2 days and using approximately 3e15 floating point operations (FLOP). These training regimes are highly computationally expensive and their potential environmental impacts should be considered. These limitations notwithstanding, we hope that the use of targeted interventions on naturalistic data can be used to compare the plausibility of hypotheses in situations where such interventions are not possible in human research.

To what extent do our results extend to human learners? We have shown through our frequency intervention (Experiment 2A) that models can leverage the relative frequency of the active and passive constructions in training data to learn exceptions; this finding is consistent with usage-based approaches to human language acquisition (Tomasello, 2000; Goldberg, 2006), and can be taken as an existence proof that a learner could demonstrate a human-like pattern through some degree of reliance on frequency statistics.

At the same time, humans and language models clearly differ in their learning mechanisms, goals, and the resources they have for learning. It is possible, then, that while our models’ acceptability judgments are largely similar to those of humans, they achieve such behavior via a very different developmental pathway. Indeed, since our neural networks learn the task of next word prediction by tracking word co-occurrences across a corpus, the fact that they are reliant on frequency statistics is unsurprising. Although human learners are also highly sensitive to statistical information in their linguistic input (Saffran et al., 1996; Thompson and Newport, 2007), they do not rely solely on statistical knowledge in learning: interactive and communicative social pressures also affect how people learn and use language. When a child produces an utterance that an adult finds unacceptable, the adult may repeat that utterance but produce a change in the erroneous portion which the child may then take up in their next utterance if the correction matches their intended meaning (Chouinard and Clark, 2003). Children can learn from these reformulations in order to further their communicative goals — goals which our models lack.

In addition to learning from language, humans can make use of their experiences with objects to learn the concepts of causality and affectedness, which are key abstractions in Experiment 2B. Our models were trained without access to sensorimotor experiences in the real world, and thus may lack these conceptual primitives (e.g. Lake et al. 2017). Having sensory grounding for the physical actions that correspond to verbs could make lexical semantics a more dominant factor in how humans generalize about passivization than our models. In the same vein, the fact that our lexical semantic intervention was not significant in Experiment 2B does not show conclusively that semantics has no role to play in the learning of passive exceptions. If Experiment 2B were conducted on a model with access to the conceptual primitives of cause and effect, that model may be able to leverage its existing sensitivity to cause and effect to learn sensitivity to affectedness, and thus use the correlational evidence provided by sensitivity to affectedness to learn passive exceptions.

In sum, while we have illustrated a potential pathway by which a neural network learner might learn the passivizability of a verb, the same learning mechanisms might not be at play in human learners. Repeating these experiments using models that differ in architecture, and in particular models that have access to interaction and/or causal primitives, may help to disentangle the extent to which these capacities are required to learn about passivizability.

Frequency of occurrence is likely not the only way in which our models learn a verb’s passivizability. In fact, neither of the hypotheses we tested fully accounts for how the model itself judges a sentence’s passivizability: our interventions in both Experiments 2A and 2B never made any mutating verb as passivizable as the corresponding target verb.

What could account for the gap between passivizable and unpassivizable verbs that is not accounted for by our interventions? One possible explanation for this result could be an interaction between frequency and lexical semantics that boosts the (un)passivizability of a verb: while we tested entrenchment and lexical semantics independently as sources of evidence for passive exceptions, our study does not account for potential super-additive interactions between these two hypotheses.

Both humans and language models may also rely on evidence other than construction frequency and lexical semantics to learn which verbs passivize. One potential such source of evidence comes from the existence of similar constructions to the long passive — such as a sentence like (\nextxa), which differs from the passive sentence (\nextxb) by just one word: \pex\aTwo hours were required for the meeting. \aTwo hours were required by the meeting. \xe