Abstraction Alignment: Comparing Model and Human Conceptual Relationships

To compute abstraction alignment, we first represent the human abstraction as a directed acyclic graph (DAG), where nodes represent concepts and edges represent child-to-parent relationships between concepts. For example, in the medical abstraction in Section 4.3, nodes represent medical diagnoses and edges map from specific diagnoses, like frontal sinusitis, to broader diagnostic categories, like respiratory infections (World Health Organization, 1978). Every node in the DAG exists at a level of abstraction, ranging from the leaf level to the root level, computed based on its shortest path from a leaf node.

The DAG data structure is well suited to representing abstractions because it efficiently encodes both the abstraction’s concepts and conceptual relationships. We can easily access a concept’s level of abstraction by measuring its height and move up and down the level of abstraction by getting its ancestors or descendants. Since the graph is acyclic, it guarantees the hierarchical structure that underpins abstraction relationships. Further DAGs are commonly used to represent abstractions (World Health Organization, 1978; Miller, 1995) and are built into many ML datasets (Krizhevsky et al., 2009; Deng et al., 2009; Johnson et al., 2016a, b), allowing abstraction alignment to apply to a wide variety of domains.

The next step in computing abstraction alignment is to compare the model’s behavior to the given human abstraction DAG. To do so, we map the model’s output space (e.g., classes or tokens) to nodes in the DAG. Often the human abstraction is built into the modeling task, so this mapping is straightforward — e.g., CIFAR-100 includes a human abstraction mapping classes to higher-level superclasses Krizhevsky et al. (2009). However, even when the human abstraction is separate from the modeling task, the model’s output space can often be easily computationally mapped to the DAG. For instance, in Section 4.2, we map words in the model’s vocabulary to nodes in the WordNet DAG (Miller, 1995).

We use this mapping to analyze the model’s behavior based on the human abstraction. Following Algorithm 1, we create a weighted DAG for each dataset instance, where nodes have a value and aggregated value. The value corresponds to the model’s output probability. If a node corresponds to a model’s output, then the value is the model’s predicted probability for that output. Otherwise, the value is zero. The aggregated value is the model’s propagated probability and is computed as the sum of values of its descendants. For example, in CIFAR-100 (Krizhevsky et al., 2009), the aggregated value of the flower node is the sum of the model’s probabilities that the image is a orchid, poppy, rose, sunflower, or tulip. By propagating the model’s confidences through the abstraction, aggregated value provides a measure for the model’s confidence in non-output nodes, including high-level concepts.

Using the weighted DAG, we can measure the abstraction alignment of a model’s decision for a specific instance or an entire dataset. While there are potentially many metrics one could use to analyze abstraction alignment patterns, we define an initial set of four metrics that we have found useful for downstream tasks. During analysis we assume we have a dataset $X$, a model trained to predict classes or tokens $c$, and an abstraction DAG containing nodes $n$ across levels of abstraction $l$. We use the model’s outputs (e.g., $y_{ij}$ for instance $x_{i}$ and class $c_{j}$) to compute the aggregated value $v_{ki}$ of node $n_{k}$.

Abstraction alignment improves model interpretability by expanding the number and complexity of concepts we can use to characterize model decisions and comparing them to accepted human abstractions. A common interpretability task is understanding a model’s mistakes; however not all mistakes are equally problematic. For example, we would be more likely to forgive a model that regularly mistakes cars for trucks than a model that consistently mistakes cars for stop signs. The former aligns with our human abstractions that treat vehicles similarly while driving; whereas, the latter suggests model abstractions do not follow accepted human reasoning with potentially dangerous consequences. In these cases, abstraction alignment helps differentiate the severity of a model’s mistakes, distinguishing benign low-level errors from problematic higher-level misalignment.

To demonstrate abstraction alignment’s ability to characterize model behavior, we use it to interpret a ResNet20 (He et al., 2016a) trained on CIFAR-100 (Krizhevsky et al., 2009). We use the CIFAR-100 class and superclass structure as the human abstraction, resulting in a DAG with 121 nodes across 3 levels (see Section A.1) (Krizhevsky et al., 2009). We compute each test image’s weighted DAG by applying a softmax to the model’s outputs and mapping the output probabilities to the corresponding class nodes (see Section 3.2).

Quantifying abstraction alignment reveals abstraction aligned errors. In Figure 1, we report the model’s accuracy alignment (Equation 1) and uncertainty alignment (Equation 2) across the CIFAR-100 test set. The people, tree, and flower abstractions resolve a large proportion of the model’s errors with their subclasses, indicating that the model has learned those abstractions For instance, $76\%$ of prediction errors for images of maple, oak, palm, pine, and willow are resolved by predicting at the level 2 concept tree. Similarly, uncertainty for images of baby, boy, girl, man, and woman is almost entirely resolved by moving up one level of abstraction to people. While our model only achieves $67.7\%$ test accuracy, seeing that many of its errors align with our human abstraction may increase our trust that it will behave acceptably on unseen data in these categories.

Our abstraction alignment metrics also reveal areas where the model is misaligned with the human abstraction. For instance, the model’s errors are not accounted for by abstractions like vehicles 2 nor does it appear to learn animal categorizations like medium mammals and large carnivores. In both cases, we might consider these results to be acceptable model performance in light of poorly designed or ill-fitting human abstractions respectively. In particular, the CIFAR-100 hierarchy artificially restricts each superclass to contain exactly 5 subclasses — a constraint that produces two abstract nodes for vehicles that arbitrarily distinguish their children rather than meaningfully capture abstracted patterns. In contrast, although the higher-level animal categories are semantically meaningful, they reflect abstract biological concepts like size (medium, large), reproduction (mammals), and diet (carnivores) that are seemingly hard for a model to learn visually from 32x32 images. If learning accurate biological abstractions are important for our task, then we may prefer to train on an alternate dataset or modality that more precisely expresses these characteristics; on the other hand, if learning visual abstractions are acceptable, we may update our human abstractions to better reflect what can be learned from the data (i.e., categorizing animals based on visual similarity).

Abstraction alignment also provides a structure to explore types of model behavior. In Figure 2, we use the weighted DAGs to query for particular types of abstraction alignment. We can define types of abstraction alignment based on the number of nodes the model considers at each level and how it distributes its confidence across nodes. To explore instances where the model’s decision aligns and misaligns with human abstractions, we compare images where the model’s confusion resolves at the level 2 concept against images where the model’s uncertainty is split over four level 2 concepts. We find that while $15.5\%$ of instances are harmless low-level confusion, $25\%$ of images result of confusion at higher levels of abstraction. We can also look at particular types of model behaviors, querying for instances where the model is confused between two distinct concepts. Validating our quantitative analysis, we see instances where model confusion is split between vehicles 1 and vehicles 2. By measuring instance similarly based on the pattern of model decision making, as opposed to semantic similarity, abstraction alignment enables qualitative analysis of model alignment.

Benchmarking the specificity of language models helps us distinguish valuable models that output precise answers from those that output correct but meaningless text. For instance, while “Dante is a person” and “Dante is a poet” are both correct, we would prefer a language model that outputs the latter since it is operating at the correct level of abstraction (Huang et al., 2023). Metrics for benchmarking language model specificity use a dataset of language prompts to test the model’s preference between a specific and general response (Huang et al., 2023). However, these metrics are limited to testing only two human-defined responses at two levels of abstraction, even through generative models may output a variety of correct answers spanning many levels of abstraction (e.g., “writer” or “artist”).

With abstraction alignment, we expand existing specificity benchmarks to more thoroughly test models against a variety of correct answers spanning multiple levels of abstraction. Instead of testing one specific and one general answer, we can use the abstraction DAG to compare many possible answers, such as all answers more specific or more general than the specific answer. Using the expressivity of the abstraction DAG, we can also test the model preference for particular topics, such as testing whether the model prefers a correct answer over an incorrect answer that is still related to the topic. By leveraging an existing linguistic abstraction, like WordNet (Miller, 1995; Fellbaum, 1998), we can test these additional aspects of model behavior without the need for additional human labeling.

We apply abstraction alignment to benchmark BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GPT-2 (Radford et al., 2019) language models. We test the models using the S-TEST dataset which contains sentence prompts for masked token prediction of the sentence’s subject’s occupation, location, and birthplace (Huang et al., 2023). Each of the prompts is labeled with a corresponding specific answer and general answer. For instance the prompt “Lake Louise Ski Resort is in” is paired with the specific answer “Alberta” and the general answer “Canada” (Huang et al., 2023). For each of the three tasks we create a human abstraction DAG by mapping the S-TEST specific answers to nodes in the WordNet abstraction DAG (Miller, 1995; Fellbaum, 1998). We compute edges between nodes using WordNet’s hypernym/hyponym and holynym/meronym functions, creating an abstraction graph of precise and general answers related to the task.

To quantify the model’s specificity, we use subgraph preference to compare the model’s preference for answers in different regions of the abstraction DAG. As a baseline, we recreate Huang et al. (2023)’s specificity metric. Their metric compares the model’s probability in the specific answer to its probability in the general answer. We replicate this metric using subgraph preference by comparing the values of the specific answer node and the the general answer node ($P(s_{s},s_{g})$). Next, we extend this metric to test specificity across additional words and levels of abstraction. Instead of testing one specific and one general answer, we compare all answers more specific than the specific answer (specific answer and its children) to all answers more general than the specific answer (specific answer’s parents) ($P(s_{s\downarrow},s_{s\uparrow})$). We extend these metrics further, testing whether the model prefers a correct answer at any level of abstraction to an incorrect answer by comparing all answers related to the specific answer to all answers related to the task (e.g., all occupation words) ($P(s_{s\downarrow\uparrow},s_{t})$).

Benchmarking models with abstraction alignment reveals aspects of model behavior overlooked by prior metrics. Existing metrics indicate that language models only have a slight preference for specific answers, with most $P(s_{s},s_{g})$ near $50\%$ (Huang et al., 2023). However, by expanding to a larger set of possible answers, abstraction alignment reveals that language models have a strong preference for specific answers. For example, bert-large prefers a specific answer on over $80\%$ of instances across all tasks. This result suggests that prior metrics are too strict and do not account for variety of model preferences, whereas abstraction alignment more accurately reflects model specificity.

Beyond making specificity testing more accurate, abstraction alignment also allows us to test other aspects of specificity. Using $P(s_{s\downarrow\uparrow},s_{t})$, we can test the model’s preference for a correct answer at any level of abstraction to an incorrect answer related to the task. For instance, when predicting the occupation for “Enrico Castellani is a” we compare all answers that are direct ancestors or descendants of the correct answer painter to all other answers that are ancestors or descendants of any other occupation in the dataset. While previously we found models prefer a specific correct answer to a general correct answer, here, we find that models often prefer a incorrect answer to any correct answer. This is not always correlated with accuracy or other specificity metrics — for instance, gpt-2 has the lowest accuracy and specificity on occupation prediction but the highest preference for correctness. By using the abstraction alignment methodology, we have expanded traditional benchmarks, exposing otherwise hidden aspects of model behavior.

Abstraction alignment can also be valuable in dataset analysis by revealing differences in the abstractions we expect our models to learn and those codified in the dataset. Models learn correlations between input features and output decisions from their training data. However, the correlations in the dataset are not always the ones a model developer expects their model to learn. Often, dataset issues are only identified after models trained on them produce problematic outputs (Zech et al., 2018; Caliskan et al., 2022). Applying abstraction alignment to datasets can help us understand how the abstractions they implicitly encode correspond with expected human abstractions before the datasets are released as training data.

To demonstrate abstraction alignment as a dataset analysis tool, we use it to compare the medical abstractions encoded in the MIMIC-III dataset to medical hierarchy standards set by global health authorities. The MIMIC-III dataset contains patients’ medical notes labeled with a set of ICD-9 codes representing the patient’s diseases and procedures (Johnson et al., 2016a, b). The codes are part of the ICD-9 medical hierarchy used by hospitals to justify healthcare costs to insurance (Alexander et al., 2003). However, discrepancies between clinical code application and the ICD-9 guidelines are known to occur due to lack of coder experience, complexity of the coding system, and intentional misuse to increase insurance payout (O’Malley et al., 2005). Since MIMIC-III contains real-world patient records that could be affected by clinical misuse, the code labels in the dataset may not reflect ICD-9’s intended use. Abstraction alignment can reveal how well MIMIC-III aligns with the ICD-9 abstraction to inform model developers of the abstractions their models may learn and perpetuate in deployment.

To apply abstraction alignment in this setting, we use the ICD-9 hierarchy as the abstraction DAG and the dataset’s ICD-9 code labels to represent the dataset’s encodings. The ICD-9 hierarchy Mullenbach et al. (2018) contains 21,116 nodes over 7 levels of abstraction. Each nodes represents an ICD-9 code (e.g., frontal sinusitis) or higher-level code grouping (e.g., respiratory infections). To compute the abstraction alignment of each dataset instance, we map the dataset’s labels to the ICD-9 nodes (Section 3.2) to create a weighted DAG where the aggregate value of a node represents how many labels were assigned to its descendants. To understand possible discrepancies between the dataset’s abstractions and the ICD-9 abstraction, we use concept confusion to analyze pairs of nodes that co-occur across dataset instances. We can think of pairs with high concept confusion as concepts the dataset represents similarly because both concepts often apply to the same medical note.

We begin our analysis by filtering to nodes representing high-level code groupings (i.e., direct descendants of the root). In ICD-9, there are four top-level groupings: procedures, diseases and injuries, v supplementary health factors, and e supplementary causes of injury and poisoning. In Figure 3, we see it is common for the dataset to contain code labels from multiple of these high-level code groupings. Assigning procedure codes with disease and injury codes makes sense because a disease defines a treatment procedure. However, the dataset frequently contains disease and injury and v supplementary health factors codes. In the ICD-9 hierarchy, v supplementary health factors codes are “provided to deal with occasions when circumstances other than a disease or injury are recorded as diagnosis or problems” (American Speech-Lanugage-Hearing Association, 2015). The fact that patient notes in the MIMIC-III dataset often contain both disease and injury and v supplementary health factors codes when the ICD-9 hierarchy expects them to be used disjointly, suggests a misalignment in the dataset’s abstractions.

Next, we analyze confusion between lower-level nodes to understand how specific dataset labels may be misaligned with the ICD-9 abstraction. Often diseases share a medical correlation, so it is expected that the dataset commonly contains both labels for metabolic factors like disorders of lipid metabolism and essential hypertension. However, we also see frequent co-labeling between “other” codes, like other disease of lung and other and unspecified hypertension. In ICD-9, a code grouping often contains sibling codes representing specific variants of that code followed by an “other” catchall code. The frequent occurrence of “other” code labels in the MIMIC-III dataset could cause models to learn to over apply “other” codes when they are unwarranted. It could also suggest that there are common diseases and procedures missing from ICD-9 or medical issues that have arisen since its development in 1977.

Our abstraction alignment analysis of MIMIC-III reveals discrepancies between how ICD-9 codes are applied in the dataset and the ICD-9 abstraction expectations for disease classification. These discrepancies suggest that even models that achieve high performance on the dataset may not align with medical standards and, if deployed to label patient clinical notes in hospitals, could perpetuate code misapplication, leading to inaccurate insurance billing. Further, our abstraction alignment analysis suggests ways the ICD-9 abstraction does not support real-world coding. In fact, the overuse of “other” codes and joint coding between disease and injury and v supplementary codes that we found in via abstraction alignment corresponds to real-world changes made during the transition from ICD-9 to ICD-10, such as increasing code specificity and incorporating supplementary codes into the main hierarchy (Cartwright, 2013; World Health Organization, 2022). This result suggests that beyond dataset quality analysis, abstraction alignment can also identify opportunities for improving ground truth human abstractions.

Here we describe the experimental details for each experiment in Section 4. Code to recreate our experiments can be found at: https://github.com/mitvis/abstraction-alignment. An interactive interface for exploring experimental results is provided at: https://vis.mit.edu/abstraction-alignment/.

All abstraction alignment analysis is performed on CPU. Time to build the DAG and compute abstraction alignment depends on the number of nodes in the DAG and the abstraction alignment metric. On the CIFAR-100 DAG (121 nodes) [Krizhevsky et al., 2009], computing the weighted DAG for 10,000 CIFAR-100 test images takes approximately 2 minutes, computing accuracy alignment and uncertainty alignment takes under a minute, and computing concept confusion takes on the order of 15 minutes. On the MIMIC-III DAG (21,166 nodes), creating the weighted abstraction DAGs and computing concept confusion takes around 30 minutes.

We train and evaluate the models used in the experiments on 1 NVIDIA V100 GPU with 1TB of memory. Training the CIFAR-100 ResNet20 model takes approximately 30 minutes. Running inference on the S-TEST dataset takes roughly 10 minutes per language model.

Additional examples of the abstraction alignment of data instances from Section 4.1 and Section 4.3 are available in an exploratory interface https://vis.mit.edu/abstraction-alignment/.