A Picture is Worth 500 Labels: A Case Study of Demographic Disparities in Local Machine Learning Models for Instagram and TikTok

The Android Runtime (ART) loads all APKs on the device. Upon opening the APK, it loads all DEX files and the natively compiled code. Since 2019, the ART has used Ahead Of Time (AOT) compilation alongside Just-in-Time (JIT) compilation [23]. AOT compilation creates .oat files, which are essentially ELF binaries with additional OAT-specific sections [1] from the DEX files. The system does not compile some DEX files directly into an .oat file. [5]. The JIT or Java interpreter handles the remaining DEX files at runtime. Figure 2 shows the code execution paths in Android.

On the other hand, native code is directly executed on the CPU in their compiled format. For an APK to execute native code, it must use the Java Native Interface (JNI) [74]. Native libraries are typically introduced to an Android app to increase performance [46] or implement third-party code [18]. This feature is especially useful for computationally expensive tasks. Because of the computational needs, JNI calls commonly facilitate ML execution [65].

A JNI call executes in three phases: call the JNI linking function, translate function arguments into native arguments, and execute the native function. In Java, apps create links to JNI calls through the native keyword [8]. Android’s Native Development Kit (NDK) links native functions to the shared library [2]. JNI functions, at the native level, have two arguments: the JavaVM and the JNIEnv. The JavaVM variable allows apps to have a JavaVM in native code. A JavaVM can interact, create, and destroy Java objects. Whereas the JNIEnv performs local thread storage. These low-level objects prepare data for transmission or translate data for native usage [7]. Then, the native code executes as it would on any Linux system.

There is, however, another type of JNI function that bears mentioning: JNI trampolines. JNI trampolines determine their native address dynamically. The operating system catches all trampoline calls and prepares the call for its actual execution point [34]. The artQuickGenericJniTrampoline retrieves the actual native code and then executes it on the CPU directly. The return values occur asynchronously [34] in the GenericJniMethodEnd function.

At the OS level, all JNI and AOT calls are quick codes within the Android Runtime (ART). Quick codes are a C++ object that represents the executable code for a function. Quick codes are directly executed by the CPU at runtime. When executing the code, the ART passes the arguments to the CPU as a raw C pointer. A high-level C++ object represents the return value of the executing function. Executing the function sets the value of the return object. To access the arguments, the ART uses a shorty; a shorty is a character array that indicates the return value type and the types of the arguments. Later in the work, we describe how we interpret the shorty to retrieve the function arguments and return value.

Native code also communicates with Java code asynchronously using callbacks [17]. Message objects, transmitted using a callback, facilitate the communication. A handler in the Java layer receives the messages. All applications implement callbacks uniquely; however, applications commonly use the Android.os.message object. Current works trace a callback function by the name of the callback object [33]. When the objects are obfuscated, determining what the callback object is becomes complicated [33].

Tools like Frida [3] require root access to function properly. When working with Android, acquiring root differs depending on the Android OS. There are two distinct categories for an Android OS: custom ROMs and production ROMs. A custom ROM is an Android OS derivative that a user installs. Manufacturers install production ROMs, which are closed-source.

Root access is normally attainable with a custom ROM, because it is common for production ROMs disallow root access entirely. However, it is possible to attain root on production ROMs, [4], which requires the phone to be OEM unlocked. In such a case, we gain root access by first downloading the production ROM’s boot image and overwriting it with Magisk [66]. We then replace and boot into the new boot image. After initialization, Magisk’s Android app will have root privileges, install the ‘su’ binary, and deploy Frida on the device.

Applications detect if they are on a custom ROM by utilizing Google’s integrity API [12]. The integrity API allows app developers to gain insights into the device their app is currently running on. Therefore, it is crucial to evaluate an app on both a production ROM and our custom ROM. In this paper, we perform our valuation and analysis on both a custom ROM and a production ROM.

Pytorch is an open-source library commonly used on mobile devices [65]. Instagram uses Pytorch’s API calls to perform their ML calls. Pytorch has a publicly available Java API that apps utilize to perform ML tasks.

Pitaya is Bytedance’s own ML library [41]. Pitaya is privately deployed to Bytedance’s applications. Pitaya consists of both native and Java components. Our work will focus on their native library libbytenn.so, which controls their ML engine. Native executions control TikTok’s computer vision ML suite. Java code never directly calls any ML API while the camera of the application is active. This implies that prior works [65, 56], that do not trace native code, cannot detect Bytedance’s apps ML behavior.

Other approaches perform static analysis on decompiled Java and native libraries to find ML-related functions [72, 56] and use these to create scripts for further dynamic analysis. While these approaches can cover a broad range of apps, they are not sufficient to detect relevant ML execution in every app. The reason is that performing static analysis to detect ML exhibits two shortcomings. First, there can be ML functions in native libraries that are not exported and thus do not have a searchable name. These ML functions may be called by other native functions instead of through explicit JNI calls from the Java layer. Second, we do not know when the app executes the statically detected functions or if it executes these functions at all. This means that static analysis approaches may generate false positives if the ML functions are inside unused code.

At a high level, we detect evidence for ML execution by logging the targeted function calls within an app. One approach to achieve this objective is through hooking into all of the functions with Frida. This approach, however, is not feasible because of the large number of functions in an app, which would overload this kind of monitoring. Therefore, we build a more scalable method by instrumenting the ART to log function calls at runtime together with their passed arguments and return values.

To implement our logging system, we instrument the methods in the ART that are responsible for executing function invocations. By managing function invocations, these ART methods handle invoked function parameters and return values. We edit the ART methods to log each invoked function name, parameters, and return values. Additionally, we log the corresponding process id, thread id, and timestamp. The two ART methods of interest are the one that executes quick codes and the other one that executes JNI trampolines. Instrumenting these two methods ensures we can log all targeted function invocations without having to instrument each function invocation separately. Now, we discuss how we log quick code executions and JNI trampolines.

Therefore, we parse the shorty strings of each invoked function to determine the type and size of each argument. This allows us to correctly log the values and iterate over the array with the appropriate step size. We note that for non-static functions, the first argument is the object’s this pointer, which we skip by stepping over the first four bytes of the argument array. The result of an invoked function is stored in a special native object. To retrieve its value, we call a specific function tied to that object, which is determined by the first character in the shorty string. For example, we call result->GetI(); to retrieve the value from the result object if the first character in the shorty string is “I.”

The values we read for arguments and return values are raw bytes. As we know their type, we can correctly cast these bytes to log them in a readable format. An exception is the “L” type which is a pointer to an object or array of primitives. The ART method does not receive the size of this object because it assumes the receiving function knows its size. Thus, instead of casting “L” values to a certain type, we log a fixed constant of 500 bytes.

As explained in section 4.1, other approaches commonly use static keyword analysis to detect ML functions within Java and native libraries. However, the reconstructed ML pipelines are only in the Java layer or use JNI functions, which are exported from native libraries. This strategy fails to rebuild an ML pipeline if the ML execution happens solely inside the native functions. The first reason is that the ML function might not be discovered in the first place as it is not labeled with a name when it is not exported (we address this issue in section 4.1). Second, the function calls in a native library can depend on addresses that are dynamically calculated at runtime and cannot be determined statically.

The ML-related functions we identify in the first layer can be at an arbitrary point in the execution chain. Therefore, we inspect functions both prior to and after the call. First, to assist in the recovery of previous functions, we use Frida to hook into the given functions and display the call stacks in both Java and native code. We note that the functions in the reconstructed pipeline may have obfuscated names. Since these obfuscations can vary between different devices and app versions, we adapt the Frida scripts per APK instance. Then, we reconstruct the complete ML pipeline by exploring both decompiled Java functions and native libraries.

We use JADX to decompile the APK and statically analyze the Java code by following previous approaches [56, 65, 72, 47, 73, 20]. One caveat here is that prior approaches decompile APKs downloaded directly from an app store, instead of retrieving it from the device. Such approaches would miss important functionality contained in the split APKs. So, we make sure to analyze the split APKs as well. To rebuild ML pipelines on the native layer, we perform static analysis by using Ghidra to disassemble and decompile the native libraries.

Since ML execution may also be performed in non-exported functions, we trace ML execution throughout the native libraries using a hybrid static-dynamic approach. Tracing execution flows within non-exported functions is challenging to perform through static analysis because of jump instructions to addresses, which are determined at runtime and are nondeterministic. To determine at which address such an instruction continues, we use dynamic analysis to monitor the execution and find the jump destination.

Toward that end, we first take the address inside the native library of this jump instruction. Then, we use Frida to hook into this address dynamically. This approach allows us, once the instruction is executed, to obtain the address where the execution continues. While this address is not constant due to memory randomization, it still shows us the byte offset within the native library, which is static. Using this byte offset, we can then continue statically tracing the native function execution. If we find multiple destination addresses for the jump, we follow each execution flow. We disregard those flows that do not contain ML as these are not important to reconstruct the ML pipeline.

There exist various datasets which other approaches use to assess the performance of vision models. These datasets often show the general accuracy of models with respect to the label. Such datasets are not designed to evaluate the bias of a model with respect to a certain feature. Even if the datasets contain diverse images, they do not contain the features that the model in an app considers. For instance, the model in a given app may calculate the probability that a person wears a necklace. Existing datasets would not contain enough sample images of individuals wearing a necklace, with diverse demographic attributes, to sufficiently assess the model’s performance.

From the ML pipeline we reconstructed in the previous layer, we know the ML model and its output labels. To create a meaningful assessment of the model, we create a dataset tailored to the features that the model extracts. As will be evident later, we assess vision models for faces. As such, we utilize and extend the framework by Rosenberg et al. [57] to synthesize images of faces belonging to diverse demographics. We apply different semantic attributes to the generated faces using concepts of interest from Instagram. We then manually annotate the generated dataset to ensure the images are not distorted and their ground truth labels are correct.

We design both internal and external data injection such that we can efficiently evaluate the model on datasets. Note that in both cases, we utilize the app’s default behavior as a vehicle to execute the ML pipeline. Our approach has the benefit of ensuring the ecological validity of the model analysis. Also, it does not require accessing or reconstructing the model weights, which has ethical and legal implications.

We focus our analysis on the computer vision model deployed by TikTok. By searching for probability values in $[0,1]$, we detect logging messages (MessageCenter.postMessage) which display a face count, bounding box, age, and value named “boy_prob”. We hypothesize that the last value describes the probability that the image contains the face of a male person. We thus refer to this variable as sex. These logging messages show up always when the camera is active, even when no filter is selected.

We use these logging messages as a starting point to reconstruct the complete ML pipeline as shown in Figure 4. Through a string search of “boy_prob”, we find that the logging function comes from the libeffect.so library which is responsible for applying face filters. This library has a native hook into the camera feed (1), applies pre-processing on each frame (2) and passes them to a computer vision model in libbytenn.so which performs an inference call on the image data (3) and saves the output to a thread safe register (4). Note that it is a native library which directly calls the native ML functions. This differs from most Android apps which perform ML calls through a JNI call from Java [65, 56]. Back in the face filter library, the logging function reads the inference values from the register (5) to build the message and execute the callback (6) which we catch in the OS logs. We highlight that the logging function is a native callback which acts as an asynchronous message handler. TikTok uses their native callbacks to send the information from the native to the Java layer. The logging function is not an exported function, meaning it has no external entry point, and there are no comments describing this function. This means that automated frameworks likely overlook its existence [56, 65]. The native callback is received by a Java function (7) which saves the data to an encrypted log file on the device (8). In summary, we discovered the ML pipeline inside the native libraries through a logging function which is not directly related to ML. Through our pipeline reconstruction we are able to confidently connect it to the ML process.

In the pipeline reconstruction, we found that the inference call on images happens continuously on the camera feed without a user’s explicit action. Therefore, we do not need to rebuild and execute the whole ML pipeline. Instead, we utilize the existing ML execution and inject the images externally by displaying them to the phone’s camera through a display. We capture the model outputs by hooking into the native callback function containing the “boy_prob” message with Frida. For each image, we then intercept the message function which displays the inference values for sex and age.

For the first null hypothesis group, we evaluate the disparities of age across sex. TikTok assigns a score close to one if the model is confident a person is a male and close to zero if it is confident the person is a female. For each null hypothesis to be true all age bins should have the same sex prediction distribution. All of the $p$-values were below the threshold. However, we cannot confidently reject the hypothesis for the group 20-29 as its power was estimated $0.17$. All other groups are confidently rejected.

Figure 2 shows that the sex distributions vary in between the age bins. At younger ages, the sex of a person is difficult to differentiate, whereas the gap between male and female widens as people become older. However as the individual become older, the model becomes more confident when discerning males but not with females. In the oldest age bin, the model seems to revert back to the first age bin where it is unsure the sex of a person.

The second null hypothesis group states that each ethnicity should have the same sex distribution for each ethnicity. For this to be true, each ethnicity should show the same predicted distributions. For each sex, we compared each ethnicity’s distributions for male and female scores. For each sex group, we were able to reject their null hypothesis. Figure 6 depicts the distributions of sex scores grouped by ethnicity. The figure shows that black individuals have the most variance, where the model appears to be unsure with both black males and black females. On the other hand, east Asian females and middle eastern males are confidently identified.

Our third hypothesis states that the predicted age scores fall within the corresponding age bin. Figure 7 shows how the model fails to correctly predict ages, especially for younger individuals. Younger age bins all have medians outside of their age bin. For example, the $0-2$ age bin has a median at 13. However, as the individuals get older the median age falls within range around the age of 40. There is special significance to these results. If TikTok were to rely on this model to perform age verification, as they have alluded to in public statements in front of Congress [28], then their platform fails to properly classify children.

Our fourth and final null hypothesis group states that each ethnicity must have the same distribution for age scores. This means that each age bin should show a similar distribution across the ethnic demographics. Similar to the previous cases, the statistical tests reject the null hypothesis all except for the ethnicities aged 60-69 with a p-value of $0.103$ and ethnicities aged 70-100 with a $p$-value of $0.038$. Figure 8 shows that for younger ages the demographics exhibit different age prediction performance. The TikTok model predicts accurate age for younger individuals from Asian demographics more than all other demographics. However, when looking at older individuals the distributions become more similar.

Our analysis begins on the custom ROM where we observe that ML is triggered whenever the user selects a photo to be uploaded as a Reel or records a Reel using the camera. At this time, we notice that the app calls the forward function from the Pytorch interface, which is an inference call of a ML model. We observe that Instagram, similar to other Android applications [65, 56], implements these ML calls through the Pytorch ML framework which executes ML operations as JNI calls from the Java layer.

Starting at the forward call, we perform static analysis on the decompiled Java code to reconstruct Instagram’s ML pipeline which we depict in Figure 9. We find that when a user selects an image (1), this image is converted (2) into the corresponding input format for the model, which is a tensor of a Bitmap of size 224x224. The app then creates an ML thread and passes the model input to the thread (3) which calls the forward function of the ML model (4). The output of the model is a list of 512 tensors which are cast to Java objects (5), each holds two key pieces of information: an output class label, which describes a concept, and its corresponding inference score, which ranges from 0 to 1. The tensor elements are then sorted by value and the top 10 concept scores are selected for further processing (6). We present the full list of concept scores which are calculated for each Reel in Appendix A.

In the Instagram Reels, the ML model invocation is triggered by a specific user action. Therefore, we create a script from the reconstructed ML pipeline functions to execute the ML task and inject the data internally. First, we load all of our images into the phone’s storage. Then, we overwrite the implementation of the threading function which initiates the original ML calls. To ensure that the app functions normally, we intercept the original thread call and create our own threads based on the pipeline we observed. In the new thread function’s body, we inject a script that iterates over all the images uploaded to the phone. For each image, we cast it to the appropriate Bitmap format, instantiate the ML model object, trigger the inference call, and capture the inference scores. Once completed, the script gives control back to the application to run the original threaded task.

To validate that Instagram concepts accurately represent their semantic meaning, we manually cross-matched Instagram concepts to ImageNet [31] labels (see Appendix B). We identified 87 concepts that have matching labels in ImageNet. We evaluated Instagram’s model using samples from these 87 ImageNet classes. Figure 12 shows higher scores for Object in Image compared to Object not in Image samples. This confirms that Instagram scores accurately reflect the semantic meaning of their concepts.

While there exist many open-source face datasets, they lack accurately annotated semantic and demographic features that map to Instagram concepts. Therefore, we compile a comprehensive dataset from multiple sources to examine all relevant concepts. First, similar to TikTok, we utilize 42,130 images from FairFace [43] which has two gender labels, seven ethnicity labels, and nine age ranges. We use FairFace to assess Instagram’s age-related concepts across different demographic subgroups (gender and ethnicity). Second, to examine the more challenging concepts of facial features, we construct a synthetic dataset with explicit semantic features following Rosenberg et al.’s [57] method. Similar to that work [57], we use an open-source Text-to-Image (TTI) diffusion model³³3https://huggingface.co/SG161222/Realistic_Vision_V4.0_noVAE that is fine-tuned to generate images of human faces. We guide the model to generate face images with diverse demographic and semantic features that reflect Instagram’s concepts such as facial hair, glasses, and hairstyles. We generate 28,919 image-concept pairs belonging to eight demographic groups spanning four ethnicities {Asian, Black, Indian, White} and two sexes {Male, Female} with an average of 2,892 images per concept, and 3,615 images per demographic group.

With the pipeline, we design an experiment where the images are passed to Instagram internally and we collect the 512 concepts scores for each image. We use a Pixel 4 phone on a production Android build version TP1A.221005.002.B2, which was last updated in February 2023, and Instagram version 309.1.0.41.113. We upload all images to the phone using ADB. The dynamic analysis script uses Frida version 16.14. We aim to use the described datasets to characterize Instagram’s vision ML potential demographic disparities and spurious correlations.

Null hypotheses 5, 6, 7 capture whether or not performance disparities associated with facial concepts scores appear among images of different demographic groups. Each null hypothesis is evaluated with a Kruskal-Wallis H-test (the non-parametric version of one-way ANOVA, preferred when groups distributions are not normal or skewed). The statistical significance of the three null hypotheses, using a significance threshold of 0.05 divided by the number of groups (correcting for multiple hypothesis testing), for Instagram facial concepts scores are indicated in Table II. The three hypotheses indicate significant performance (scores) disparity across the groups.

Moreover, since the dataset has positive and negative samples per concept, we compute the Area under the ROC curve (ROC-AUC). Table II shows the AUC values per concept and demographic group. An extension of this analysis on the overall dataset is shown in table IV.

Figure 11 presents the distribution of ‘child’ scores across ethnicity and age groups. We conducted a Kruskal-Wallis H-test to assess null hypothesis 6 across ethnicity groups within every age group. The test rejected the null hypothesis for all age groups except the 70-100 group. It is worth noting that both TikTok and Instagram fail to reject the null hypothesis for younger age groups.

Aside from the facial concepts, Instagram infers about 502 other concepts about each image (Appendix A). While these concepts are not related to the face, they can shed light on the model’s spurious correlations. We analyze potential correlations between these concepts and the demographic groups using our generated data. For a fair assessment, we convert the image’s background to a plain grey color, eliminating any possible leakage of these concepts into the images from the generative model. We feed the grey background images to Instagram and collect the scores again. We identify concepts with an average score above 0.15 within any demographic group to filter out low-score concepts. Next, for each concept, we determine the demographic group with the highest average score and assess whether it significantly differs from all other groups using Null hypothesis 7. Table III shows the potential spuriously correlated concepts with each demographic group. For example, the ‘great_wall_of_china’ concept is correlated with Asian women, ‘nudity’ is correlated with White men, and ‘art_painting’ is correlated with White women.