Open-Source-AI-Generated Images:
Characterizing the Civitai Ecosystem.

We compile a dataset containing the metadata of all models, images, and creators in Civitai. To accomplish this, we utilize the Civitai REST API.¹¹1https://github.com/civitai/civitai/wiki/REST-API-Reference In addition, we employ selenium webdrivers to crawl the relevant Civitai webpages, enabling us to gather the volume of tips to each of models, images and creators.

Our study involves two qualitative analyses – (i) extracting the themes of models and images (§LABEL:sec:theme), and (ii) identifying person names as well as occupations from models’ descriptions (§4.1). Considering that our studies cover a large-scale dataset, we leverage ChatGPT, as relevant literature have highlighted its potential in facilitating open coding (xiao2023supporting; gao2023collabcoder) and named entity recognition (wei2023zeroshot).

Our study also contains a comparative analysis of usage of NSFW content in prompts between Civitai and two mainstream AIGC platforms, Stable Diffusion and Midjourney.

For this, we collect two prompt datasets, DiffusionDB (1,528,512 distinct prompts from Stable Diffusion Discord) (wang2022diffusiondb) and JourneyDB (1,466,884 distinct prompts from Midjourney) (sun2023journeydb). Each of the two datasets we selected pertains a large volume of user-generated prompts rooted on one specific platforms, providing a comprehensive lens for us to understand how NSFW content distribute in prompts on the corresponding platform.

Afterwards, we employ OpenAI’s moderation API, configured with the text-moderation-006 model, to quantify the degree of NSFW content exposed in each prompt’s text in our Civitai datasets plus DiffusionDB and JourneyDB. OpenAI’s moderation API takes a prompt’s text as a input and then reports the value of the degree of NSFW content (ranging from 0 to 1), as well as a flag defining whether the prompt is NSFW. We choose this moderation model because it is effective in detecting NSFW content (Nekoul_Lee_Adler_Jiang_Weng_2023) and has shown its ability to process prompt text by being practically employed to moderate ChatGPT’s prompt input (Civitai_moderation).

Moreover, by comparing the image base generated by abusive models, we also highlight that the real-human deepfake and NSFW models have different influence on creators’ usage patterns. Figure 2 presents the comparison of the distribution of productivity among distinct types of models, measured as the number of images per model. Compared with non-NSFW models ($min=0,mid=14,max=133,\mu=24.20$), NSFW models are used to generate more images ($min=0,mid=22,max=100,\mu=36.36$). In contrast, real-human deepfake models are used to generate less images ($min=0,mid=7,max=133,\mu=11.08$) than the non-real-human deepfake models ($min=0,mid=16,max=100,\mu=27.88$). Such a significant difference implies that the types of abusive models act a crucial part in creators’ productivity as well. In the case of Civitai, creators have a much stronger propensity to generate images with NSFW rather than real-human deepfake models. This insight could guide moderators in pinpointing the models that are likely to spur the creation of abusive images by creators.

First, we explore the distribution of NSFW content in prompts’ text. Figure 3 presents the distribution of the degree of NSFW content exposed in prompts text reported by OpenAI’s moderation API. The threshold of NSFW degree for the API to raise a NSFW flag is set to 0.53 by default. Generally, NSFW prompt is not a minor class, where 404,330 (27.24%) prompts are reported as NSFW. Additionally, we notice the distribution appears to be bimodal with the main peak at around degree $=0$ and a lower peak around degree $=1$. Notably, 39.12% of NSFW prompts contain a very high degree of NSFW content ($>0.9$). Moreover, Figure 4 illustrates the distribution of NSFW content degree in prompts within Civitai, Stable Diffusion Discord, and Midjourney. Regarding to all prompts, Civitai possesses an overall higher distribution of NSFW content degree in prompts than other two platforms (Figure 4(a)). When it comes to only NSFW prompts, promopts in Civitai and Midjourney possess more NSFW content than those in Stable Diffusion (Figure 4(b)). Nonetheless, a one-sided two-sample Kolmogorov–Smirnov test reports that Civitai still holds a significantly ($p<0.001$) higher distribution of NSFW content in prompts than Stable Diffusion Discord ($D=0.150$) and Midjourney ($D=0.023$). These findings indicate that emerging AIGC platforms lacking rigorous moderation may face a considerable influx of NSFW prompts, alongside a pronounced inclination among creators to engage with NSFW content in more extreme manifestations.

In all, within deepfake models, ChatGPT recognizes 8,297 distinct person names from 10,170 (75.24%) models, as well as 116,994 (78.40%) images generated by these models. These results suggests a prevalence among creators to target at celebrities when creating real-human deepfakes. Moreover, Table 1 summarizes the topic-10 occupations and statistics of corresponding models and images. Generally, we find that celebrities from three industries are the main targets of deepfakes on Civitai – entertainment (e.g., actress/actor, model, and singer), adult (e.g., pornstar and adult model), and social media (Internet influencer and streamer). Interestingly, regarding NSFW deepfakes, models associated with celebrities from social media industries are more likely to be employed to create NSFW images (14.01% labeled as NSFW) than those associated with celebrities from entertainment (10.01%), or even adult (13.61%) industries. By further exploring the affiliations of these online celebrities with social platforms, we find that most of them are either closely associated with subscription platforms (e.g., Belle Delphine with OnlyFans and Andrea Botez with Fanhouse) or well-known as Instagram models (e.g., Kris H Collins and Brooke Monk). Diverging from the traditional focus on the entertainment industry and politicians (10.1145/3583780.3614729), our findings highlight social media industry as a new domain suffering significant deepfake creation, notably online celebrities exposed to bodily and sexual content, who are more susceptible to being targeted for NSFW deepfake model training (van2020verifying; maddocks2020deepfake).

Existing literature have underscored the importance of moderating communities’ active consumption, as it can potentially encourage creators to produce more abusive AIGC . Inspired by this, we here inspect the influence by abusive AIGC on users’ consumption and the association between such consumption and abusive creation.

Moreover, we observe the communities presenting opposite attitudes between real-human deepfakes and NSFW AIGC. According to Table 2(a) and 2(b), NSFW AIGC pertain higher volume of almost all consumption metrics on average. Compared with not NSFW ones, NSFW models and images seems more popular and more likely to be downloaded or viewed. Meanwhile, creators can gain more favorites by sharing NSFW models and images. Additionally, NSFW models not only are assigned with higher ranking score, but also can prompt viewers to leave more comments. However, these trends get reversed when regarding to real-human deefakes. Either models or images belong to real-human deepfakes pertain lower volume of all consumption metrics on average.

Reminding that a same reversion is observed on abusive models’ productivity (Figure 2), we hypothesize above result is caused by the fact that more productive abusive models and their generated images are more likely to be consumed. Thus, we conduct the Pearson correlation analysis between models’ productivity and users’ consumption on these models and their generated images. Table 3 presents the results of correlation analysis on models and images grouped as real-human deepfakes or NSFW. Aligning to our assumption, except images’ Buzz, all consumption metrics pertain a significant ($p<0.001$) positive correlation with models’ productivity ($r>0$). Meanwhile, such a relation mainly present on both real-human deepfake and NSFW models’ downloads, favorites, comments and rating scores ($r>0.2$), while the connection on all metrics of images is minor ($r<0.2$). Our results reveal the crucial role of users’ consumption in promoting image creation with abusive models, particularly noting that more productive real-deepfake and NSFW models attract greater consumption. Consequently, managing user consumption emerges as a potent strategy to mitigate the image generation by abusive models.

In a nutshell, models’ commercial use permission on Civitai is featured by its overall unpopular requirements on creator crediting and broad allowance for derivatives creation. Meanwhile, most models are available for selling or renting with commercial intents. GT: The above is very boilerplate descriptive analysis. Are there any implications you can derive here?

We next inspect how communities respond towards models on Civitai. We focus on analyzing following metrics about social interaction generated by shared models:

GT: I’d suggest you break the below into headed paragraphs that discuss each metric one by one (no need to have this bullet point list). Instead, at this point, you should make explicit how they fit together into the overall analysis of model consumption.

• •

  Number of downloads: The total number of times that a model has been downloaded by other users. This metrics can serve as a proxy to how popular a model is . More downloads represent the model is spread more broadly and gain higher popularity in Civitai communities.

• •

  Number of favorites: The total Number of favorites that a model has received from other users. This metrics can serve as a proxy to communities’ reception towards the model . More likes indicates higher reception of the model.

• •

  Number of comments: The total number of comments that a model has received from other users. This metrics can serve as proxy to consumers’ engagement towards the model . Higher volume of comments indicates more users engaged in discussion under the model.

• •

  Rating score: The overall rating score the model possesses. This metric can serve as proxy to consumers’ evaluation of the model’s quality . A superior rating score suggests that the model is evaluated as with high quality.

We aim to investigate the potential impact of models’ creation involving abusive GT: this comes out of nowhere. You might want to split off the abusive stuff into a dedicated analysis section themes (such as real-human deepfakes and NSFW content) and the permissions for commercial use on social interactions among consumers. For binary model features, including official flags indicating real-human deepfakes/NSFW content, credit requirements, and derivatives allowances, we categorize models based on these features. Subsequently, we utilize the Mann-Whitney U test to examine whether there are significant differences in the metrics related to consumers’ social interactions among these model groups. The results of the Mann-Whitney U test comparing models grouped by the four binary features are summarized in Table 4. GT: Above needs a bit more motivational breadcrumb text to set the scene as to why we are doing this, and exactly what our task is

For models with multiple-value features, such as permission levels, we classify them into three classes based on their permitted commercial activities: “Sellable” (encompassing “Sell” and “Image”), “Rentable” (involving “Rent” and “RentCivit”), and “None” (related to “None”). We then employ the Kruskal-Wallis H test, followed by Dunn’s post-hoc test, to explore inter-group differences. The comparison results of the Kruskal-Wallis H test among the three groups (“Sellable”, “Rentable”, and “None”) are summarized in Table 5.

GT: This could benefit from some more structure. Why are we diving specifically into NSFW/deep fakes (out of all the possible things we could consider). It might be better to do a broadbrush characterisation here, and then have a separate session dedicated to NSFW/abuse/deepfakes?

In the realm of abusive model creation on themes, consumer attitudes exhibit a notable contrast between deepfake models and NSFW models. Specifically, deepfake models experience significantly ($p<0.001$) fewer downloads, favorites, comments, and a lower rating score compared to non-deepfake models. These findings suggest that deepfake models tend to have lower popularity and are less likely to be disseminated within Civitai communities. Furthermore, consumers demonstrate reduced reception and engagement with deepfake models, reflecting a tendency to assign lower evaluations to their quality. In contrast, NSFW models garner considerable favor among consumers. They consistently receive significantly ($p<0.001$) more downloads, favorites, comments, and a higher rating score than their non-NSFW counterparts. These results suggest that NSFW models are more prone to achieving heightened popularity and are more readily shared within Civitai communities. Additionally, consumers exhibit greater reception toward NSFW models, actively participating in relevant discussions under these models. Moreover, NSFW models tend to receive more favorable public evaluations of their quality. GT: There are 3 stakeholders here: model developers, model users, and content consumers. You might want to make these stakeholders more explicit in the narrative.

Concerning commercial use permissions, consumers exhibit varied responses to different aspects of model permissions. Models without credit requirement are not actively promoted among consumers, receiving significantly ($p<0.001$) fewer favorites, comments, and a lower rating score. Models with derivatives allowance, despite generating significantly ($p<0.001$) fewer comments and lower rating scores, appear to be more likely to be embraced by communities, evidenced by significantly ($p<0.001$) higher download and favorite counts. In terms of permission levels, there is a notable preference among consumers for rentable models. Overall, “Rentable” models garner significantly ($p<0.001$) more downloads, favorites, comments, and a higher rating score compared to both “Sellable” and “None” models. GT: Any explanations for the above patterns?

In summary, deepfake models face lower promotion in communities compared to NSFW models, which tend to encourage more social interaction among consumers, resulting in increased popularity, reception, engagement, and better evaluation. Regarding commercial use permission for models, consumers exhibit less active interaction with those lacking credit requirements and derivatives allowance. These findings indicate that the creation of NSFW content models has gained public promotion, emphasizing the necessity for content moderation in NSFW models. Additionally, it is recommended for model creators to implement credit requirements and open derivatives allowance to enhance the visibility and acceptance of their models within communities. Finally, creators are recommended to assign commercial permission for renting services to their models, as consumers respond more actively towards rentable models.

GT: Seems odd that we dive straight into abusive image creation, considering the overall section is about image creation more generally. Perhaps we could start with an overview of all entities, and then deep dive into abuse? Abusive image creation. Previous research has suggested that creation of images can be highly related to the mainstream themes of shared models on AIGC platforms . Implied by popular abusive creation among models, we first inspect whether NSFW content and deepfakes are similarly prevalent among image creators. For this, we measure the distribution of NSFW levels and frequency of used deepfake models in images.

First, for a prompt’s length, we measure it by the number of phrases divided by the commas in text. Figure 8 presents the distribution of the length of “prompts” and “negative prompts”. We observe that, first, both the length of “prompts” ($\mu=17.55,\leavevmode\nobreak\ SD=21.11,\leavevmode\nobreak\ min=1,\leavevmode\nobreak\ max=15,248$) and “negative prompts” ($\mu=24.68,\leavevmode\nobreak\ SD=39.62,\leavevmode\nobreak\ min=1,\leavevmode\nobreak\ max=3,158$) follow long-tail distributions. This suggests using prompts with few phrases (e.g., 80% prompts contain less than 28 phrases) are more prevalent among image creators. Additionally, according to the cdf-lines in Figure 8, we find that “negative prompts” tend to contain more phrases than “prompts”. A one-sided two-sample Kolmogorov-Smirnov test evidences this with statistical significance ($D=0.040,\leavevmode\nobreak\ p<0.001$) by implying that “negative prompts” has a higher distribution of the number of phrases than “prompts”. Our analysis suggests that image creators tend to input more phrases on content removal and modification than its original generation when creating images on Civitai. GT: Anything interesting from the above? Could we constrast across different models? Or different model users? Any patterns of note?

GT: I’d suggest we turn the below into a dedicated section. Second, for contained named entities, we focus on three primary types of entities: person, location, and organization. These entities have been identified as crucial elements influencing the generated content of images (dehouche2023s). We employ a pre-trained named entity recognition model introduced by (tedeschi-etal-2021-wikineural-combined) to extract these entities. Subsequently, we refine the set of named entities by eliminating those with names that contain “#” symbols, any digits, or are fewer than three characters in length. In general, our analysis reveals that 247,017 (19.44%) “prompts” and 5,104 (1.87%) “negative prompts” include named entities. GT: define negative here This indicates that while named entities are frequently employed for content generation, they are infrequently utilized for content elimination and modification.

We evaluate the distribution of three types of named entities across prompts and present the summarized results in Table 6. GT: Could we have a few plots here, e.g. CDF of num per image etc The most frequently used named entities fall under the category of Person, accounting for 11.77% in “prompts” and 0.63% in “negative prompts”. The top-10 person entities with the highest frequency predominantly consist of names of professional artists (e.g., Jeremy Mann, Antonio Moro, and Greg Rutkowski) and illustrators (e.g., Atey Ghailan, Greg Manchess, and Ilya Kuvshinov). GT: Would be great to get these into generic categories, e.g. celebs, artists, politicians etc Creators primarily employ these artists’ names to fine-tune the artistic styles of generated images. For instance, as depicted in Figure 9, a prompt such as ”Selfie of Super Mario, laughing and drinking at a poolside party, surrounded by girls, art by Jeremy Mann, art by Agnes Cecile” utilizes the named entities “Jeremy Mann” and “Agnes Cecile” to blend the image’s style into a mix of these two artists’.

The second most frequently used named entities belong to the Location category, comprising 5.26% in “prompts” and 0.89% in “negative prompts”. The top-10 location entities predominantly include names of famous tourist countries (e.g., Tokyo, Egypt, and China) and cities (e.g., Hollywood, Paris, and London). Creators leverage these location entities to craft scenes set in these specific locations. For example, a prompt like “Dystopian futuristic Paris, photorealistic, great shot, year 2300, cyberpunk” involves the entity “Paris” to create a future Cyberpunk scene depicting the city. GT: Apart from person, location and organisation, what other things could be included?

The third most frequently used named entities fall under the Organization category, representing 4.63% in “prompts” and 0.39% in “negative prompts”. The top-10 organization entities predominantly include brands of photography (e.g., Fuji, Kodak, and Sharp) and animation (e.g., CGSociety, Studio Ghibli, and ArtStation) companies. Creators employ these organization entities to incorporate filming or CG effects into the generated images. For instance, a prompt such as “Kodak Portra 400, 8k, highly detailed, Britt Marling style, color studio - portrait of a handsome 8-year-old ((sks man)) holding a sandwich, muted colors, up face with 1920s hair style and cloth style, asymmetrical, Hasselblad” aims to add Kodak Portra 400’s film effects to enhance the image’s performance. GT: So, this seems a bit different to ’organisation’. Instead, it looks like they’re using camera brands/features to shape the images. Could this be fleshed out.

JZ: I’m not sure whether this part of named entities should go like this. since most of above analyses are just simple content analysis and I found it hard to go further on them. and also I feel this part a bit out of track. above for model we are talking about NSFW and deepfake. would it be better to relate the named entity analysis to deepfake topic? also following with analysis of prompts’ NSFW degree would be better choice rather than randomly topic analysis? GT: I think the NSFW thing is the strongest angle, and could be split off into a separate section. We could start with a big correlation analysis to show co-location of entities within prompts/creators. We could then use that to motivate a focus on NSFW.

JZ: distribution of social properties, correlation analysis between social properties and above creation features, content analysis on comment