PoPreRo: A New Dataset for Popularity Prediction of Romanian Reddit Posts

PoPreRo gathers Reddit posts from five different Romanian subreddit channels, which represent either one of the biggest cities in Romania or the country-wide subreddit. The subreddits are: Romania, Bucureşti, Cluj, Iaşi and Timişoara. These subreddits were collected at first using Reddit API, divided into JSON files to extract the information needed for analyzing the popularity of each reddit post, such as title, content, number of comments, number of up and down votes. However, Reddit API has a limitation of 1000 requests for extraction of different data. Due to the large number of samples that we target for the dataset, the API could not provide all necessary data. Therefore, we use an open-source archive, from where the samples are collected. As mentioned above, all the data is stored in separate JSON files for each subreddit, containing relevant information for determining the popularity of posts.

The dataset comprises 28,107 samples (14,289 unpopular and 13,818 popular) containing over 1 million tokens in total (see detailed statistics in Table 1). Each sample consists of a title, a content, and a binary label, where the title and content are concatenated into a single text. We divide the posts into “popular” or “unpopular” based on the sum of upvotes and downvotes for each post, where the threshold between the two categories is given by the median number of votes (15). To enable consistent evaluation and comparison with future studies, we provide an official split with distinct training, validation, and test sets. Inspired by McHardy et al. [18], we utilize disjoint subreddits for each set, ensuring models cannot capitalize on knowledge of specific topics. To further mitigate potential biases arising from uneven topic or time distributions, we select posts from the same time frame across all subreddits.

Additionally, to control for a potential bias related to the time of day when posts were submitted, we performed an analysis of post popularity by hour. We divided each day into four-hour intervals and categorized the number of popular and unpopular posts within each interval. The detailed results are presented in Figure 2. Notably, we observe a consistent trend across all time intervals for both popular and unpopular posts. This finding suggests that the hour of submission does not exert a significant influence on post popularity within our dataset.

After gathering the data from Reddit, we implement a two-step preprocessing pipeline to ensure data quality and consistency. First, language identification was performed on post titles using FastText [12] to filter out non-Romanian posts (filtered posts are not counted in Table 1). This step guarantees the linguistic homogeneity of the dataset. Subsequently, upvote/downvote scores are normalized to the $[0,1]$ interval. Finally, a binary popularity label is assigned with respect to the median value of the normalized scores, which corresponds to 15 votes. This approach provides a clear threshold for distinguishing popular and unpopular posts. Notably, our data collection and labeling procedure is directly transferable to other languages.

Our first baseline relies on fine-tuning a Ro-GPT2 model [19], a large language model specifically trained on Romanian text. It is based on the original GPT2 architecture, but trained on a Romanian dataset consisting of over 1 million tokens. This allows it to capture the nuances and specificities of the Romanian language, making it more suitable for tasks involving Romanian than the general-purpose GPT2. The Ro-GPT2 encoder is utilized to encode each text sequence into a list of token IDs. Subsequently, the model processes these tokens, generating corresponding 768-dimensional embeddings. We then incorporate a global average pooling layer to capture a Continuous Bag-of-Words (CBOW) representation for each text sequence. This representation is fed into a Softmax output layer comprising two neurons, each predicting the probability of belonging to either the unpopular or popular category. To assign the final class label, we apply the argmax function on the two predicted probabilities. The entire model is fine-tuned for 5 epochs on mini-batches of 32 samples. We employ the Adam optimizer with decoupled weight decay (AdamW) [15] with a learning rate of $5\cdot 10^{-7}$ and $\epsilon=5\cdot 10^{-7}$.

As our second baseline, we employ a fine-tuned Romanian Bidirectional Encoder Representations from Transformers (Ro-BERT) model [7]. Sharing the same transformer-based architecture as the original BERT [6], Ro-BERT has been demonstrated to outperform multilingual BERT on various tasks, as reported by Dumitrescu et al. [7]. Consequently, we anticipate Ro-BERT to be a strong baseline for our Romanian corpus.

Similarly to the previous baseline, we use the Ro-BERT encoder to encode each text into a list of token IDs. We keep the same design as before, where the model generates 768-dimensional embeddings, followed by a global average pooling layer which is fed into a Softmax output layer with two neurons. To assign the final class label, we apply the argmax function on the two predicted probabilities. The entire model is fine-tuned for 10 epochs on mini-batches of 32 samples. We employ the AdamW optimizer [15] with a learning rate of $2\cdot 10^{-7}$ and the default value for $\epsilon$.

For our third classification approach, we leverage pre-trained Ro-BERT embeddings in conjunction with a Logistic Regression (LR) classifier. Consistent with the fine-tuned Ro-BERT baseline, we first tokenize all input samples from the three datasets. Subsequently, we utilize the Ro-BERT model to extract 768-dimensional vector representations for each sample. These representations, corresponding to the final hidden layer of Ro-BERT, are then fed into the LR model for classification.

The first shallow classification approach is based on FastText embeddings [3] and a Support Vector Machines (SVM) classifier. After textual cleaning and tokenization using NLTK’s word tokenizer, we fine-tune a FastText model on the training corpus. This model provides word embeddings for train, validation, and test sets. For each text sample, the word embeddings are averaged to produce a 300-dimensional feature vector, which is subsequently passed to the SVM. Finally, we train the SVM classifier using the linear kernel and the regularization hyperparameter $C$ set to $10$.

Our second shallow classification approach is based on the Term Frequency-Inverse Document Frequency (TF-IDF) representation and a Random Forest (RF) classifier. As for the previous method, we initiate the process by cleaning and tokenizing the text using NLTK’s word tokenizer. Subsequently, we employed a TF-IDF vectorizer to quantify the importance of words within the corpus, generating numerical features for each document. These features are then used to train a Random Forest classifier.

To explore the feasibility of large language models (LLMs) for post popularity prediction in PoPreRo, we employ a prompt-based approach utilizing the 7-billion parameter Falcon LLM [1] (Falcon-7B). Due to computational limitations, we prompt the LLM with contexts comprising two unpopular and two popular examples. Subsequently, we attach an individual test sample to each prompt and ask the LLM to predict the corresponding label. Below, we illustrate the structure of our prompt via a concrete example:

PROMPT (Original): Text: ’Nu vreau sa mai traiesc pe aceasta planeta !’ Label: ’Popular’. Text: ’Unde pot verifica compozi\textcommabelowtia unui produs?. Să testez de exemplu dacă ingredientele unui produs sunt într-adevăr acelea. Sau dacă ni\textcommabelowste tablete de vitamine chiar con\textcommabelowtin vitamine. În ce propor\textcommabelowtii? Sau câtă vitamina A con\textcommabelowtine un morcov - unde pot verifica asta? Ceva laboratoare?’ Label: ’Unpopular’. Text: ’Azi a venit mitropolitul ardealului la noi la liceu să ne convingă să facem religie. Primul lucru care mi-a venit în cap când am văzut ce ma\textcommabelowsină \textcommabelowsi-a parcat în curtea institu\textcommabelowtii..’ Label: ’Popular’. Text: ’Daca intereseaza pe cineva, sa stiti ca e reddit si in romana’ Label: ’Unpopular’. Text: ’Am prins niste fulgere faine zilele trecute’ Label:

PROMPT (Translated): Text: ’I don’t want to live on this planet anymore!’ Label: ’Popular’. Text: ’Where can I check the composition of a product?. To test for example whether the ingredients of a product are indeed those. Or if some vitamin tablets

actually contain vitamins. In what proportions? Or how much vitamin A contains a carrot - where can I check this? Some laboratories?’ Label: ’Unpopular’. Text: ’Today the metropolitan of Transylvania came to us at high school to convince us to do religion. First thing that came to mind when I saw what car he has parked in the courtyard of institutions..’ Label: ’Popular’. Text: ’If anyone is interested, there’s reddit in Romanian’ Label: ’Unpopular’. Text: ’I caught some fine lightning the other day’ Label:

Our binary classification experiments focus on predicting the popularity of text within the PoPreRo dataset. Each text sample is categorized as either popular or unpopular. To evaluate the performance of our models, we employ several metrics. For each class, we calculate precision (proportion of true positives among the identified positives) and recall (proportion of true positives with respect to all positives). Additionally, we aggregate these scores using macro $F_{1}$ and micro $F_{1}$ (accuracy) measures.

The hyperparameters of all models are determined via grid search. For the transformer-based methods (Ro-BERT, Ro-GPT2), we employ a grid search over the maximum number of input tokens in the set $\{50,70,100,120,150,200\}$, as well as the learning rate in the set $\{10^{-5},5\cdot 10^{-5},10^{-6},5\cdot 10^{-6},10^{-7},2\cdot 10^{-7},5\cdot 10^{-7},10^{-8},5\cdot 10^{-8}\}$ and the value of $\epsilon$ for AdamW in the set $\{10^{-6},10^{-7},10^{-8}\}$.

For the FastText + SVM approach, we vary the FastText word-embeddings dimension ($\{150,200,300,350\}$), the window size for the input ($\{2,3,4\}$), as well as the kernel (linear or RBF) and the parameter $C$ ($\{0.1,1,10,100,1000\}$) of the SVM classifier. Similarly, for the Ro-BERT + Logistic Regression approach, we run a search over the maximum numbers of Ro-BERT input tokens in the same set as before ($\{50,70,100,120,150,200\}$) and test different penalty term values (‘l1’, ‘l2’, ‘elastic net’ or ‘None’) for the classifier.

Lastly, for the TF-IDF + Random Forest method, we vary the minimum ($\{4,5,6\}$) and maximum ($\{0.6,0.7,0.8\}$, in percentages) document frequency of the TF-IDF Vectorizer, together with the number of decision trees in the set $\{50,100,150,200\}$ for the Random Forest classifier.

All other hyperparameters are set to their default values. Please note that we release the code to reproduce all baselines, along with the PoPreRo dataset¹¹1https://github.com/ana-rogoz/PoPreRo.

We present the results of our five baselines on the PoPreRo validation and test sets in Table 3. We find that Ro-GPT2 exhibits the best performance, with an accuracy (micro $F_{1}$) and a macro $F_{1}$ score above $0.6$ on both validation and test sets, in contrast to the other baselines which seem to perform similarly well on the validation set, but reach worse performance on the test set.

Evaluating the two state-of-the-art transformer models, Ro-GPT2 and Ro-BERT, reveals some interesting findings. While both achieve comparable accuracy on the validation set ($0.6525$ for Ro-GPT2 and $0.6343$ for Ro-BERT), Ro-GPT2 clearly outperforms Ro-BERT on the test set, indicating the superior ability of the former model to generalize to unseen data. Analyzing the precision-recall trade-off, we observe a shared propensity for both models to exhibit higher recall for the “popular” category, followed by a shift towards higher precision when identifying the “unpopular” class.

The FastText + SVM, TF-IDF + RF and Ro-BERT + LR models achieve comparable performance. All three models obtain accuracy rates higher than $65\%$ on the validation set, which drop below $60\%$ on the test set. In terms of precision and recall, almost all of them achieve higher precision for the “popular” category on both validation and test sets, with one exception being the FastText + SVM method on the test set, where the precision on the two classes is comparable. A distinctive behavior of the three models is that the TF-IDF + RF obtains a higher recall for the “popular” category, while FastText + SVM and RoBERT + LR attain a higher recall for the “unpopular” category.

Table 3 also shows the results on the test set of our few-shot prompted LLM. While this approach exhibits a bias similar to our other baselines, favoring recall for unpopular predictions and precision for popular ones, its overall performance falls below that of a random chance classifier. This suggests a limitation in the generalization capacity of LLMs to the popularity prediction task, particularly for languages with limited online resources, such as Romanian.

We analyze the discriminative features learned by the fine-tuned Ro-BERT and by the FastText + SVM. The motivation behind this analysis is to validate that the decisions of these models are not based on some biases that escaped our data collection, but on actual data understanding.

For the Ro-BERT model, we use the Captum [14] library via its Layer Integrated Gradients method to infer valuable insights from the fine-tuned model. This technique delves into the BERT embeddings layer, attributing importance scores to individual input words which led to the final label prediction.

To find the words with higher influence on the decisions given by the SVM, we consider the cosine similarities between the primal weights of the SVM and the FastText embedding of each word. We sort the words based on the similarity values, and keep the first 10 and last 10 words from the sorted list as features for the positive (“popular”) and negative (“unpopular”) classes, respectively.

In Tables 4 and 5, we present a few examples of interesting patterns that were picked up by the models. In predicting post popularity, the Ro-BERT model demonstrates a bias toward content reflecting current trends, including news and events, and posts encouraging community engagement through calls to action. Conversely, references to proper nouns like city names or historical landmarks appear to hinder popularity, as do posts seeking community advice or expressing dissatisfaction with platitudes. Similar to Ro-BERT, we find that the SVM labels posts that share news as popular, and posts by people seeking advice as unpopular.

Furthermore, we extend the feature analysis for the SVM in order to determine the most discriminative words for the popular and unpopular classes. To achieve this, we determine the discriminative weight of each word based on the cosine similarity between the respective word embedding and the SVM weights. We sort the words according to their weights, and select the ones with the highest and lowest weights. In Table 6, we provide the five most discriminative words for the popular and unpopular classes, according to the SVM based on FastText features. We observe that posts mentioning “online” or “youtube” are more popular, likely because readers appreciate posts that provide links to YouTube videos. We also note the preference for posts that discuss particular cases/experiences, which are usually introduced by the word “cazul” (translated to “case” in English). On the other hand, posts that recommend searching on “google” are unpopular, as the readers consider such suggestions unhelpful. Moreover, discussing subjective perspectives, using the singular first person pronoun “eu”, is again unpopular, likely because the readers appreciate more objective posts.