Capturing Style in Author and Document Representation

Word embedding, popularized by [22], was then extended to document embedding by the same authors. More recent works [4] propose different aggregation functions of word embeddings, based on LSTM, Transformers, and Deep Averaging Networks, to build (short) document level representations. The aggregations is learnt through classification or document pairing. More recently, [25] proceed in a similar way by fine tuning a BERT model [6].

There are also specific works focusing on author embeddings. The Author Topic Model (ATM) [26] is a hierarchical graphical model, optimized through Gibbs sampling. It produces a distribution over jointly learnt topic factors that can be used as author features. Aut2vec [10] allows to learn representations of authors and documents that can separate true observed pairs and negative sampled (document, author) pairs. The distance between two representations modifies an activation function producing a probability that the pair is observed in the corpus. This approach concatenates two sub models: the Link Info model, which takes pairs of collaborating authors, and the Content Info model, which uses pairs of author and documents. It cannot infer representations of unseen documents and authors: the embeddings are parameters of an embedding layer. The Usr2vec model [2] learns author representation from pretrained word vectors. Authors use the same objective than [22], and add an author id to learn the representations.

While there is no consensual definition of writing style, it has always been a widely addressed research topic. In computational linguistic, the approach of [16] is often cited as a reference and gives the following definition: “Style is, on a surface level, very obviously detectable as the choice between items in a vocabulary, between types of syntactical constructions, between the various ways a text can be woven from the material it is made of.”, and the author to conclude further to the “impossibility of drawing a clean line between meaning and style”. That’s why style is commonly defined as every writing choice without semantic information.

Based on this definition, it is hard, if not impossible, to produce a clear annotated dataset classifying different writing style. The workaround in most studies is to identify the most useful stylistic features to associate an author to its production. It starts in the 19th century with [21] and the most basic features (e.g., word and punctuation frequencies, hapax legomena, average sentence length). More recent works focus on function words frequencies [41], hybrid variables such as character n-grams [34, 31] or even Part-Of-Speech (POS) and Name Entity Recognition (NER) tag frequencies, using authorship prediction as evaluation.

Several methods try to use these stylistic features to learn document representations. For example, [20] use Doc2Vec on documents of character trigrams annotated regarding their position in the word or if they contain punctuation (NGRAM Doc2Vec). According to the authors, it allows to capture both content and writing style. In an other work, words and POS tags embeddings are learnt together before passing them through a CNN to get a sentence representation [14]. Then these sentences are fed into an LSTM with a final attention layer to compute document representation. This model is trained on the authorship attribution task.

Some works claim to capture this information in an unsupervised manner. DBert-ft [13] fine-tunes DistilBERT on the authorship attribution task, assuming that an author writing style must be consistent over its documents, and thus, that this task allows to build a “stylometric latent space” when the model is trained on a reference set. Yet, for all above models, no author representation is explicitely learnt.

We deal with a set of documents, such as literature or blog posts. We assume each document is written by one author. Each document of indice $d$ is preprocessed to extract a vector $z_{d}^{f}$ of $r=300$ stylistic features following [35].

Our goal is threefold: i) We want to build author and document representations in the same space $\mathbb{R}^{r}$ such that their proximity captures their stylistic similarity (Figure 1), ii) We want to learn a measure of variability in style for each document and author , and iii) We want our model to incorporate an on-the-shelf pre-trained text encoder such as Sentence-BERT or USE to benefit from their complex language understanding, fine-tuned on the dataset at hand using the objective we have just defined. To do that, we build an architecture based on the Variational Information Bottleneck (VIB) framework.

The VIB framework is a variational extension of the Information Bottleneck principle [36] proposed by [1]. The general objective function is, for a set of observations $x$, to associate labels $y$ and latent representations $z$ of these observations:

$$\arg\max\limits_{z}I(z,y)-\beta I(z,x),$$

(1)

where I is the well-known Mutual Information measure, defined as:

$$I(x,y)=\int\int p(x,y)\log\frac{p(x,y)}{p(x)p(y)}d_{x}d_{y}.$$

(2)

Information Bottleneck aims at maximally compressing the information in $z$, such that $z$ is highly informative regarding the labels, i.e. $z$ can be used to predict the labels $y$. With $y$ being a set of relevant stylistic features, we would like to maximize the stylistic information captured by the representation, while minimizing the semantic one. $\beta\geq 0$ is a hyper-parameter that controls the balance between the two sub-objectives.

[23] propose to use this framework to learn probabilistic representations of images. They leverage an instance of this framework based on siamese networks with a (soft) contrastive loss objective function, to separate positive observed pairs of images ($y=1$) and negative examples ($y=0$). We extend this model to document and author embedding with stylistic constraint. Each author $a$ (resp. document $d$) is associated to a stochastic representation $z_{a}$ (resp. $z_{d}$) that is unobserved (i.e., latent). Additionally, each document is associated to a stylistic feature vector $z_{d}^{f}$ that is beforehand extracted from the corpus with usual NLP toolkits. We assume that the dimensions of $z_{a}$, $z_{d}$ and $z_{d}^{f}$ are the same ($r$).

We build a set of pairs $(a,d)$ with label $y_{a}=1$ if $a$ wrote $d$. We additionally draw $k$ negative pairs $(a^{\prime},d)$ for each observed pair, associated with label $y_{a}=0$, where $a^{\prime}$ is not an author of $d$. The encoding laws ($p(z|x)$) for authors and documents are normal laws. To capture stylistic information, we also build a set of pairs $(d,d)$ with label $y_{f}=1$ and we draw $k$ negative pairs $(d,d^{\prime})$ for each observed pair, associated with label $y_{f}=0$. These pairs are used to train the stylistic objective : the representation $z_{d}$ of a document should be close to its feature vector $z_{d}^{f}$.

We learn the following parameters for each author $a$: mean $\mu_{a}$ and diagonal variance matrix with diagonal $\sigma_{a}^{2}$ (these are embedding layers). For a document $d$, we use a trainable text encoder to map a document’s content to a vector $d_{0}\in\mathbb{R}^{r_{0}}$. We then build the document mean $\mu_{d}=f(d_{0})\in\mathbb{R}^{r}$ and diagonal variance matrix with diagonal $\sigma_{d}^{2}=g(d_{0})\in\mathbb{R}^{r}$. As we will show later, the dimension $r$ should match the number of stylistic features to gain in comprehension of the learning space, but the text encoder can output vectors of any dimension (here, $r_{0}$). Following [1, 23], $f$ and $g$ are neural networks. We give more details on $f$, $g$ (the “encoding functions”), and the text encoder later.

Following [23], the probability of a label is the soft contrastive loss:

$$\begin{split}q(y_{a}=1|z_{a},z_{d})&=\sigma(-c_{a}||z_{a}-z_{d}||_{2}+e_{a})\\ q(y_{f}=1|z_{d},z_{d}^{f})&=\sigma(-c_{f}||z_{d}-z_{d}^{f}||_{2}+e_{f}),\end{split}$$

(4)

Here, $\alpha=0$ will produce representations that well predict the author-document relation but will not capture the stylistic features of the documents, as shown by [35]. With $\alpha=1$, on the contrary, the model will simply bring document embeddings closer to their feature vectors. Hence, the value of $\alpha$ needs to be carefully tuned on the dataset, regarding if the corpus is writing style specific or not thanks to domain knowledge.

Eventually, computing the expected values in Eq.(5) is intractable for a wide range of encoders. We therefore approximate it by sampling $L$ examples by observation (here, a triplet document, author, feature vector), following $p(z|x)$ as done in [23]. We get (the same goes for feature vector/documents pairs) :

$$\begin{split}\mathbb{E}[\log q\left(y_{a}|z_{a},z_{d}\right)]&\approx\frac{1}{L}\sum_{l=1}^{L}{\log q(y_{a}|z^{(l)}_{a},z^{(l)}_{d})}\\ \end{split}$$

(6)

We then use the reparametrization trick, following what is done in VAE [17]:

$$z^{(l)}_{a}=\mu_{a}+\sigma_{a}\odot\epsilon\text{,}\hskip 5.0ptz^{(l)}_{d}=\mu_{d}+\sigma_{d}\odot\epsilon\hskip 5.0pt\text{with}\hskip 5.0pt\epsilon\sim\mathcal{N}(0,1)$$

This loss can now be minimized using backpropagation. In Figure 2, we show a schematic representation of our model, called VADES for Variational Author and Document Representations with Style.

The entering bloc of our model for documents is a text encoder, mapping a document in natural language to a vector in $\mathbb{R}^{r}$. Many deep architectures could be used here and trained from scratch. Nevertheless, we propose to use a pretrained text encoder.

Models that are pretrained on large datasets are now easily available online¹¹1e.g., https://huggingface.co/models. They have been proved successful on many NLP tasks with a simple fine-tuning phase (the only constraint being to avoid catastrophic forgetting). Additionally, the VIB framework allows to naturally introduce a pretrained text encoder as shown by [19]. The encoder’s output should then be mapped to document mean and variance. Both [19, 12] map the text encoder output to the document’s mean (the $f$ function) and variance (the $g$ function) using a Multi Layer Perceptron (MLP). This approach is simple, and fast. In our experiments, we build $f$ and $g$ as two-layer MLP with tanh and linear activation with same input and intermediate dimensions ($r_{0}$). Note that the output dimension of $f$ and $g$ should be the same as the number of stylistic features ($r$).

Several constraints arise regarding the pretrained encoder itself. We would like our model to be able to capture stylistic information from a given document. As shown in [5, 35], state-of-the-art models trained on large datasets already capture complex grammatical and syntactic notions in their representations, and therefore have the explanatory power requested for our objective. Moreover, our model must be able to deal with long text as it will be used in a literary context. Processing novels, dramas, essays, where writing style interferes the most. This is a serious problem: for example, the widely used BERT model is limited to 512 tokens. Alternative models such as [39] allow to apply transformers to long documents. To circumvent this issue, we use the Deep Averaging Network implementation of the Universal Sentence Encoder (USE) from [4]. It has several advantages over the latter works: it gives no length constraint, it is faster than transformer-based methods and it outperforms Sentence-BERT on stylistic features prediction [35]. The test of other encoder models is left to future works. Finally, note that our model is language agnostic (as it depends on a out-of-the-box text encoder) and can infer representations for unseen documents.

The IMDb (Internet Movie Database) corpus is one of the most used ones regarding the authorship attribution task. It was introduced by [32] and is composed of $271,000$ movie reviews from $22,116$ online users. However, most of the works are evaluated on the reduction of this dataset to only 62 authors with $1000$ texts for each (IMDb62). Thus, we benchmark our model on IMDb62. As shown later, the task of authorship attribution on this corpus is more or less solved, due to the low number of authors.

The Project Gutenberg is a multilingual library of more than 60,000 e-books for which U.S. copyright has expired. It is freely available and started in 1971. We gathered the corpus using [11]. Most of the books are classical novels, dramas, essays, etc. from different eras, which is relevant when studying writing style and represents quite well our context of application. To keep the most authors possible, we randomly sample 10 texts for each author with such a production, leaving 664 authors in our Reduce Project Gutenberg Dataset (R-PGD) (10 times more than IMDB). To be able to deal with such works, we only keep the 200 first sentences of each book.

This dataset is composed of 681,288 posts from 19,320 authors gathered in the early 2000s by [30]. There are approximately 35 posts and 7,250 words by user. We only take 500 bloggers with at least 50 blogposts to build our reduced dataset of the Blog Authorship Corpus (R-BAC). This dataset is also used in several authorship attribution benchmark, only keeping the top 10 or 50 authors with most productions. We will also test our model on these extraction of the corpus.

These two last datasets (PGD and BAC) represent two common uses of author embedding (classic literature and web analysis) with a large number of authors. Usual datasets for authorship attribution (CCAT50, NYT, IMDb62) contain far less classes, further from our context of a web extracted corpus (from Blogger or Wordpress for example)… They are also stylistically and structurally different, allowing to evaluate our approach on various textual formats. For each dataset, we perform a 80/20 train-test stratified split.

In this section, we present implementation details for our method and competitors. For the encoder functions $f$ and $g$, we use the architectures presented in the previous section with batch normalization and dropout equal to 0.2 with L2 regularization ($1e-5$). Grid-search parameters are detailed in Table 2. For $L$, we obtain a good trade-off between accuracy and speed with $L=10$, as we quickly reach a plateau of performance when increasing its value. We can summarize the tuning of $\alpha$ as follows:

• •

  $\alpha=0$ implies no feature loss and stylistic information,

• •

  $\alpha=0.5$ gives the same importance to feature loss and author loss,

• •

  $\alpha=0.9$ pushes feature loss to boost style detection.

We train the model for 15 epochs on R-PGD and R-BAC, and for 5 epochs on IMDB, BAC10 and BAC50 as the number of authors is around ten times smaller. We use a partition of 2 GPUs V100. On a single GPU, training the model on the R-PGD dataset takes around 10 hours. In the following section, we report the results for the best version of VADES only. As an ablation study, to justify the use of both the VIB framework and stylistic features, we compare our model with and without these components (respectively called VADES no-VIB and VADES ($\alpha=0$)). The code is available on github and will be shared if the paper is accepted. All the datasets are available online.

We compare our model with several baselines. We use [20] (NGRAM Doc2Vec), a simple average based version of USE [4] (a document representation is built from the average of its sentence encoding, and an author representation is an average of its documents). We also compare our approach to DBert-ft [13], a document embedding method where DistilBERT is fine-tuned on the authorship attribution task. The author embeddings are built by averaging the representations of the documents it wrote. We use the parameters detailed in the authors’ implementation²²2https://github.com/hayj/DBert-ft.

We first evaluate the baselines and VADES regarding how well each method captures writing style. As writing style is a complex and a still discussed notion, there is no supervised dataset to evaluate how a model can grasp it. We therefore use a proxy task that consists in predicting stylistic feature from the latent representations. We follow the experimental protocol of [35]. The stylistic features are extracted using spacy word and sentence tokenizer, POS-tagger and Name Entity Recognition, spacy English stopwords and nltk CMU Dictionary. For each author, we aim to predict the value of all stylistic features from their embeddings. Each feature is standardized before regression. We use an SVR with Radial Basis Function (rbf) kernel as it offers both quick training time and best results among other kernels in our experiments. We evaluate models using Mean Squared Error (MSE) following a 10-fold cross validation scheme.

Secondly, we perform authorship attribution, the task of predicting the author of a given document. We compare VADES with several other authorship attribution methods even though they do not necessarily perform representation learning. Each dataset is split into train and test sets with a 80/20 ratio. For our model, we repeated 5 times the evaluation scheme. For embedding method without classification head, we associate each document with its most plausible author using cosine similarity. We use accuracy to evaluate these results (the percentage of correctly predicted authors out of all data points).

As explained earlier, we use the author embeddings to perform regression and predict each stylistic features. As shown in Table 4, only using a simple logistic regression on these stylistic features allows to reach decent scores in authorship attribution, close to these of Universal Sentence Encoder, which is a state-of-the-art method in sentence embedding. As they contain strictly no topic information, it demonstrates how good they are as a proxy of writing style. Thus, a model able to capture them is able to capture writing style.

Results on the style MSE metric are shown in Table 3. As expected, our model easily outperforms every baseline on all axes. DBert-ft, only trained on the authorship attribution objective performs the worst. Even though this approach is based on fine-tuned language models which already capture syntactic and grammatical notions [5], this is not the information that seems to be retained by the network when trained on the author attribution task. This is consistent with what was shown in [35]. The models may mainly focus on the semantic information to predict author-document relation. Interestingly, we observe that a simple average of USE representations performs quite well, which confirms that it can successfully capture complex linguistic concepts. VADES is guided by the feature loss to do so.

On a qualitative note, we present two additional visualisations to underline the strong advantage of VADES for linguistic and stylistic applications. In Figure 5, we present a T-SNE 2D projection of the books of the R-PGD dataset colored by their publication year. A clear color gradient appears, demonstrating that our model can grasp the evolution of writing style through the last centuries. Figure 1 shows a toy example of a T-SNE 2D projection of well-known authors from the R-PGD dataset and their books (we use $\alpha=0.5$). The objects are distributed in the space across clear author specific clusters. The most interesting observation is related to documents that are outside of their author cluster: Thus Spake Zarathustra: A Book for All and None by Nietzsche is a philosophical poem, closer to Hugo, while the rest of its production is mostly essays. The same conclusion goes with The Power of Darkness by Tolstoï, a 5 acts drama, whose embedding is closer to Shakespeare than to Tolstoï novels. The version of Hamlet presented here is fully commented, and thus is closer to analytical and philosophical works of Nietzsche and Plutarch as shown on the figure. We also represent the variance learnt by the model in the size of the author dot. Hugo, who wrote famous novels as well as poetry and dramas, has a greater variance than other authors.

As we use the $L_{2}$ distance between document representations and stylistic feature vectors, each of the 300 embedding axes correspond to one given stylistic feature. The soft contrastive loss allows to ensures the $L_{2}$ constraint (bringing document embedding and stylistic features vectors closer) while being more flexible than a simple regression loss. When experimenting with the latter, the task showed up to be too hard and disadvantageous regarding both authorship attribution scores and writing style loss.

On Figure 3, we show the Pearson correlation score between the $i^{th}$ stylistic feature and the corresponding embedding axis. These correlation values are always maximum for each feature regarding every other embedding coordinate. To further illustrate the interpretability of the embedding space, Figure 4 shows a selection of 4 stylistic features, the representation value of the matching coordinate for each author. The representation space learnt by VADES is interpretable in terms of writing style. In the context of a multidisciplinary project, involving several searchers in literature and linguistic this is a significant added value.

Results on the authorship attribution task for IMDb62 and Blog Authorship Corpus are presented respectively in Table 4 against state-of-the-art solutions (not necessarily embedding models). On both datasets, our model ranks in top 4, outperforming recent competitors while authorship attribution is not its main task. Our model is outpaced by Syntax CNN [40], DBert-ft [13] and BertAA [8], two variants of BERT fine tuned on the authorship attribution task. As shown by [8], BERT and DistilBERT are really tailored for balanced datasets with short texts such as IMDB62 and Blog Authorship Corpus. The DBert-ft model splits every document in 512 chunks during training, building an even bigger corpus with important improvement, but it is hardly reproducible with our feature loss. BertAA feeds encoded documents from a finetuned BERT together with a set of stylistic features and of most frequent bi-grams and tri-grams to a Logistic Regression. It clearly allows to better perform on Blog Authorship Corpus as this dataset is a mix of several genres and styles, compared to IMDB62 concerning only movie reviews. This confirms our use of stylistic features. Syntax CNN encodes each sentence of a document separately with its syntax. Unfortunately, this model was hardly reproducible and cannot be tested in feature regression using intermediate representation. For VADES  lower values of $\alpha$ allow to reach the best accuracy in authorship attribution on these datasets. Additional information bring by stylistic features benefit to the authorship attribution when texts are longer.

We here compare our model to no-VIB and without feature loss. Both variations underperform on both tasks. First, the VIB paradigm offers more versatility than fixed document and author representation which is key to grasp a complex notion such as writing style. Then, the feature loss brings additional information for authorship prediction, as shown by BertAA, which use it to improve BERT classification results. Here, our framework enable to use it directly for document and author embeddings. On Figure 6, we evaluate the influence of $\alpha$ which balances the importance given to author loss and feature loss on both feature regression and authorship attribution. Adding just a few stylistic features information ($\alpha=0.1$) allows to improve the precision of our model in authorship attribution. It forces the model to extract discriminant stylistic information from the input. Surprisingly the same phenomenon appears when shutting down the author loss ($\alpha=1$). It creates a deterioration of the style score as authors tend to use a consistent writing style among their documents. Thus gathering a writer with its documents representation also helps to capture its writing habits. ([13] call it the “Intra-author consistency”).