\section{Introduction}

Fine-tuning pre-trained large-scale language models (LMs) is the dominant paradigm of current NLP. The LMs proved to be a versatile technology that can help to solve an array of NLP tasks, such as parsing, machine translation, text summarization, sentiment analysis, semantic similarity etc. The LMs can be used for tasks on various levels of linguistic complexity (syntactic, semantic, etc.) but also with various types of data modalities (text classification, text generation, text comparison, etc.). As such, it seems to be vital for any speech community to develop a proper model for their language and thus push the limits of what NLP can do for them.

In this paper, we introduce the first Slovak-only transformers-based language model trained with a non-trivial corpus called \textit{SlovakBERT}\footnote{Available at \url{https://github.com/gerulata/slovakbert}}. Although several multilingual models already support Slovak, we believe that developing a Slovak-only model is still important, as it can lead to better results and more compute-wise and memory-wise efficient processing of Slovak language. \textit{SlovakBERT} has RoBERTa architecture~\cite{roberta} and it is trained on a Web-crawled corpus.

We evaluate \textit{SlovakBERT} on various existing datasets to study how well it handles different tasks. We also compare it to other available LMs (mainly multilingual) and other existing approaches. The tasks we study are: part-of-speech tagging, semantic textual similarity, sentiment analysis and document classification. As a by-product of our experimentation, we have developed and published the best performing models for selected tasks. These might be used by other researchers or Slovak NLP practitioners in the future as strong baseline.

Our main contributions in this paper are:

\begin{itemize}
    \item We trained and published the first proper Slovak-only LM on a dataset of non-trivial size that we collected.
    \item We evaluated the LM on a series of Slovak NLP tasks.
    \item We published several fine-tuned models based on this LM, namely a part-of-speech tagger, sentiment analysis model and sentence embedding model.
    \item We published several additional datasets for multiple tasks, namely sentiment analysis test sets and semantic similarity translated dataset.
\end{itemize}

The rest of this paper is structured as follows: In Section~\ref{sec:related} we discuss related work about language models and their language mutations. In Section~\ref{sec:model} we describe the corpus crawling efforts and how we train our LM with it. In Section~\ref{sec:evaluation} we evaluate the model with four NLP tasks. In Section~\ref{sec:concl} we conclude our findings.

\section{Related Work}\label{sec:related}

\subsection{Language Models}

Most of the LMs are currently based on self-attention layers called \textit{transformers}~\cite{transformer}. The models differ in the details of their architecture, as well as in the task they are trained with~\cite{whichbert}. The most common task is the so called \textit{masked language modeling}~\cite{bert}, where certain parts of the sentence are masked and the model is expected to fill these parts with the original tokens. The models like these are useful mainly as backbones for further fine-tuning. Another approach is to train a generative model~\cite{gpt2}, that always predicts the next word in the sequence, which can be used for various text generation tasks. Sequence-to-sequence models~\cite{t5} are also an option, which is in a sense a combination of the previous two approaches. They have an encoder that encodes an arbitrary input and then a decoder that is used to generate text as an output. For all these types, variants exist to make the underlying LMs more efficient~\cite{electra, tinybert}, able to handle longer sentences~\cite{longformer} or fulfill various other requirements.

\subsection{Availability in Different Languages}

English is the most commonly used language in NLP, and a \textit{de facto} standard for experimental work. Most of the proposed LM variants are indeed trained and evaluated only on English. Other languages usually have at most only a few LMs trained, usually with a very safe choice of model type (e.g. BERT models or RoBERTa models are popular). Languages with available native models are e.g. French~\cite{french}, Dutch~\cite{dutch} or Arabic~\cite{arabic}. There are also models for related Slavic languages, notably Czech~\cite{czech} and Polish~\cite{polish}.

There is no Slovak-specific large scale LM available so far. There is a Slovak version of WikiBERT model~\cite{wikibert}, but it is trained only on texts from Wikipedia, which is not a large enough corpus for proper language modeling at this scale. The limitations of this model will be shown in the results as well.

\subsection{Multilingual Language Models}

Multilingual LMs are sometimes proposed as an alternative to training language-specific LMs. These LMs can handle more than one language. In practice, they are often trained with more than 100 languages. Training them is more efficient than training separate models for all the languages. Additionally, cross-lingual transfer learning might improve the performance with the languages being able to learn from each other. This is especially beneficial for low-resource languages.

The first large-scale multilingual LM is MBERT~\cite{bert}  trained on 104 languages. The authors observed that by simply exposing the model to data from multiple languages, the model was able to discover the multilingual signal and it spontaneously developed interesting cross-lingual capabilities, i.e. sentences from different languages with similar meaning also have similar representations. Other models explicitly use multilingual supervision, e.g. dictionaries, parallel corpora or machine translation systems~\cite{xlm, unicoder}

\section{Training}\label{sec:model}

In this section we describe our own Slovak masked language model, what data were used for training, what is the architecture of the model and how it was trained.

\subsection{Data}

We used a combination of available corpora and our own Web-crawled corpus as our training data. The available corpora we used were: Wikipedia (326MB of text), Open Subtitles (415MB) and OSCAR corpus (4.6GB). We crawled \texttt{.sk} top-level domain webpages, applied language detection and extracted the title and the main content of each page as clean texts without HTML tags (17.4GB). The text was then processed with the following steps:

\begin{itemize}
    \item URL and email addresses were replaced with special tokens.
    \item Elongated interpunction was reduced, i.e. if there were sequences of the same interpunction character, these were reduced to one character (e.g. \texttt{--} to \texttt{-}).
    \item Markdown syntax was deleted.
    \item All text content in braces \texttt{\{.\}} was eliminated to reduce the amount of markup and programming language text.
\end{itemize}

We segmented the resulting corpus into sentences and removed duplicates to get 181.6M unique sentences. In total, the final corpus has 19.35GB of text.

\subsection{Model Architecture and Training}

The model itself is a RoBERTa model~\cite{roberta}. The details of the architecture are shown in Table~\ref{tab:lms} in the \texttt{SlovakBERT} column. We use BPE~\cite{bpe} tokenizer with the vocabulary size of 50264. The model was trained for 300k training steps with a batch size of 512. Samples were limited to a maximum of 512 tokens and for each sample we fit as many full sentences as possible. We used Adam optimization algorithm~\cite{adam} with $5 \times 10^{-4}$ learning rate and 10k warmup steps. Dropout (dropout rate $0.1$) and weight decay ($\lambda = 0.01$) were used for regularization. We used \texttt{fairseq}~\cite{fairseq} library for training, which took approximately 248 hours on 4 NVIDIA A100 GPUs. We used 16-bit float precision.

\section{Evaluation}\label{sec:evaluation}

In this section, we describe the evaluation procedure and results of \textit{SlovakBERT}. We use two main methods to examine the performance of \textit{SlovakBERT}, but also of various other LMs:

\begin{enumerate}
    \item \textit{Downstream performance.} We fine-tune the LMs for various NLP tasks and we analyze the achieved results. We compare the results with existing solutions based on other approaches, e.g. rule-based solutions or solutions based on word embeddings.
    \item \textit{Probing.} Probing is a technique that aims to measure how useful the information contained within individual layers of the LM is. We can check how important  the individual layers for various tasks are. We use simple \textit{linear probes} in our work, i.e. the hidden representations from the LMs are used as features for linear classification.
\end{enumerate}

We conducted the evaluation on four different tasks: part-of-speech tagging, semantic textual similarity, sentiment analysis and document classification. For each task, we introduce the dataset that is used, various baselines solutions, the LM-based approach we took and the final results for the task.

\subsection{Evaluated Language Models}

We evaluate and compare several LMs that support Slovak language to some extent:

\begin{itemize}
    \item \textbf{XLM-R}~\cite{xlm-r} - XLM-R is a suite of RoBERTa-style LMs. The models support 100 languages, including Slovak. Training data are based on CommonCrawl Web-crawled corpus. Slovak part has 23.2 GB (3.5B tokens). The XLM-R models differ in their size, ranging from Base model with 270M parameters to XXL model with 10.7B parameters.
    \item \textbf{MBERT}~\cite{mbert} - MBERT is a multilingual version of the original BERT model trained with Wikipedia-based corpus containing 104 languages. Authors do not mention the amount of data for each language, but considering the size of Slovak Wikipedia, we assume that the Slovak part is in the tens of millions of tokens.
    \item \textbf{WikiBERT}~\cite{wikibert} - WikiBERT is a series of monolingual BERT-style models trained on dumps of Wikipedia. The Slovak model was trained with 39M tokens.
\end{itemize}

Note that both XLM-R and MBERT models were trained in cross-lingually unsupervised manner, i.e. no additional signal about how sentences or words from different languages relate to each other was provided. The models were trained with simple multilingual corpora, although language balancing was performed.

In Table~\ref{tab:lms} we provide a basic quantitative measures of all the models. We compare their architecture and training data. We also studied the tokenization productivity on texts from \textit{Universal Dependencies}~\cite{ud} train set. We show the average length of tokens for each model. Longer tokens are considered to be better, because they can be more semantically meaningful. Also, with longer tokens, we need fewer of them. Model that needs more tokens will take longer to compute, since a self-attention layer has a $O(N^2)$ time complexity w.r.t. input length $N$. We also show how many tokens were used (effective vocabulary) for the tokenization in this particular dataset. Multilingual LMs have smaller portion of their vocabulary used, since they contain many tokens used in other languages and not in Slovak. These tokens are redundant for Slovak text processing.

\begin{table}[]
    \centering
    \tiny
    \begin{tabular}{l|l|l|l|l|l}
\textbf{Model} & SlovakBERT & XLM-R-Base & XLM-R-Large & MBERT & WikiBERT \\ \hline
Architecture & RoBERTa & \multicolumn{2}{c|}{RoBERTa} & BERT & BERT \\
Num. layers & 12 & 12 & 24 & 12 & 12 \\
Num. attention head & 12 & 12 & 16 & 12 & 12 \\
Hidden size & 768 & 768 & 1024 & 768 & 768 \\
Num. parameters & 125M & 278M & 560M & 178M & 102M \\
Languages & 1 & 100 & 100 & 104 & 1 \\
Training dataset size (tokens) & 4.6B & \multicolumn{2}{c|}{167B} & n/a & 39M \\
Slovak dataset size (tokens) & 4.6B & \multicolumn{2}{c|}{3.2B} & 25-50M & 39M \\
Vocabulary size & 50K & \multicolumn{2}{c|}{250K} & 120K & 20K \\
Average token length * & 3.23 & \multicolumn{2}{c|}{2.84} & 2.40 & 2.70 \\
Effective vocabulary * & 16.6K & \multicolumn{2}{c|}{9.6K} & 6.7K & 5.8K \\
Effective vocabulary (\%) * & 33.05 & \multicolumn{2}{c|}{3.86} & 5.62 & 29.10 \\

    \end{tabular}
    \caption{Basic statistics about the evaluated LMs. *Data are calculated based on \textit{Universal Dependencies} dataset.}
    \label{tab:lms}
\end{table}

\subsection{Part-of-Speech Tagging}

The goal of part-of-speech (POS) tagging is to assign a certain POS tag from the predefined set of possible tags to each word. This task mainly evaluates the syntactic capabilities of the models.

\subsubsection{Data}

We use Slovak Dependency Treebank from \textit{Universal Dependencies} dataset~\cite{slovak-ud, ud} (UD). It contains annotations for both Universal (UPOS, 17 tags) and Slovak-specific (XPOS, 19 tags) POS tagsets. We mainly work with UPOS, but we use XPOS for comparison with other systems as well. XPOS uses a more complicated system and it encodes not only POS tags, but also other morphological categories in the label. In this work, we only use the first letter from each Slovak XPOS label, which corresponds to a typical POS tag. The tagsets and their relations are shown in Table~\ref{tab:pos-tagsets}.

\begin{table}[]
    \centering
    \begin{tabular}{l|l||l|l}
\multicolumn{2}{c||}{\textbf{XPOS}} & \multicolumn{2}{c}{\textbf{UPOS}}  \\ \hline
 \textbf{Tag} & \textbf{Description} & \textbf{Tag} & \textbf{Description} \\ \hline \hline
 A & adjective & \multirow{2}*{ADJ} & \multirow{2}*{adjective} \\ \cline{1-2}
 G & participle & & \\ \hline
 E & preposition & ADP & adposition \\ \hline
 D & adverb & ADV & adverb \\ \hline
 Y & conditional morpheme & \multirow{2}*{AUX} & \multirow{2}*{auxiliary} \\ \cline{1-2}
 \multirow{2}*{V} & \multirow{2}*{verb} & & \\ \cline{3-4}
 & & VERB & verb \\ \hline
 \multirow{2}*{O} & \multirow{2}*{conjuction} & CCONJ & coordinating conjunction \\ \cline{3-4}
 & & SCONJ & subordinating conjunction \\ \hline
  \multirow{2}*{P} & \multirow{2}*{pronoun} & DET & determiner \\ \cline{3-4}
 & & \multirow{2}*{PRON} & \multirow{2}*{pronoun} \\ \cline{1-2}
 R & reflexive pronoun & & \\ \hline
 J & interjection & INTJ & interjection \\ \hline
   \multirow{2}*{S} & \multirow{2}*{noun} & NOUN & noun \\ \cline{3-4}
 & & PROPN & proper noun \\ \hline
 N & numeral & \multirow{2}*{NUM} & \multirow{2}*{numeral} \\ \cline{1-2}
 0 & digit & & \\ \hline
 T & particle & PART & particle \\ \hline
 Z & punctuation & PUNCT & punctuation \\ \hline
 W & abbreviation & \multirow{4}*{X} & \multirow{4}*{other} \\ \cline{1-2}
 Q & unidentifiable & & \\ \cline{1-2}
 \# & non-word element & & \\ \cline{1-2}
 \% & citation in foreign language & & \\ \hline
 & & SYM & symbol \\
    \end{tabular}
    \caption{Slovak POS tagsets and their mapping~\cite{slovak-ud}.}
    \label{tab:pos-tagsets}
\end{table}

\subsubsection{Previous work}

Since Slovak is an official part of the UD dataset, systems that attempt to cover multiple or all UD languages can often support Slovak as well. The following systems were trained on UD data and support both UPOS and XPOS tagsets:

\begin{itemize}
    \item \textbf{UDPipe 2}~\cite{udpipe} - A deep learning model based on multilayer bidirectional LSTM architecture with pre-trained Slovak word embeddings. The model supports multiple languages, but the models themselves are monolingual.
    \item \textbf{Stanza}~\cite{stanza} - Stanza is a very similar model to UDPipe, it is also based on multilayer bidirectional LSTM with pre-trained word embeddings.
    \item \textbf{Trankit}~\cite{trankit} - Trankit is based on adapter-style fine-tuning~\cite{adapter} of XLM-R-Base. The adapters are fine-tuned for specific languages and they are able to handle multiple tasks at the same time.
\end{itemize}

\subsubsection{Our Fine-Tuning}

We use a standard setup for fine-tuning the LMs for token classification. The final layer of an LM that is used to predict the masked tokens is discarded. A classifier linear layer with dropout is used in its place to generate POS tag logits for each token. These logits are then transformed to a probability vector with softmax function and a cross-entropy is calculated for each token. The loss function for batch of samples is defined as an average cross-entropy across all the tokens. For inference, we simply pick the class with the highest probability for each token. Note that there is a discrepancy between what we perceive as words and what the models use as tokens. Some words might be tokenized into multiple tokens. In that case, we only make the prediction on the first token and the final classifier layer is not applied to the subsequent tokens for this word. We use \texttt{Hugging Face Transformers} library for LM fine-tuning.

We use similar setup for probing, but with two changes: (1) We freeze all the weights apart from the classifier layer, and (2) we remove several top layers from the LM, i.e. instead of making predictions from the topmost layer, we make them from other layer instead. This way we can analyze how well the representations generated on the given layer work.

\subsubsection{Results}

We have performed a random hyperparameter search with \textit{SlovakBERT}. The range of individual hyperparameters is shown in Table~\ref{tab:pos-hparams}. We have found out that weight decay is a beneficial regularization technique, while label smoothing proved itself to be inappropriate for our case. Other hyperparameters showed to have very little reliable effect, apart from the learning rate, which proved to be very sensitive. We have not repeated this tuning for other LMs, instead, we only tuned the learning rate. We have found out that it is appropriate to use learning rate of $10^{-5}$ for all the models, but XLM-R-Large. XLM-R-Large, the biggest model we tested, needs a smaller learning rate of $10^{-6}$.

\begin{table}[]
    \centering
    \begin{tabular}{l|r|r}
  \textbf{Hyperparameter} & \textbf{Range} & \textbf{Selected} \\ \hline
  Learning rate & $[10^{-7}, 10^{-3}]$ & $10^{-5}$ \\
  Batch size & $\{8, 16, 32, 64, 128\}$ & $32$ \\
  Warmup steps & $\{0, 500, 1000, 2000\}$ & $1000$ \\
  Weight decay & $[0, 0.1]$ & $0.05$ \\
  Label smoothing & $[0, 0.2]$ & $0$ \\
  Learning rate scheduler & Various\footnotemark & \texttt{linear} \\
    \end{tabular}
    \caption{Hyperparameters used for POS tagging. Adam was used as an optimization algorithm.}
    \label{tab:pos-hparams}
\end{table}
\footnotetext{See the list of schedulers supported by Hugging Face Transformers library.}

The results for POS tagging are shown in Table~\ref{tab:pos-results}. We report accuracy for both XPOS and UPOS tagsets. WikiBERT seems to be the worst-performing LM, probably because of its small training set. \textit{SlovakBERT} seems to be on par with larger XLM-R-Large. Other models lag behind slightly. From existing solutions, only transformers-based Trankit seems to be able to keep up.

\begin{table}[]
    \centering
    \begin{tabular}{l|r|r}
 \textbf{Model} & \textbf{UPOS} & \textbf{XPOS} \\ \hline
 UDPipe 2.0 & 92.83 & 94.74 \\
 UDPipe 2.6 & 97.30 & 97.87 \\
 Stanza & 96.03 & 97.29 \\
 Trankit & 97.85 & 98.03 \\ \hline
 WikiBERT & 94.41 & 96.54 \\
 MBERT & 97.50 & 98.03 \\
 XLM-R-Base & 97.61 & 98.23 \\
 XLM-R-Large & \textbf{97.96} & 98.34 \\ \hline
 SlovakBERT & 97.84 & \textbf{98.37} \\
    \end{tabular}
    \caption{Results for POS tagging (accuracy).}
    \label{tab:pos-results}
\end{table}

To make sure that the results are statistically significant, we have performed McNemar test among all the models. We have used the information about whether each token was predicted correctly or incorrectly as the observed variables. The results are shown in Table~\ref{tab:pos-mcnemar}. We can see that the differences between the top-performing models are mostly not considered statistically significant.

\begin{table}[]
    \centering
    \tiny
    \begin{tabular}{l|r|r|r|r|r|r|r|r|r}
     & \rotatebox{90}{UDPipe 2.0} & \rotatebox{90}{UDPipe 2.6} & \rotatebox{90}{Stanza} & \rotatebox{90}{Trankit} & \rotatebox{90}{WikiBERT} & \rotatebox{90}{MBERT} & \rotatebox{90}{XLM-R-Base} & \rotatebox{90}{XLM-R-Large} & \rotatebox{90}{SlovakBERT} \\
UDPipe 2.0 & - & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\
UDPipe 2.6 & 0.0000 & - & 0.0000 & 0.0001 & 0.0000 & \textbf{0.5345} & \textbf{0.0543} & 0.0000 & 0.0057 \\
Stanza & 0.0000 & 0.0000 & - & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\
Trankit & 0.0000 & 0.0001 & 0.0000 & - & 0.0000 & 0.0006 & \textbf{0.0105} & \textbf{0.1776} & \textbf{0.2693} \\
WikiBERT & 0.0000 & 0.0000 & 0.0000 & 0.0000 & - & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\
MBERT & 0.0000 & \textbf{0.5345} & 0.0000 & 0.0006 & 0.0000 & - & \textbf{0.2214} & 0.0000 & \textbf{0.0302} \\
XLM-R-Base & 0.0000 & \textbf{0.0543} & 0.0000 & \textbf{0.0105} & 0.0000 & \textbf{0.2214} & - & 0.0001 & \textbf{0.2652} \\
XLM-R-Large & 0.0000 & 0.0000 & 0.0000 & \textbf{0.1776} & 0.0000 & 0.0000 & 0.0001 & - & \textbf{0.0166} \\
SlovakBERT & 0.0000 & 0.0057 & 0.0000 & \textbf{0.2693} & 0.0000 & \textbf{0.0302} & \textbf{0.2652} & \textbf{0.0166} & - \\
    \end{tabular}
    \caption{Statistical significance from McNemar test applied on POS results. Bold are results, where the difference between the two models is not deemed statistically significant ($p > 0.001$).}
    \label{tab:pos-mcnemar}
\end{table}

We also analyzed the dynamics of the LM fine-tuning. We analyzed the performance for various checkpoints of our LM (checkpoints were made after 1000 training steps). We can see in Figure~\ref{fig:pos}, that \textit{SlovakBERT} was saturated w.r.t POS performance quite soon, after approximately 15k steps. We stopped the analysis after the first 125k steps, since the results seemed to be stable. Similar results for probing can be seen in the same figure. We show the performance for all the layers for selected checkpoints. Again, we can see a rapid saturation of the model. Note that the performance on layers peaks quite soon at layer 6 and then plateaus. The last layers even have degraded performance. This shows, that the morphosyntactic information needed for POS tagging is stored and processed mainly in the middle part of the model. This is in accord with the current knowledge about how LMs work, i.e. that they process the text in a bottom-up manner~\cite{bert-pipeline}.

\begin{figure}[h]
\centering
\includegraphics[width=12cm]{figures/fig_pos.pdf}
\caption{Analysis of POS tagging learning dynamics. \textit{Left:} Accuracy after fine-tuning for several checkpoints. \textit{Right:} Accuracy of probes on all the layers with selected checkpoints.}\label{fig:pos}
\end{figure}

\subsection{Semantic Textual Similarity}

Semantic textual similarity (STS) is an NLP task where a similarity between pairs of sentences is measured. In our work, we train the LMs to generate sentence embeddings and then we measure how much the cosine similarity between embeddings correlates with the ground truth labels provided by human annotators. We measure how well the models work with semantics on this task. On top of that, we can use the resulting models to generate universal sentence embeddings for Slovak.

\subsubsection{Data}

Currently, there is no native Slovak STS dataset. We decided to translate existing English datasets, namely STSbenchmark~\cite{stsb} and SICK~\cite{sick}, into Slovak. These datasets use a $\langle 0,5 \rangle$ scale that expresses the similarity of two sentences. The meaning of individual steps on this scale is shown in Table~\ref{tab:sts-labels}. English STS systems also usually use natural language inference (NLI) data to perform additional pre-training. NLI is a task where the goal is to identify cases of entailment or contradiction between two sentences. We translated SNLI~\cite{snli} and MNLI~\cite{mnli} datasets to Slovak as well. We use M2M100 (1.2B parameters variant) machine translation system~\cite{m2m100}.

\begin{table}[]
    \centering
    \begin{tabular}{l|p{10cm}}
         \textbf{Label} & \textbf{Meaning} \\ \hline
         0 & The two sentences are completely dissimilar.  \\ \hline
         1 & The two sentences are not equivalent, but are on the same topic. \\ \hline
         2 & The two sentences are not equivalent, but share some details. \\ \hline
         3 & The two sentences are roughly equivalent, but some important information differs.\\ \hline
         4 & The two sentences are mostly equivalent, but some unimportant details differ. \\ \hline
         5 & The two sentences are completely equivalent, as they mean the same thing. \\
    \end{tabular}
    \caption{Annotation schema for STS datasets~\cite{sick}.}
    \label{tab:sts-labels}
\end{table}

\subsubsection{Previous Work}

No Slovak-specific sentence embedding model has been published yet. We use a naive solution based on Slovak word embeddings and several available multilingual models for comparison in this work:

\begin{itemize}
    \item \textbf{fastText}~\cite{fasttext} - We use pre-trained Slovak fastText word embeddings to generate representations for individual words. The sentence representation is an average of all its words. This represents a very naive baseline, since it completely omits the word order.
    \item \textbf{LASER}~\cite{laser} - LASER is a model trained to generate multilingual sentence embeddings. It is based on an encoder-decoder LSTM machine translation system that is trained with 93 languages. The encoder is shared across all the languages and as such, it is able to generate multilingual representations.
    \item \textbf{LaBSE}~\cite{labse} - LaBSE is an MBERT model fine-tuned with parallel corpus to produce mutlilingual sentence representations.
    \item \textbf{XLM-R$_{EN}$}~\cite{multi-sbert} - XLM-R model fine-tuned with English STS-related data (SNLI, MNLI and STSbenchmark datasets). This is a zero-shot cross-lingual learning setup, i.e. no Slovak data are used and only English fine-tuning is done.
\end{itemize}

\subsubsection{Our Fine-Tuning}

We use a setup similar to~\cite{multi-sbert}, but we actually fine-tune it with the translated Slovak data. A pre-trained LM is used to initialize a Siamese network. Both branches of the network are an LM with a mean-pooling layer at the top that generates the final sentence embeddings. The embeddings from the two sentences are compared using cosine similarity. The network is trained as a regression model, i.e. the final computed similarity is compared with the ground truth similarity with \textit{mean squared error} loss function. We use \texttt{SentenceTransformers} library for the fine-tuning.

We also performed a layer-wise analysis, where we analyzed which layers have the most viable representations for this task. We conducted the mean-pooling at different layers and ignored all the subsequent layers. This is similar to probing, but probing is usually done with frozen LM layers. In this case, we can not freeze the layers, since all the additional layers we added (mean-pooling, cosine similarity calculation) are not parametric.

\subsubsection{Results}

We compare the systems using Spearman correlation between the cosine similarity of the generated sentence representations and the ground truth data. The original STS datasets are using $\langle 0,5 \rangle$ scale. We normalize these scores to $\langle 0,1 \rangle$ range so that they can be directly compared to the cosine similarities. We performed a hyperparameter search in this case as well. Again, we have found out that the results are quite stable across various hyperparameter values, with learning rate being the most sensitive hyperparameter. The details of the hyperparameter tuning are shown in Table~\ref{tab:sts-hparams}. We show the main results in Table~\ref{tab:sts-results}.

\begin{table}[]
    \centering
    \begin{tabular}{l|r|r}
  \textbf{Hyperparameter} & \textbf{Range} & \textbf{Selected} \\ \hline
  Learning rate & $[10^{-7}, 10^{-3}]$ & $10^{-5}$ \\
  Batch size & $\{8, 16, 32, 64, 128\}$ & $32$ \\
  Warmup steps & $\{0, 500, 1000, 2000\}$ & $1000$ \\
  Weight decay & $[0, 0.2]$ & $0.15$ \\
  Learning rate scheduler & Various\footnotemark & \texttt{cosine with hard restarts} \\
    \end{tabular}
    \caption{Hyperparameters used for STS tagging. Adam was used as an optimization algorithm.}
    \label{tab:sts-hparams}
\end{table}
\footnotetext{See the list of schedulers supported by Sentence Transformers library.}

\begin{table}[]
    \centering
    \begin{tabular}{l|r}
         \textbf{Model} & \textbf{Score} \\ \hline
         fastText & 0.383 \\
         LASER & 0.711 \\
         LaBSE & 0.739 \\
         XLM-R$_{EN}$ & \textbf{0.801} \\ \hline
         WikiBERT & 0.673 \\
         MBERT & 0.748 \\
         XLM-R-Base & 0.767 \\
         XLM-R-Large & 0.791 \\ \hline
         SlovakBERT & 0.791 \\
    \end{tabular}
    \caption{Spearman correlation between cosine similarity of generated representations and desired similarities on STSbenchmark dataset translated to Slovak.}
    \label{tab:sts-results}
\end{table}

We can see that the results are fairly similar to POS tagging w.r.t. how the LMs are relatively ordered. The existing solutions are worse, except for XLM-R trained with English data, which is actually the best performing model in our experiments. It seems that their model fine-tuned with real data without machine-translation-induced noise works better, even if it has to perform the inference cross-lingually on Slovak data.

We also experimented with Slovak-translated NLI data in a way where the model was first fine-tuned on NLI task and then the final STS fine-tuning was performed. However, we were not able to outperform the purely STS fine-tuning with this approach and the results remained virtually the same. This result is in contrast with the usual case for English training, where the NLI data regularly improve the results~\cite{sbert}. We theorize that this effect might be caused by noisy machine translation.

Figure~\ref{fig:sts} shows the learning dynamics of STS. On the left, we can see that the performance takes much longer to plateau than in the case of POS. This shows that the model needs longer time to learn about semantics. Still, we can see that the performance ultimately stabilizes just below $0.8$ score. Similarly, unlike POS, we can see that the best performing layers are actually the last layers of the model. This suggests that a model with bigger capacity might have been trained successfully.

\begin{figure}[h]
\centering
\includegraphics[width=12cm]{figures/fig_sts.pdf}
\caption{Analysis of STS learning dynamics. \textit{Left:} Spearman correlation after fine-tuning with various checkpoints. \textit{Right:} Spearman correlation on all the layers with selected checkpoints.}\label{fig:sts}
\end{figure}

\subsection{Sentiment Analysis}\label{sec:sa}

The goal of sentiment analysis is to identify the affective sentiment of a given text. It requires semantic analysis of the text, as well as certain amount of emotional understanding.

\subsubsection{Data}

We use a Twitter-based dataset~\cite{twitter-sa} annotated on a scale with three values: \textit{negative}, \textit{neutral} and \textit{positive}. Some of the tweets have already been removed since the dataset was created. Therefore, we work with a subset of the original dataset.

We cleaned the data by removing URLs, retweet prefixes, hashtags, user mentions, quotes, asterisks, redundant whitespaces and trailing punctuation. We have also deduplicated the samples, as there were cases of identical samples (i.e. retweets) or very similar samples (i.e. automatically generated tweets) with the same, or even with different labels. After the deduplication, we were left with 41084 tweets with 11160 negative samples, 6668 neutral samples and 23256 positive samples.

Additionally, we have also constructed a series of test sets based on reviews from various domains: accommodation, books, cars, games, mobiles and movies. Each domain has approximately 100 manually labeled samples. These are published along with this paper. They serve to check how well the model behavior transfers to other domains. This dataset is called \textit{Reviews} in the results below, while the original Twitter-based dataset is called \textit{Twitter}.

\subsubsection{Previous Work and Baselines}

The original paper introducing the Twitter dataset introduced an array of traditional classifiers (Naive Bayes and 5 SVM variants) to solve the task. The authors report the macro-F1 score for positive and negative classes only. Additionally, unlike us, they worked with the whole dataset. Approximately 10K tweets have been deleted since the dataset was introduced. \cite{pecar} use the same version of the dataset as we do. They use approaches based on word embeddings and ELMO~\cite{elmo} to solve the task. Note that both published works use cross-validation, but no canonical dataset split is provided in either of them.

There are several existing approaches we use for comparison:

\begin{itemize}
    \item \textbf{NLP4SK}\footnote{\url{http://arl6.library.sk/nlp4sk/webapi/analyza-sentimentu}} - A rule-based sentiment analysis system for Slovak that is available online
    \item \textbf{Amazon} - We also translated the Slovak data into English and used Amazon's commercial sentiment analysis API and tested its performance on our test sets.
\end{itemize}

We have implemented several baseline classifiers that were trained with the same training data as the LMs:

\begin{itemize}
    \item \textbf{TF-IDF linear classifier} - A perceptron trained with SGD algorithm was used as a classifier. The text was represented as a TF-IDF vector computed over N-grams.
    \item \textbf{fastText classifier} - We used the built-in fastText classifier with and without pre-trained Slovak word embedding models.
    \item \textbf{Our STS embedding linear classifier} - A perceptron trained with SGD algorithm was used as a classifier. The text was represented using the sentence embedding model we have trained for STS.
\end{itemize}

We performed a random search hyperparameter optimization for all the approaches.

\subsubsection{Our Fine-Tuning}

We fine-tuned the LMs as classifiers with 3 classes. The topmost layer of an LM is discarded and instead a multilayer perceptron classifier with one hidden layer and dropout is applied on the representation of the first token. Categorical cross-entropy loss function is used as loss function. The class with the highest probability coming from the softmax function is selected as the predicted label during inference. We use \texttt{Hugging Face Transformers} library for fine-tuning.

\subsubsection{Results}

We report macro-F1 scores for all three classes as our main performance measure. The LMs were trained on the Twitter dataset. We calculate average F1 from our \textit{Reviews} dataset as an additional measure.

Again, we have performed a hyperparameter optimization of \textit{SlovakBERT}. The results are similar to results from POS tagging and STS. We have found out that learning rate is the most sensitive hyperparameter and that a small amount of weight decay is a beneficial regularization. The main results are shown in Table~\ref{tab:sa-results}. We can see that we were able to obtain better results than the results that were reported previously. However, the comparison is not perfect, as we use slightly different datasets for the aforementioned reasons.

\begin{table}[]
    \centering
    \begin{tabular}{l|r|r|r}
        \textbf{Model} & \multicolumn{2}{c|}{\textbf{Twitter F1}} & \textbf{Reviews F1} \\
        & \textbf{3-class} & \textbf{2-class} & \textbf{3-class}\\ \hline
        \cite{twitter-sa}* & - & 0.682 & -  \\
         \cite{pecar}* & 0.669 & - & - \\
         Amazon & 0.502 & 0.472 & 0.766 \\
         NLP4SK & 0.489 & 0.468 & \textbf{0.815} \\ \hline
         TF-IDF & 0.571 & 0.603 & 0.412 \\
         fastText & 0.591 &  0.622 &  0.416 \\
         fastText w/ emb. & 0.606 &  0.631 &  0.426 \\
         STS embeddings & 0.581 & 0.597 & 0.582 \\ \hline
         WikiBERT & 0.580 & 0.597 & 0.398 \\
         MBERT & 0.587 & 0.622 & 0.453 \\
         XLM-R-Base & 0.620 & 0.651 & 0.518 \\
         XLM-R-Large & 0.655 & \textbf{0.716} & 0.617 \\ \hline
         SlovakBERT & \textbf{0.672} & 0.705 & 0.583 \\
    \end{tabular}
    \caption{Macro-F1 scores for sentiment analysis task. The 2-class F1 score for Twitter is calculated only from positive and negative classes -- a methodology introduced in the original dataset paper. *Indicates different evaluation sets.}
    \label{tab:sa-results}
\end{table}

The LMs are ordered in performance similarly to how they are ordered in the two previous tasks. \textit{SlovakBERT} seems to be among the best performing models, along with the larger XLM-R-Large. The LMs were also able to successfully transfer their sentiment knowledge to new domains and they achieve up to 0.617 macro-F1 in the reviews as well. However, both Amazon commercial sentiment API and NLP4SK have even better scores, even though their performance on Twitter data was not very impressive. This is probably caused by the underlying training data they use in their systems, that might match our \textit{Reviews} datasets more than the tweets used for our fine-tuning.

\subsection{Document Classification}

The final task which we evaluate our LMs on is classification of documents into 6 news categories. The goal of this task is to ascertain how well LMs handle common classification problems. We use a Slovak Categorized News Corpus~\cite{scnc} that contains 4.7K news articles classified into 6 classes: Sports, Politics, Culture, Economy, Health and World. We do not use the \textit{Culture} category, since it contains significantly smaller number of samples.

Unfortunately, no existing work has used this dataset for document classification, so there are no existing results publicly available. We use the same set of baselines and LM fine-tuning as in the case of sentiment analysis, since both these tasks are text classification tasks, see Section~\ref{sec:sa} for more details.

\subsubsection{Results}

The main results from our experiment are shown in Table~\ref{tab:dc-results}. We can see that the LMs are again the best performing approach. In this case, the results are quite similar with \textit{SlovakBERT} being the best by a narrow margin. The baselines achieved significantly worse results. Note that our sentence embedding model has the worst results on this task, while it had competitive performance in sentiment classification. We theorize, that the sentence embedding model was trained on sentences and is therefore less capable of handling longer texts, typical for the dataset used here.

\begin{table}[]
    \centering
    \begin{tabular}{l|r}
        \textbf{Model} & \textbf{F1} \\
         TF-IDF & 0.953 \\
         fastText & 0.963 \\
         fastText w/ emb. & 0.963 \\
         STS embeddings & 0.935 \\ \hline
         WikiBERT & 0.935 \\
         MBERT & 0.985 \\
         XLM-R-Base & 0.987 \\
         XLM-R-Large & 0.985 \\ \hline
         Our model & \textbf{0.990} \\
    \end{tabular}
    \caption{Macro-F1 scores for document classification task.}
    \label{tab:dc-results}
\end{table}

\section{Conclusions}\label{sec:concl}

We have trained and published \textit{SlovakBERT} -- the first large-scale transformers-based Slovak masked language model using 19.35GB of Web-crawled Slovak text. We evaluated this model on multiple NLP tasks. We conclude, that \textit{SlovakBERT} achieves state-of-the-art results on these tasks. We also release the fine-tuned models for the Slovak community. Existing multilingual models can achieve comparable results on some tasks, however they are less efficient memory-wise and/or compute-wise.

\bibliographystyle{apalike}