\section{Introduction}

Stories are central to human culture and communication. However, it seems that stories are easier said than generated. Despite incredible recent progress in natural language processing generation of longer texts is still a challenge \cite{van2019narrative, rashkin2020plotmachines}. \citet{ostermann2019mcscript2} present a machine comprehension corpus for the end-to-end evaluation of script knowledge with 50\%  of the questions in the corpus that require script knowledge for the correct answer. The authors demonstrate that though the task is not challenging to humans, existing machine comprehension models fail to perform well on the data, even if they make use of a commonsense knowledge base.

Partially, this challenge could be attributed to the lack of adequate memory models. Longer texts demand better memory mechanisms and possible ways to construct such mechanisms are discussed in the literature for the last 25 years. Long short-term memory networks \cite{hochreiter1997long}, Neural Turing Machines \cite{graves2014neural}, memory networks \cite{weston2014memory} and many other architectures try to tackle this problem. Attempts to introduce some form of memory in transformers, such as \cite{guo2019star} or \cite{burtsev2020memory}, could be regarded as the next steps in this long line of work.

 There are some interesting recent attempts to generate long texts using some form that makes such longer text feasible for a human reader. For example, \citet{agafonova2020paranoid} generate a diary of a neural network. Yet the generation of a narrative is still challenging. For a detailed review of earlier approaches to narrative generation, we address the reader to \cite{kybartas2016survey}. Even modern models for narrative generation rely heavily on some form of expert knowledge or some type of hierarchical structure of the narrative. For example, \citet{fan2019strategies} first generate the predicate-argument structure of the text, then generate a surface realization of the predicate-argument structure, finally replace the entity placeholders with context-sensitive names and references. \citet{fan2019strategies,ammanabrolu2020story} propose a hierarchical generation framework that first plans a storyline, and then generate a story based on it.  present a technique for preprocessing textual story data into event sequences. \citet{xu2018skeleton} develop a model that first generates the most critical phrases, called skeleton, and then expands the skeleton to a complete and fluent sentence. Similarly, \citet{martin2018dungeons} provide a mid-level of abstraction between words and a sentence to minimize event sparsity and present a technique for automated story generation whereby the problem is decomposed into the generation of successive events and the generation of natural language sentences from events. Finally, \citet{brahman2020cue} develop an approach, where the user provides the model with such mid-level sentence abstractions in the form of cue phrases during the generation process.

However, we should take into consideration that modern Natural Language Processing (NLP) is fundamentally an experimental discipline, so the lack of dedicated data could be another bottleneck for the development of narrative generation. This paper tries to amend this problem.

Unfortunately, the majority of available narrative datasets deal with some constrained form of a short plot that is usually called {\em scenario}. These scenarios are centered around common activities, i.e. going grocery shopping or taking a shower. These narrative datasets available in the literature are also extremely small and could not be used with the most advanced modern NLP models. \citet{regneri2010learning} collect  493 event sequence descriptions for 22 behavior scenarios. \citet{modi2016inscript} present InScript dataset that consists of 1,000 stories centered around 10 different scenarios.  \citet{wanzare2019detecting} provide 200 scenarios and attempt to identify all references to them in a collection of narrative texts. \citet{mostafazadeh2016corpus} present a corpus of 50k five-sentence commonsense stories. Finally, there is an MPST dataset that contains 14K movie plot synopses, \cite{kar2018mpst}, and WikiPlots\footnote{https://github.com/markriedl/WikiPlots} that contains 112 936 story plots extracted from English language Wikipedia. Recently \citet{malyshevaDYP} provided a dataset of TV series along with an instrument for narrative arc analysis. These datasets are useful yet as well as a vast majority of the narrative datasets they are only available in English.

This paper provides a large multi-language dataset of stories in natural language.  The stories have a cross-language index and every story and character are cross-linked if they occur in different languages. Additionally, the texts have tags such as a genre or a topic. This is the first story dataset of such magnitude that we know of.  We hope that a large dataset of long storylines could be used for various aspects of narrative research as well as to facilitate experiments with end-to-end narrative generation.

\section{Data}

StoryDB is motivated by several interesting experiments that used WikiPlots — one of the larger English datasets of narratives available for all-purpose narrative research that we have mentioned earlier. Seeing various applications that Wikiplots dataset found in the NLP community, we believe, that StoryDB would be even more useful due to multiple languages, advanced filtering that guarantees higher quality of obtained data, and genre tagging. To improve reproducibility and make StoryDB usable as Wikipedia is further updated we publish the data as well as the code for the filtering pipeline\footnote{https://drive.google.com/drive/folders/ 1RCWk7pyvIpubtsf-f2pIsfqTkvtV80Yv}. The stories that form StoryDB are extracted from any Wikipedia article that contains a sub-header that contains the word "plot" (e.g., "Plot", "Plot Summary", etc.) in a corresponding language.

\subsection{Dataset structure}

The dataset consists of several index files and includes a directory \verb"plots". Every file in the directory has a similar structure. Two first letters of the filename stand for the ISO 639-1\footnote{https://en.wikipedia.org/wiki/ISO\_639-1} code of the language for the texts presented in the file. For example, \verb"hy_plots.tsv" contains 4 861 plots in Armenian language. The file \verb"simple_plots.tsv" contains stories in Simple English. Every entry in the plots file has a similar structure and includes the following fields:

\begin{itemize}
    \item ID — the unique number of a plot that is the same across every language in the dataset;
    \item Lang — the language of this particular entry;
    \item Link — a link to the Wikipedia page containing the plot;
    \item Title	— the title of the story;
    \item Text — the text of the story;
    \item Categories — the categories that Wikipedia assigns to this story.
\end{itemize}

One can navigate across plot files using StoryDB's Index file \verb"plot_matrix.tsv". The rows of the file stand for languages. If a given plot is available in a given language then the title of this plot stands in the corresponding cell of the \verb"plot_matrix.tsv". For example, if "Wee free men" is available in Simple English it could be found by its title in the corresponding  \verb"simple_plots.tsv". StoryDB also includes \verb"plot_rake.tsv" that contains keywords extracted with RAKE algorithm \cite{rose2010automatic} for every story.

Finally, the files \verb"ID_lang_tag.tsv" and \verb"ID_tag_average.tsv" include information about tags that correspond to the given story. We discuss tagging procedure in detail later.

\subsection{Preprocessing}

Our motivation is to provide a dataset of storylines for various languages including the low-resource ones. Roughly speaking, we want to be sure that every story that ends up in StoryDB is a legitimate storyline description in the corresponding natural language. Thus we are more interested in the precision of the dataset rather than in the recall. To guarantee a higher quality of the obtained stories we implemented several heuristical filters that we briefly describe here.

English Wikipedia is an order of magnitude bigger than any other Wikipedia both in terms of users and in terms of admins\footnote{https://meta.wikimedia.org/wiki/List\_of\_Wikipedias}. This makes the English list of storylines to be the most extensive one. We regard it as the least noisy one and use it as a reference source for the filtering procedure. We exclude every page that includes a plot yet has no plot section in English Wikipedia for the same entry.

If Wikipedia in language X has a page with title A and this page is also available in language Y under title B, we list such pair of stories as \verb"[language_X," \verb"title_A," \verb"language_Y," \verb"title_B"]". Every entry in this list is an edge in a graph of stories. Every vertex in this graph has a corresponding name \verb"language, title". Unlike connected stories from different languages that usually contain similar storylines, the stories listed under the same name in the same language might differ significantly. Say, two stories in language X \verb"[language_X," \verb"title_A]" and \verb"[language_X," \verb"title_B]" are both linked to one story in another language Y \verb"[language_Y," \verb"title_C]". To avoid such ambiguities we exclude fully connected components that contain more than one entry in the same language. Obtained list of stories ends up in the resulting matrix of stories to navigate the dataset. We experimented with various filtering procedures and found this combination to produce a sufficiently rich dataset with a minimum amount of duplicates.

StoryDB is also equipped with a catalog of characters. If a given character that has an individual Wikipedia page is mentioned in a story, its description in the original language is saved into the corresponding tsv-file alongside the ID of a story and the language of the description.

\subsection{Tagging}

We annotate the resulting stories using meta-information on categories from Wiki API\footnote{https://www.mediawiki.org/w/api.php?action=help\&
modules=query\%2Bcategories}. For every plot, we list all translated categories marked in every language in which this plot is available. Then we search these category lists for substrings that include tags from the manually created list of tags\footnote{The list of tags is published with the dataset.}. This allows us to proved language-specific tags for every language, that are listed in  \verb"ID_lang_tag.tsv". For example, Czech version of Black Night has tags \verb"action;" \verb"crime;" \verb"drama;" \verb"superhero;" \verb"comics;" and \verb"thriller", while the same story in Persian has no tag  \verb"comics", but has additional tags  \verb"neo-noir;"  \verb"psychological;" \verb"epic;" and \verb"screenplays;".

File \verb"ID_tag_average.tsv" includes the scores of the tags available for every story. The scores are calculated as follows: we count the number of times that a given tag is associated with a given story. Then we divide this number over the total number of languages in which the story is represented. The obtained space of tags could be useful for narrative exploration. Every story becomes a vector with every coordinate on the interval $[0,1]$. Figure \ref{fig:tags} shows a t-SNE visualisation of this space \cite{van2008visualizing} alongside the centroids of the more distinctive tags.

\begin{figure}[t!]\centering
     \includegraphics[scale=0.2]{tags.png}
  \caption{t-SNE visualisation for plots in StoryDB clustered according to their tags. Figure shows centroids of the tags with higher variance across the dataset.}
  \label{fig:tags}
\end{figure}

\subsection{StoryDB}

Figure \ref{fig:size} shows the relative size of the datasets in every language presented in StoryDB. English heavily dominates followed by Italian, French, Russian, and German.

\begin{figure*}[h!]\centering
     \includegraphics[scale=0.2]{plots_500.png}
  \caption{Number of stories in every language that has more that five hundred entries in StoryDB.}
  \label{fig:size}
\end{figure*}

There are more than 20 languages that have three thousand or more stories available, including such languages as Finnish, Hungarian, or Persian. Table \ref{tab:DB} summarises some of the resulting parameters of the obtained dataset.

\begin{table}[h!]
\centering
\begin{tabular}{lr}
\multicolumn{2}{c}{Story DB}                                                                \\ \hline
Number of languages                                                                & 42     \\
\begin{tabular}[c]{@{}l@{}}Median \# of stories in a language\end{tabular}   & 2 772  \\
\begin{tabular}[c]{@{}l@{}}Maximal \# of stories in a language\end{tabular} & 63 756 \\
\begin{tabular}[c]{@{}l@{}}Minimum \# of stories in a language\end{tabular} & 568
\end{tabular}
  \caption{Some resulting parameters of the StoryDB.}
  \label{tab:DB}
\end{table}

\section{Evaluation}

We have used three modern transformer-based architectures for the evaluation:
\begin{itemize}
    \item mBERT\footnote{https://huggingface.co/bert-base-multilingual-cased} \cite{devlin2018bert} — a multi-language version of BERT;
    \item mDistilBERT\footnote{https://huggingface.co/distilbert-base-multilingual-cased} \cite{Sanh2019DistilBERTAD} — a distilled version of multi-language BERT;
    \item XLM-Roberta\footnote{https://huggingface.co/xlm-roberta-base} \cite{conneau2020unsupervised} — a model that is two times larger than BERT in terms of the number of parameters.
\end{itemize}

 These models are the most widely used multi-language models to date. The results of the experiments are publicly available at  Weights and Biases\footnote{https://wandb.ai/altsoph/storydb
\_eval.task1\\ https://wandb.ai/altsoph/storydb\_eval.task3 \\ https://wandb.ai/altsoph/storydb\_eval.task3}, see \cite{wandb}. The evaluation was performed on ten largest languages in StoryDB, namely: English — 'en', French — 'fr', Italian — 'it', Russian — 'ru', German — 'de', Dutch — 'nl', Ukrainian — 'uk', Portuguese — 'pt', Polish — 'pl', and Spanish — 'es'.

 We evaluated three tasks:
 \begin{itemize}
    \item Task A. Multilabel classification for tags on a multilanguage corpus of plots;
    \item Task B. Multilabel classification for tags in cross-lingual learning;
    \item Task C. Multilabel classification for tags in cross-lingual learning with a corpus of overlapping plots that occur in every language.
\end{itemize}

 Let us now describe every task in detail.

 \subsection{Task A}

 We have sampled the ten most frequents tags out of StoryDB (tag 'film' was the most frequent yet was excluded as a somewhat redundant one). These tags were: 'drama', 'comedy', 'television', 'fiction', 'series', 'action',
 'thriller', 'black-and-white', 'science fiction', 'horror'. These ten tags form a vector, where every dimension corresponds to one particular tag. '1' encodes the presence of the tag and '0' stands for the absence of it.

For every language out of the top ten in StoryDB, we have sampled 2000 plots such that every plot has at least one tag out of the list of the ten most popular tags. In Task A the plots were sampled randomly for every language, so there is some overlap between languages. On average, 2\% of the plots in one language reoccur in another one. It is important to note that the set of tags for a given plot might differ across languages and one plot could have several tags simultaneously. Thus, multilabel classification is a natural evaluation task under these circumstances.

Since the dataset is not balanced with respect to tags we used the binary cross-entropy loss\footnote{https://pytorch.org/docs/stable/generated/ torch.nn.BCELoss.html} over the vector of tags. Table \ref{tab:task_a} and Table \ref{tab:task_a_detail} sum up the results of three models on a multilanguage dataset of plots. Further details across languages and tags are available online\footnote{https://wandb.ai/altsoph/storydb\_eval.task1}.

\begin{table}[]
\begin{tabular}{lll}
             & \begin{tabular}[c]{@{}l@{}}Hamming\\ Score\end{tabular} & \begin{tabular}[c]{@{}l@{}}Multilabel\\ Accuracy\end{tabular} \\
             \hline
mDistillBERT & 0.47                                                    & 0.31                                                          \\
mBERT        & 0.50                                                    & 0.33                                                          \\
XLM-RoBERTa  & 0.50                                                    & 0.33
\end{tabular}
\caption{Task A. Hamming score and multilabel accuracy for the vector of predicted tags on a validation set. Training data consists of sixteen thousand plots in ten languages, with one tenth of the dataset in every language.}
\label{tab:task_a}
\end{table}

\begin{table}[]
\begin{tabular}{llll}
                                                           & mD-BERT & mBERT & XLM-R \\
\hline
Comedy                                                     & 0.69         & 0.67 & 0.69        \\
Action                                                     & 0.67         & 0.70 & 0.67        \\
Fiction                                                    & 0.78         & 0.80 & 0.81        \\
Thriller                                                   & 0.67         & 0.63 & 0.64        \\
Horror                                                     & 0.70         & 0.76 & 0.75        \\
Drama                                                      & 0.73         & 0.74 & 0.74        \\
Series                                                     & 0.77         & 0.78 & 0.78        \\
Television                                                 & 0.74         & 0.76 & 0.76        \\
\begin{tabular}[c]{@{}l@{}}Science\\ Fiction\end{tabular}  & 0.78         & 0.80 & 0.81        \\
\begin{tabular}[c]{@{}l@{}}Black and \\ White\end{tabular} & 0.68         & 0.65 & 0.62
\end{tabular}
\caption{Task A. AUC-ROC for binary tag classifiers on a validation set. Training data consists of sixteen thousand plots in ten languages, with one tenth of the dataset in every language.}
\label{tab:task_a_detail}
\end{table}

 \subsection{Task B}

 Now let us do a similar setup yet train every model on one language in StoryDB and test its accuracy on another language. The parameters of the training datasets and labels are the same as in Task A above but every model is trained on one dataset and is then tested on other languages. Table \ref{tab:task_b} show the performance of mBERT, yet mDistillBERT and XLM-RoBERTa demonstrate similar behavior. The detailed results could be found online\footnote{https://wandb.ai/altsoph/storydb\_eval.task2}.

 \begin{table*}[]
 \centering
\begin{tabular}{l|llllllllll}
   & en   & de   & nl   & fr   & it   & es   & pt   & ru   & uk   & pl   \\
   \hline
en &    0.36 & 0.16 & 0.14 & 0.16 & 0.10 & 0.17 & 0.15 & 0.13 & 0.12 & 0.15 \\
de & 0.15 & 0.40 & 0.16 & 0.18 & 0.12 & 0.16 & 0.20  & 0.19 & 0.18 & 0.21 \\
nl & 0.20 & 0.33 & 0.41 & 0.20 & 0.22 & 0.32 & 0.29 & 0.31 & 0.25 & 0.30 \\
fr & 0.16 & 0.20 & 0.16 & 0.51 & 0.13 & 0.18 & 0.18 & 0.16 & 0.14 & 0.19 \\
it & 0.19 & 0.30 & 0.24 & 0.21 & 0.21 & 0.26 & 0.28 & 0.27 & 0.24 & 0.30 \\
es & 0.23 & 0.24 & 0.20 & 0.22 & 0.18 & 0.45 & 0.27 & 0.22 & 0.22 & 0.23 \\
pt & 0.15 & 0.21 & 0.17 & 0.22 & 0.10 & 0.19 & 0.44 & 0.19 & 0.14 & 0.23 \\
ru & 0.12 & 0.21 & 0.16 & 0.13 & 0.12 & 0.20 & 0.22 & 0.45 & 0.16 & 0.20 \\
uk & 0.10 & 0.20 & 0.16 & 0.14 & 0.09 & 0.19 & 0.19 & 0.23 & 0.25 & 0.20 \\
pl & 0.19 & 0.27 & 0.19 & 0.21 & 0.11 & 0.20 & 0.24 & 0.24 & 0.20 & 0.48
\end{tabular}
\caption{Task B. Multilabel accuracy for the vector of predicted tags by mBERT. Training data consists of one thousand six hundred plots in one language. Every row shows validation accuracy of a model pretrained on the corresponding language and validated on the plots in a language from the corresponding column.}
\label{tab:task_b}
\end{table*}

 Table \ref{tab:task_b} demonstrates that if we train the model on one language and validate it on the other the quality of the multilabel tag classification drops. This drop varies across languages and tends to be smaller for the languages that belong to the same language family.

 \subsection{Task C}

 The last validation is similar to Task B, yet now we sample plots that overlap in every language. This limits us to 1500 plots in six languages that we split into train and test. Now every plot occurs in every language. Table \ref{tab:task_c} shows the model manages to recover certain tags in one language after the pre-training on the other. Table \ref{tab:task_c} shows the performance of XLM-RoBERTa, yet mDistillBERT and mBERT demonstrate similar behavior. The performance of the models tends to be better on overlapping plots if we compare it to Task B. The detailed results could be found online\footnote{https://wandb.ai/altsoph/storydb\_eval.task3}.

  \begin{table}[]
\begin{tabular}{l|llllll}
   & en   & de   & nl   & fr   & it   & es    \\
   \hline
en & 0.29 & 0.16 & 0.12 & 0.12 & 0.10 & 0.08 \\
de & 0.27 & 0.31 & 0.18 & 0.19 & 0.15 & 0.11 \\
nl & 0.34 & 0.30 & 0.32 & 0.17 & 0.13 & 0.17 \\
fr & 0.22 & 0.20 & 0.16 & 0.27 & 0.15 & 0.10 \\
it & 0.34 & 0.29 & 0.23 & 0.25 & 0.25 & 0.16 \\
ru & 0.25 & 0.21 & 0.18 & 0.18 & 0.18 & 0.13
\end{tabular}
\caption{Task C. Multilabel accuracy for the vector of predicted tags by XLM-RoBERTa across the dataset of plots withour cross-language overlaps. Training data consists of one thousand two hundred plots in one language. Every row shows validation accuracy of a model pretrained on the corresponding language and validated on the plots in a language from the corresponding column.}
\label{tab:task_c}
\end{table}

The multilabel accuracy for tag prediction declines further yet it can neither be attributed to specific lexical properties of a particular language nor any form of overlap of plots across languages.

This series of evaluation tasks demonstrates two crucial properties of StoryDB:

\begin{itemize}
\item StoryDB could be used to work with narrative structures on the most abstract cross-lingual level;
\item StoryDB allows controlling for various cross-lingual similarities of plots during ablation experiments with models of narrative.
\end{itemize}

\section{Discussion}

We believe that a broad multilanguage dataset of narratives can facilitate several areas of narrative research.

\begin{itemize}
    \item Cross-cultural research of narrative structure. StoryDB provides possibilities to compare the structure of narrative in various languages. Since StoryDB includes every story in its original language and is equipped with a universal system of tags it is a natural source for such cross-cultural research.
    \item Classification of narratives. StoryDB includes an extensive amount of narratives for various languages alongside their genre tags. This allows to develop new methods for narrative classification as well as extensively test the ones that already exist, see for example \cite{reiter2014nlp}.
    \item Quantitative research of the narrative structure. \cite{y2007employing} represents a story as a cluster of emotional links and tensions between characters that progress over storytime. StoryDB includes the description of the plots alongside the key characters. Such information could be insightful for a deeper quantitative understanding of narrative as a by-product of character interaction.
    \item Summarization of narrative. Parallel corpora in different languages contain similar descriptions of the narrative that could vary in terms of details and length. That makes StoryDB a useful tool for potential narrative summarization research such as \cite{barros2019natsum}.
    \item End-to-end narrative generation. StoryDB is the first dataset of narratives that we know of that contains narrative descriptions in various natural languages.
\end{itemize}

\section{Conclusion}

This paper presents StoryDB — a broad multi-language dataset of narratives. We describe the construction of the dataset, provide the code for the whole pipeline, list the parameters of the resulting dataset, and briefly discuss several areas of natural language processing research, where StoryDB could be useful for the community.

We hope that StoryDB could be broadened as more plot descriptions are added to various languages. These considerations make StoryDB a flexible resource that would be relevant for the NLP community as the subfield of quantitative narrative research moves on.

\bibliographystyle{acl_natbib}


\section{Introduction}
\label{sec:intro}

As a classic
vision task, single-image-super-resolution (SISR) restores the original high-resolution (HR) image based on a down-sampled low-resolution (LR) one. It can be applied in various applications, such as low-resolution media data enhancement or video/image upscaling for high  resolution  display panels.
Various classic \cite{irani1991improving,freeman2002example,timofte2013anchored,timofte2014a+} and deep learning (DL)-based \cite{dong2014learning,dong2016accelerating,shi2016real,yu2018wide,liu2020fast} SR methods have been proposed in the past. Compared with classic interpolation algorithms to improve  image/video resolution, DL-based methods take  advantage of   learning  mappings from  LR to HR images from external datasets. Thus most   recent SR works emerge in the DL area. However, one   major limitation of existing DL-based SR methods is their high  computation and storage overhead to  achieve superior image quality, leading to difficulties to implement real-time SR inference even on powerful GPUs, not to mention resource limited edge devices. Due to the ever-increasing popularity of mobile devices and interactive on-mobile applications (such as live streaming),  it is essential to  derive lightweight SR models with both high image quality and low on-mobile inference latency.

There exist several works targeting at efficient SR models, including using upsampling operator at the end of a network \cite{dong2016accelerating,shi2016real}, adopting channel splitting \cite{hui2019lightweight}, using wider activation \cite{yu2018wide}, and  combining lightweight residual blocks with variants of group convolution \cite{liu2020fast}. Neural architecture search (NAS) is applied  to derive the optimal architecture in  many vision tasks. Latest works \cite{chu2019fast,song2020efficient,lee2020journey,chu2020multi} try to derive fast, lightweight, and accurate SR networks via NAS. However, their models
are still too large to be implemented on mobile devices. Furthermore, these methods usually take the parameter numbers and computation counts (such as multiply-accumulate (MAC) operations) into the optimization for model efficiency, without considering the actual on-mobile implementation performance such as the inference latency. The actual mobile deployment of SR mobiles has rarely been investigated. The most relevant works are the winner of the PIRM challenge \cite{vu2018fast}, MobiSR \cite{lee2019mobisr}, and work \cite{Zhan2021AchievingOR}. But they either require nearly one second per frame for inference, far beyond real-time, or take a long search time.

Targeting at achieving real-time inference of accurate SR model for 720p resolution on various resource-limited hardware such as mobile GPU and DSP, this paper proposes a compiler-aware NAS framework. An adaptive SR block is introduced to conduct the depth search and per-layer width search. Each convolution (CONV) layer is paired with a mask layer in the adaptive SR block for the width search, while the depth search is reached by choosing a path between the skip connection and the masked SR block. The mask can be trained along with the network parameters via gradient descent optimizers, significantly saving   training overhead. Instead of using MACs
as the optimization target, the latency performance is directly incorporated into the objective function with the usage of a speed model. Our implementation can support real-time SR inference with competitive SR performance on various resource-limited platforms, including mobile GPU and DSP.
The contributions are summarized below:
\begin{itemize}[leftmargin=*]
    \item We propose a framework to search for the appropriate depth and per-layer width with  adaptive SR blocks.
    \item We introduce a general compiler-aware speed model to predict the inference speed on the target device with corresponding compiler optimizations.
    \item The proposed framework can directly optimize the inference latency, providing the foundations for achieving real-time SR inference on mobile.
    \item Our proposed framework can achieve real-time SR inference (with only tens of milliseconds per frame) for the implementation of 720p resolution with competitive SR performance (in terms of PSNR and SSIM) on mobile (Samsung Galaxy S21). Our achievements can facilitate various practical SR applications with real-time requirements such as live streaming or video communication.
\end{itemize}
\section{Related Work}
\label{sec:related work}

\noindent \textbf{SR Works.}
In recent years, most SR works have shifted their approaches from classic methods to DL-based methods with significant SR performance improvements. From the pioneering SRCNN \cite{dong2014learning} to later works with shortcut operator, dense connection, and attention mechanism \cite{kim2016accurate,lim2017enhanced,zhang2018residual,zhang2018image,dai2019second}, the up-scaling characteristic have dramatically boosted at the cost of high storage and computation overhead. Most of the works mentioned above even take seconds to process only one image on a powerful GPU, let alone mobile devices or video applications.

\noindent \textbf{Efficient SR.}
Prior SR works are hard to be implemented on resource-limited platforms due to high computation and storage cost. To obtain more compact SR models, FSRCNN \cite{dong2016accelerating} postpones the position of the upsampling operator. IDN \cite{hui2018fast} and IMDN \cite{hui2019lightweight} utilize the channel splitting strategy.
CARN-M \cite{ahn2018fast} explores a lightweight SR model by combining efficient residual blocks with group convolutions.
SMSR \cite{wang2021exploring}  learns sparse masks to prune
redundant computation for efficient inference. ASSLN \cite{zhang2021aligned} and  SRPN \cite{zhang2021learning}  leverage structure-regularized pruning and
impose regularization on the pruned structure to guarantee the alignment of the locations
of pruned filters across different layers. SR-LUT \cite{jo2021practical} uses look-up tables to  retrieve the precomputed HR output values for  LR input pixels, with a more significant SR performance degradation.
However, these SR models do not consider the actual mobile deployment, and the sizes of the models are still too large. The actual SR deployment is rarely investigated.
The winner of the PIRM challenge \cite{vu2018fast}, MobiSR \cite{lee2019mobisr}, and work \cite{Zhan2021AchievingOR} explore the on-device SR, but the models take seconds for a single image, far from real time, or require long search time. Work \cite{ignatov2021real} considers real-time SR   deployed on the powerful mobile TPU, which is not  widely adopted such as mobile CPU/GPU.

\noindent \textbf{NAS for SR.}
NAS has been shown to outperform heuristic networks in various applications. Recent SR works start to leverage NAS to find efficient, lightweight, and accurate SR models. Works \cite{chu2019fast,chu2020multi,Zhan2021AchievingOR} leverage   reinforced evolution algorithms to achieve SR as a multi-objective problem. Work~\cite{ahn2021neural} uses a hierarchical search strategy to find the connection with local and global features. LatticeNet \cite{luo2020latticenet} learns the combination of residual blocks with the attention mechanism. Work~\cite{wu2021trilevel,huang2021lightweight,ding2021hrnas} search  lightweight architectures  at  different levels with  differentiable architecture search (DARTS) \cite{liu2018darts}.
DARTS based methods introduce architecture hyper-parameters which are usually continuous rather than binary, incurring additional bias during  selection and  optimization.
Furthermore, the above-mentioned methods typically take the number of parameters or MACs into the objective function, rather than  on-mobile latency as discussed in Sec.~\ref{sec: speed_incorporation}. Thus they can hardly satisfy the real-time requirement.

\noindent \textbf{Hardware Acceleration.}
A significant emphasis on optimizing the DNN execution has emerged in recent years \cite{lane2016deepx,xu2018deepcache,huynh2017deepmon,yao2017deepsense,han2016mcdnn,niu2020patdnn,dong2020rtmobile,jian2021radio,gong2022automatic}. There are several representative DNN acceleration frameworks including Tensorflow-Lite \cite{TensorFlow-Lite}, Alibaba MNN \cite{Ali-MNN}, Pytorch-Mobile \cite{Pytorch-Mobile}, and TVM \cite{chen2018tvm}. These frameworks include several graph optimization techniques such as layer fusion, and constant folding.

\section{Motivation and Challenges} \label{sec:challenges}

With the rapid development of mobile devices and real-time applications such as live streaming, it is essential and desirable to implement real-time SR on resources-limited mobile devices. However, it is challenging.
To maintain or upscale the spatial dimensions of feature maps based on large input/output  size,  SR  models  typically  consume  tens  of  or   hundreds of GMACs   (larger than several GMACs in image classification \cite{liu2019metapruning,wan2020fbnetv2}), incurring difficulties for real-time inference. For example, prior works on mobile SR deployment \cite{lee2019mobisr} and \cite{vu2018fast} achieve 2792ms and 912ms on-mobile inference latency, respectively, far from real-time.

We can adopt NAS or pruning methods to find a lightweight SR model with fast speed on mobile devices. But there are several challenges:
(\texttt{C1}) tremendous searching overhead with NAS, (\texttt{C2}) misleading  magnitude during pruning,  (\texttt{C3}) speed incorporation issues, and (\texttt{C4}) heuristic depth determination.

\noindent \textbf{Tremendous Searching Overhead with NAS.}
In NAS, the exponentially growing search space leads to tremendous search overhead. Specifically, the RL-based \cite{zoph2016neural,zhong2018practical,zoph2018learning} or evolution-based NAS methods \cite{real2019regularized,tan2019efficientnet,yang2020cars} typically need to sample large amounts of candidate models from the search space and train each candidate architecture with multiple epochs, incurring long search time and high computation cost.
Besides,  differentiable NAS methods \cite{brock2017smash,bender2018understanding,liu2018darts} build super-nets to train multiple architectures simultaneously, causing significant memory cost and limited discrete search space up-bounded by the available memory.
To mitigate these, there are certain compromised strategies, such as proxy tasks (to search on CIFAR and target on ImageNet) \cite{real2019regularized,zhou2020econas,yang2020cars} and performance estimation (to predict/estimate the architecture performance with some metrics) \cite{abdelfattah2021zerocost,tanaka2020pruning,ming_zennas_iccv2021}.

\noindent \textbf{Misleading Magnitude during Pruning.}
Pruning can also be adopted to reduce the model size, which  determines the per-layer pruning ratio and pruning positions.
With the assumption that weights with smaller magnitudes are less important for final accuracy, magnitude-based pruning \cite{han2016deep_compression,mao2017exploring,he2017channel,zhang2021unified,wen2016learning,gong2020privacy,li2020ss,ma2020blk,yuan2021mest} is widely employed to prune weights smaller than a threshold. However, the assumption is not necessarily true, and weight magnitudes can be misleading. Magnitude-based pruning is not able to achieve importance shifting during pruning. As detailed in Appendix~\ref{app: small_magenitude}, in iterative magnitude pruning, small weights pruned first are not able to become large enough  to contribute to the accuracy. Thus layers pruned more at initial will be pruned more and more, causing a non-recoverable pruning policy.
It becomes pure exploitation without exploration.

\label{sec: speed_incorporation}
\noindent\textbf{Speed Incorporation Issues.}
\label{sec: speed_incorporation}
To achieve real-time inference on mobile, it is essential to obtain the on-mobile speed performance when searching  architectures. However, it is non-trivial to achieve this since
testing speed requires an additional process to interact with the mobile device for a few minutes, which can hardly be incorporated into a typical model training.
To mitigate this, certain methods \cite{liu2019metapruning,wan2020fbnetv2,9522982} adopt weight number or computation counts  as an estimation of the speed performance.   Other methods \cite{wu2019fbnet,dai2019chamnet,yang2018netadapt}  first collect on-mobile speed data and then build lookup tables with the speed data to estimate the speed.

\noindent\textbf{Heuristic Depth Determination.}
Reducing model depth can avoid all computations in the removed layers, thus significantly accelerating the inference. Since previous NAS works do not incorporate a practical speed constraint or measurement during optimization,  their search on model depth is usually heuristic. Designers determine the model depth according to a simple rule that the model should satisfy an inference budget, without a specific optimization method \cite{ming_zennas_iccv2021,liu2018progressive,zhou2020econas,yang2020cars,bender2018understanding,liu2018darts}.
More efforts are devoted to searching other optimization dimensions such as kernel size or width rather than model depth.

\section{Our Method}
\label{sec:method}

We first introduce the framework, then discuss the components of the framework in detail. We also specify how it can deal with the challenges   in Sec. \ref{sec:challenges}.
\subsection{Framework with Adaptive SR Block}

In the framework, we perform a compiler-aware architecture depth and per-layer width search to achieve real-time SR on mobile devices. The search space contains the width for each CONV layer and the number of stacked SR blocks in the model, which is too large to be explored with a heuristic method. Therefore, we propose an adaptive SR block to implement the depth and per-layer width search, and the model is composed of multiple adaptive SR blocks. Fig.~\ref{fig:search} shows the architecture of the adaptive SR block. It consists of a masked SR block, a speed model, and an aggregation layer. The adaptive SR block has two inputs (and outputs) corresponding to the SR features and the accumulated speed, respectively. It achieves per-layer width search with mask layers in the masked SR blocks and depth search with aggregation layer to choose a path between the skip connection and the masked SR block. Besides, to obtain the on-mobile speed performance, we adopt a speed model to predict the speed of the masked SR block. The speed model is trained on our own dataset with speed performance of various block width configurations measured through compiler optimizations for significant inference acceleration to achieve accurate speed prediction.

\begin{figure}[t]
     \centering
     \includegraphics[width=0.78\columnwidth]{figs/search_fig.pdf}
     \caption{Architecture of the adaptive SR block search.}
    \label{fig:search}
\end{figure}

\subsection{Per-Layer Width Search with Mask Layer for \texttt{C1} and \texttt{C2}}
Width search is performed for each CONV layer in a typical WDSR block \cite{yu2018wide}. WDSR is chosen as our basic building blocks since it has demonstrated high efficiency in SR tasks \cite{Yu_2019_ICCV,yu2019autoslim,Cheng_2019_CVPR_Workshops}. Note that our framework is not limited to the WDSR block and can be easily extended to various residual SR blocks \cite{ahn2018fast,lim2017enhanced,hui2018fast} in the literature.
To satisfy the real-time requirement, we perform a per-layer width search to automatically select an appropriate number of channels for each CONV layer in the  WDSR block.
Specifically, we insert  a differentiable mask layer (a depth-wise $1\times 1$ CONV  layer) after each CONV layer to serve as the layer-wise trainable mask, as shown below,
{\small
\begin{equation}
  \bm  a_l^n =  \bm m_l^n \odot (\bm w_l^n \odot \bm a_{l-1}^n),
\end{equation}}%
where $\odot$ denotes the convolution operation. $\bm w_l^n \in R^{o\times i\times k\times k}$ is the weight parameters in the $l^{th}$ CONV layer of the $n^{th}$ block, with $o$ output channels, $i$ input channels, and  kernels of size $k\times k $.
$\bm a_l^n \in R^{B\times o\times s\times s'}$ represents the output features of $l^{th}$ layer (with the trainable mask), with $o$ channels and  $s\times s'$ feature size. $B$ denotes the batch size.
$\bm m_l^n \in R^{o\times 1\times 1\times 1}$ is the corresponding weights of the depth-wise CONV layer (i.e., the mask layer).

We use each element of $\bm m_l^n$  as the pruning indicator for the corresponding output channel of  $\bm w_l^n \odot \bm a_{l-1}^n$.
Larger elements of $\bm m_l^n$  mean that the corresponding channels should be preserved while smaller elements indicate pruning the  channels.
Formally, we use a threshold to convert    $\bm m_l^n$ into a binary mask,
{\small
\begin{equation}
 \bm b_l^n =
    \begin{cases}
    1, \bm m_l^n > thres. \\
    0, \bm m_l^n \leq thres.
    \end{cases} \text{(element-wise)},
\end{equation}}%
where $\bm b_l^n \in \{0, 1\}^{o\times 1\times 1\times 1}$ is the binarized $\bm m_l^n$.
We initialize $\bm m_l^n$  with random values between 0 and 1, and the adjustable  $thres$ is  set to 0.5 in our case.  The WDSR block with the proposed mask layers   is named as masked SR block.

Thus we are able to obtain a binary mask for each CONV layer. The next  problem  is how to make the mask trainable, as the binarization operation is  non-differentiable, leading to difficulties for back-propagation. To solve this,  we integrate Straight Through Estimator (STE) \cite{bengio2013setimating} as shown below,
{\small
\begin{equation}
\frac{\partial \mathcal L}{\partial \bm m_l^n} = \frac{\partial \mathcal L}{\partial \bm b_l^n},
\end{equation}}%
where we directly pass the gradients through the binarization.  The STE method is originally proposed to avoid the non-differentiable problems in quantization tasks \cite{liu2019learning,yin2019understanding}.
Without STE, some methods adopt complicated strategies to deal with the non-differentiable binary masks  such as \cite{guo2020dmcp,guan2020dais}.

With the binarization and the STE method, we are able to build a trainable mask to indicate whether the corresponding channel is pruned or not.
Our mask generation and training are more straightforward and simpler. For example,  proxyless-NAS \cite{cai2018proxylessnas} transforms the real-valued weights  to  binary gates  with  a  probability distribution, and adopts complex mask updating procedure (such as task factorizing). SMSR \cite{wang2021exploring} adopts Gumbel softmax to perform complex sparse mask CONV.  Unlike  proxylessNAS or SMSR,  we  generate binary  masks  simply  via  a  threshold  and  train  the  masks directly  via  STE.

 \subsection{Speed Prediction with Speed Model for \texttt{C3}}
 \label{sec: speed_model_framework}

To achieve real-time SR inference on mobile devices, we take the inference speed into the optimization to satisfy a given real-time threshold.
It is hard to measure the practical speed or latency of various structures on mobile devices during optimization. Traditionally, the inference speed may be estimated roughly with the number of computations \cite{liu2019metapruning,wan2020fbnetv2,9522982} or a latency lookup table \cite{wu2019fbnet,dai2019chamnet,yang2018netadapt}, which can hardly provide an accurate speed.
To solve this problem, we adopt a DNN-based speed model to predict the inference speed of the block.
The input of the speed model is the width of each CONV layer in the block, and it outputs the block speed. As shown in Fig. \ref{fig:search},  the width of each CONV layer can be obtained through the mask layer. Thus the speed model can work perfectly with the width search,  dealing with \texttt{C3} to provide speed performance of various architectures.

To train such a speed model, we first need to build a speed dataset with block latency  of various layer width configurations in the block. Next, we can train a speed model based on the dataset to predict the speed. We find that the trained speed model is accurate in predicting the speed of different layer widths in the block (with 5\% error at most).
We show the details about the dataset, speed model, and the prediction accuracy in Sec.~\ref{sec: speed_model_compiler_optimization} and Appendix B.

We highlight that our speed model  not only takes the masks as inputs to predict the  speed, but also back-propagates the gradients from the speed loss  (Eq.~\eqref{eq:loss}) to update the masks as detailed in Sec.~\ref{sec: training_loss}, rather than just predicting performance forwardly such as
\cite{wen2020neural}. That is why we  build the speed model based on DNNs instead of loop-up tables.  The trainable masks and the  speed model are combined comprehensively to solve the problem more efficiently.

\subsection{Depth Search with Aggregation Layer for \texttt{C4}}
Although reducing the per-layer width can  accelerate the inference,
removing the whole block can avoid the computations of the whole block, thus providing higher speedup. Hence, besides width search, we further incorporate depth search to automatically determine the number of adaptive SR blocks in the model. Note that although per-layer width search may also converge to zero width, which eliminates the entire block, we find that in most cases, there are usually a few channels left in each block to promote the SR performance, leading to difficulties in removing the whole block. Thus it is necessary to incorporate depth search.

To perform depth search, we have two paths in each adaptive SR block. As shown in Fig.~\ref{fig:search}, one path is the skip connection, and the other path is the masked SR block. In the aggregation layer, there is a parameter like a switch to control which path the SR input goes through. If the SR input chooses the skip path, the masked SR block is skipped, and the latency of this block is just 0, leading to significant inference acceleration.
The aggregation layer plays a key role in the path selection. It contains two trainable parameters $\alpha_s$ and $\alpha_b$. In the forward pass, it selects the skip path or the masked WDSR block path based on the relative relationship of $\alpha_s$ and $\alpha_b$, as shown below,
{\small
\begin{align}
  \bm  \beta_s = 0 \   \text{and}  \ \beta_b =1,  \text{if} \ \alpha_s \le \alpha_b,  \\
    \bm  \beta_s = 1 \  \text{and} \ \beta_b =0,   \text{if} \  \alpha_s > \alpha_b,
\end{align}}%
where the binarized  variables $\beta_s$ and $\beta_b$   denote the path selection ($\beta_s {=}1$ means choosing the skip path and $\beta_b{=}1$ means choosing the masked SR block path).
Since the comparison operation is non-differentiable, leading to difficulties for back-propagation, similarly we adopt STE  \cite{bengio2013setimating} to make it differentiable as below,
{\small
\begin{align}
\frac{\partial \mathcal L}{\partial  \alpha_s} = \frac{\partial \mathcal L}{\partial \beta_s},  \  \ \frac{\partial \mathcal L}{\partial  \alpha_b} = \frac{\partial \mathcal L}{\partial \beta_b}.
\end{align}}

In the aggregation layer,
the forward computation can be represented below,
{\small
\begin{align}
\bm {a}^n & = \beta_s \cdot \bm a^{n-1}  + \beta_b \cdot \bm a_L^n, \\
v_n & = v_{n-1} + \beta_b \cdot  v_c,
\end{align}}
where $\bm {a}^n$ is the SR output features of the $n^{th}$ adaptive SR block.   $\bm {a}^n_L$ is the SR output features of masked SR block in the $n^{th}$ adaptive SR block, and $L$ is the maximum number of CONV layers in each block and we have $l{\le} L$.
$v_n$ is the accumulated speed or latency until the $n^{th}$ adaptive SR block and  $v_c$ is the speed of the current block which is predicted by the speed model.
By training  $\alpha_s$ and $\alpha_b$, the model can learn to  switch between the skip path and the  SR  path to determine the model depth, thus dealing with \texttt{C4}.

\subsection{Training Loss} \label{sec: training_loss}

Multiple adaptive SR blocks can form the SR model, which provides two outputs including the typical SR outputs and the speed outputs. The training loss is a combination of a typical SR loss and a speed loss as below,
{\small
\begin{align}
\mathcal L_{SPD} &= \text{max}\{0, v_N- v_T \}, \\
\mathcal L & =  \mathcal L_{SR} + \gamma \mathcal L_{SPD}, \label{eq:loss}
\end{align}}%
where $v_T$ is the real-time threshold, $v_N$ is the accumulated speed of $N$ blocks, and
$\gamma$ is a parameter to control their relative importance. The objective is to achieve high SR performance while the speed can satisfy a real-time threshold.
To summarize, with the trainable masks,
the speed model, and the aggregation layer in the adaptive SR block, our search algorithm achieves the following advantages:
\begin{itemize}[leftmargin=*]
    \item The mask can be trained along with the  network parameters via  gradient descent optimizers, thus dealing with \texttt{C1} to save search overhead compared with previous one-shot pruning \cite{he2017channel,frankle2018lottery} or NAS methods \cite{zoph2016neural,zhong2018practical} to train multiple epochs for each candidate architecture with huge searching efforts.
    \item Compared with magnitude-based threshold pruning, we decouple the trainable masks from original model parameters, thus enabling exploitation and overcoming the drawbacks of magnitude-based  pruning,  dealing with \texttt{C2}.
    \item We use the speed model for predicting the speed to solve \texttt{C3}, which is differentiable regarding the trainable mask. Thus the mask is trained to find a model with both high SR performance and fast inference speed.
    \item We also incorporate depth search though  aggregation layers  to deal with \texttt{C4}.
\end{itemize}

\section{Compiler Awareness with Speed Model }
\label{sec: speed_model_compiler_optimization}

To satisfy the speed requirement with a given  latency threshold on a specific mobile device,  it is required to obtain the actual inference latency   on the  device.
It is non-trivial to achieve this as the model speed varies with different model width and depth.
It is unrealistic to measure the actual on-mobile speed   during the   search, as the search space is quite large,
and testing the mobile speed of each candidate can take a few minutes, which is not compatible with  DNN training.

To solve this problem,
we adopt a  speed model to predict the inference latency of the masked SR block with various width configurations. With the speed model, we can obtain the speed prediction as outputs by providing the width of each CONV layer in the SR block as inputs. It is fully compatible with the trainable mask, enabling differentiable  model speed with respect to the layer width.

To obtain the speed model, we first build a latency dataset with latency data measured on the hardware platforms incorporated with compiler optimizations. Then the DNN speed model is trained based on the latency dataset.

\begin{figure}[t]
     \centering
     \includegraphics[width=0.8\columnwidth]{figs/Com_diagram.pdf}
     \caption{The overview of compiler optimizations.}
    \label{fig:compiler}
\end{figure}

\noindent\textbf{Compiler Optimization.}
To build a latency dataset, we need to measure the speed of various block configurations on mobile devices. Compiler optimizations are adopted to accelerate the inference speed during speed testing. It is essential to incorporate compiler optimizations as they can significantly accelerate the inference speed.
The overview of the compiler optimizations is shown in Fig. \ref{fig:compiler}.

To fully exploit the parallelism for a higher speedup,  the key features of SR have to be considered. As the objective of SR is to obtain a HR image from its LR counterpart, each layer has to  maintain or upscale the spatial dimensions of the feature, leading to larger feature map size and more channels compared with classification tasks. Therefore, the data movements between the memory and cache are extremely intensive. To reduce the data movements for faster inference, we adopt two important optimization techniques: 1) operator fusion and 2) decreasing the amount of data to be copied between CPU and GPU.

Operator fusion is a key optimization technique adopted in many state-of-the-art DNN execution framework \cite{TensorFlow-Lite,Ali-MNN,Pytorch-Mobile}. However, these frameworks usually adopt fusion approaches based on certain patterns that are too restrictive to cover the diversity of operators and layer connections. To address this problem, we classify the existing operations in the SR model into several groups based on the mapping between the input and output, and develop rules for different combinations of the
groups in a more aggressive fusion manner. For instance, CONV operation and depth-to-space operation can be fused together.  With layer fusion, both the memory consumption of the intermediate results and the number of operators can be reduced.  An auto-tuning process is followed to determine the best-suited configurations of parameters for different mobile CPUs/GPUs and Domain Specific Language (DSL) based code generation. After that, a high-level DSL is leveraged to specify the operator in the computational graph of a DNN model. We show more details about compiler optimization in Appendix C.

\noindent\textbf{Latency Dataset.}
To train the speed model, we first  measure and collect the inference speed of the WDSR block under various CONV layer width configurations. After that, a dataset of the WDSR block on-mobile speed with different configurations can be built.  We vary the number of filters in each CONV layer as the different width configurations. The inference time is measured on the target device (Samsung Galaxy S21) by stacking 20  WDSR blocks with the same configuration, and the average latency is
used as the inference time  to mitigate the overhead of loading data on mobile GPU. As the maximum number of CONV layers in each masked WDSR block is $L$, each data point in the dataset can be represented as a tuple with $L{+}2$ elements: $\{\mathcal{F}_{CONV^1}, \cdots, \mathcal{F}_{CONV^{L+1}}, \mathcal{T}_{inference}\}$, where $\mathcal{F}_{CONV^i},$ for $i{\in}\{1,\cdots, L\}$, indicates the number of input channels for the $i^{th}$ CONV layer, $\mathcal{F}_{CONV^{L+1}}$ is the number of output channels for the last CONV layer, and $\mathcal{T}_{inference}$ is the inference speed for this configuration measured in milliseconds. The entire dataset is composed of 2048 data points.

\noindent\textbf{Speed Model.}
With the latency dataset, the speed model can be trained on the collected data points. The inference speed estimation  is a regression problem, thus, a  network with 6 fully-connected layers combined with ReLU activation is used as the speed model.  During the speed model training, 90\% of the data is used for training  and the rest is for  validation. After training, the speed model can predict the inference time of various   block configurations with high accuracy. From our results, the speed model only incurs 5\% of deviation for the speed prediction.
The speed model has two advantages: (1) It is compatible with the width search framework as the trainable mask can be directly fed into the speed model. (2) It makes  the   model speed    differentiable with respect to the  masks, and back-propagates   gradients to update the masks, thus the model can  update the model speed by adjusting the layer width though back-propagation.
\section{Experiments}
\label{sec:exp}

\begin{table*}[t]
\scriptsize\sffamily
    \centering
    \renewcommand{\arraystretch}{1.2}
\begin{adjustbox}{max width=0.9\textwidth}
\begin{tabular}{c|c|c|c|c|cccc|cccc}
\toprule
\multirow{2}{*}{Scale} & \multicolumn{1}{c|}{\multirow{2}{*}{Method}} & \multicolumn{1}{c|}{\multirow{2}{*}{ \makecell[c]{ Params \\ (K)} }} & \multicolumn{1}{c|}{\multirow{2}{*}{\makecell[c]{  MACs \\(G)}} } & \multicolumn{1}{c|}{\multirow{2}{*}{ \makecell[c]{Latency \\ (ms)}}} & \multicolumn{4}{c|}{PSNR}                                                                                & \multicolumn{4}{c}{SSIM}                                                                                \\ \cline{6-13}
                       & \multicolumn{1}{c|}{}                        & \multicolumn{1}{c|}{}                                  & \multicolumn{1}{c|}{}                      & \multicolumn{1}{c|}{}                              & \multicolumn{1}{c}{Set5} & \multicolumn{1}{c}{Set14} & \multicolumn{1}{c}{B100} & \multicolumn{1}{c|}{Urban100} & \multicolumn{1}{c}{Set5} & \multicolumn{1}{c}{Set14} & \multicolumn{1}{c}{B100} & \multicolumn{1}{c}{Urban100} \\ \hline \hline
\multirow{14}{*}{\footnotesize $\times$2}

    & \textsc{FSRCNN}~\cite{dong2016accelerating} & 12 & 6.0 & 128.47 & 37.00  & 32.63 & 31.53  & 29.88   & 0.9558 &   0.9088 &  0.8920 & 0.9020 \\
    & \textsc{MOREMNAS-C}~\cite{chu2020multi} & 25 & 5.5 & --- & 37.06  & 32.75 &   31.50 & 29.92 & 0.9561 & 0.9094 & 0.8904 &  0.9023 \\
    & \textsc{TPSR-NOGAN}~\cite{lee2020journey} & 60 &14.0 & --- &37.38  & 33.00 & 31.75 & 30.61  &  0.9583 &  0.9123 &  0.8942 & 0.9119\\
    & \textsc{LapSRN}~\cite{lai2017deep} & 813& 29.9& --- &  37.52 &  33.08 &31.80   &  30.41 & 0.9590 &0.9130 &0.8950 &0.9100 \\
    & \textsc{CARN-M}~\cite{ahn2018fast} & 412 &91.2& 1049.92 & 37.53 &  33.26 & 31.92 &  31.23 & 0.9583 &0.9141 &0.8960 &0.9193\\
    & \textsc{FALSR-C}~\cite{chu2019fast} & 408 &93.7 & --- & 37.66  & 33.26 & 31.96  &  31.24 & 0.9586 &0.9140 &0.8965 &0.9187 \\
    & {\textsc{ESRN-V}}~\cite{song2020efficient} & 324 & 73.4 & --- & 37.85 & 33.42 &32.10  &  31.79   & 0.9600 & 0.9161 & 0.8987 & 0.9248 \\
    & \textsc{EDSR}~\cite{lim2017enhanced} & 1518 & 458.0 & 2031.65 & 37.99  & 33.57 &  32.16 &  31.98 & 0.9604 & 0.9175 & 0.8994 & 0.9272 \\
    & \textsc{WDSR}~\cite{yu2018wide} & 1203 & 274.1 & 1973.31 & 38.10 &  33.72 &32.25  & 32.37  &  0.9608 & 0.9182 & 0.9004 & 0.9302 \\
    & { \textsc{SMSR}~\cite{wang2021exploring}}  & 985 & 131.6 & --- & 38.00 & 33.64 & 32.17 & 32.19  &   0.9601 & 0.9179 &  0.8990 & 0.9284 \\
    & { \textsc{SRPN-L}~\cite{zhang2021learning}}  & 609 & 139.9& --- & 38.10 & 33.70 & 32.25 & 32.26  &   0.9608 & 0.9189 &  0.9005 & 0.9294 \\
    & \textbf{Ours ($v_T{=}100$}ms\textbf{)}  & 47 & 11.0 & \textbf{98.90} & 37.64  & 33.16 &  31.91 &  31.08 &  0.9591 & 0.9136 & 0.8961 & 0.9170 \\
    & \textbf{Ours ($v_T{=}70$}ms\textbf{)} & 28 & 6.6 & \textbf{66.09} & 37.49  & 33.05 & 31.81 & 30.76  &  0.9584 & 0.9123 & 0.8946 & 0.9135 \\
    & \textbf{Ours ($v_T{=}40$}ms, \textbf{real-time)}  & 11 & 2.5 & \textbf{34.92} & 37.19  &  32.80&  31.60 & 30.15  &0.9572 & 0.9099 &  0.8919 & 0.9054 \\
\hline
\multirow{16}{*}{\footnotesize $\times$4}
    & \textsc{FSRCNN}~\cite{dong2016accelerating} & 12 & 4.6 & 98.13 & 30.71  &  27.59 & 26.98 & 24.62 & 0.8657& 0.7535 &0.7150 &0.7280 \\
    & {\textsc{TPSR-NOGAN}}~\cite{lee2020journey} & 61 &3.6 & 55.82 &31.10  &  27.95 &  27.15 & 24.97 & 0.8779 &0.7663 &0.7214 &0.7456 \\
    & {\textsc{FEQE-P}}~\cite{vu2018fast} & 96 & 5.6 & 82.81 & 31.53 & 28.21 &27.32  &25.32  & 0.8824 & 0.7714 & 0.7273 & 0.7583 \\
    & \textsc{CARN-M}~\cite{ahn2018fast} & 412 &32.5 & 374.15 & 31.92  & 28.42  &27.44  & 25.62 & 0.8903 &0.7762 &0.7304 &0.7694\\
    & {\textsc{ESRN-V}}~\cite{song2020efficient} & 324 & 20.7 & --- & 31.99  & 28.49  & 27.50 & 25.87 & 0.8919 &0.7779 &0.7331 &0.7782 \\
    & {\textsc{IDN}}~\cite{hui2018fast} & 600 & 32.3 & --- & 31.99  & 28.52  & 27.52 & 25.92 &  0.8928 &0.7794 &0.7339 &0.7801 \\
    & \textsc{EDSR}~\cite{lim2017enhanced} & 1518 & 114.5 & 495.90 & 32.09 & 28.58 & 27.57 & 26.04 & 0.8938 & 0.7813 & 0.7357 & 0.7849 \\
    &\textsc{DHP-20}~\cite{li2020dhp}& 790 & 34.1 & --- & 31.94  &28.42  & 27.47 &  25.69 & \quad--- & \quad--- & \quad--- &\quad--- \\
    &\textsc{IMDN}~\cite{hui2019lightweight}& 715 & 40.9 & --- & 32.21  &  28.58 &   27.56 & 26.04 & 0.8948 &0.7811 &0.7353 & 0.7838\\
    & {\textsc{WDSR}}~\cite{yu2018wide} & 1203 & 69.3 & 533.02 & 32.27 & 28.67  &  27.64 &26.26  & 0.8963 &0.7838 &0.7383 &0.7911 \\
    & {\textsc{SR-LUT-S}~\cite{jo2021practical} } &  77 & --- & --- & 29.77 &  26.99 & 26.57  & 23.94  &   0.8429 & 0.7372 &  0.6990 & 0.6971 \\
    & {\textsc{SMSR}~\cite{wang2021exploring} } &  1006 & 41.6 & --- & 32.12 &  28.55 & 27.55  & 26.11  &   0.8932 & 0.7808 &  0.7351 & 0.7868 \\
    & { \textsc{SRPN-L}~\cite{zhang2021learning}}  & 623 & 35.8& --- & 32.24 & 28.69 & 27.63 & 26.16  &   0.8958 & 0.7836 &  0.7373 & 0.7875 \\

    & \textbf{Ours ($v_T{=}100$}ms\textbf{)} & 188 & 10.8 & \textbf{93.50} &  32.02 &  28.50 & 27.51 & 25.83 & 0.8922 & 0.7778 & 0.7328 & 0.7769 \\

    & \textbf{Ours ($v_T{=}70$}ms\textbf{)} & 116 & 6.7 & \textbf{64.95} & 31.88  &  28.43 & 27.46 &25.69  & 0.8905 & 0.7760 & 0.7312 & 0.7715 \\

    & \textbf{Ours ($v_T{=}40$}ms,\textbf{real-time)} & 66 & 3.7 & \textbf{36.46} & 31.73  & 28.28  & 27.34  &25.44  & 0.8878 & 0.7725 &0.7281 & 0.7620 \\
     \bottomrule
    \multicolumn{13}{l}{$*$ Some latency results are not reported as the models are not open-source or contain operators that cannot run on mobile GPU. } \\
    \multicolumn{13}{l}{$\dag$ The latency results are measured on the GPU of Samsung Galaxy S21.  }

\end{tabular}

\end{adjustbox}
    \caption{Comparison  with SOTA efficient SR models for implementing 720p.}
        \label{table:result_sr}
\end{table*}

\subsection{Experimental Settings}
\noindent\textbf{SR Datasets.}
All SR models are trained on the training set of  DIV2K \cite{Agustsson_2017_CVPR_Workshops}  with 800 training images. For  evaluation, four benchmark datasets Set5 \cite{bevilacqua2012low}, Set14 \cite{yang2010image}, B100 \cite{martin2001database}, and Urban100 \cite{huang2015single} are used for test. The PSNR and SSIM are calculated on the luminance channel (a.k.a. Y channel) in the YCbCr color space.

\noindent\textbf{Evaluation Platforms and Running Configurations.} The training codes are implemented with  PyTorch. 8 GPUs are used to conduct the search, which usually finishes in 10 hours. The latency is measured on the GPU of an off-the-shelf Samsung Galaxy S21 smartphone, which has the Qualcomm Snapdragon 888 mobile platform with a Qualcomm Kryo 680 Octa-core CPU and a Qualcomm Adreno 660 GPU. Each test takes 50 runs on different inputs with 8 threads on CPU, and all pipelines on GPU. The average time is reported.

\noindent\textbf{Training Details.}
$48\times48$ RGB image patches are randomly sampled from LR images for each input minibatch.
We use the architecture of WDSR with 16 blocks as the backbone  of our NAS process.
Considering the huge input size of SR (normally nHD--$640{\times}360$ inputs or higher resolution for $\times2$ task),
a compact version of the WDSR block is chosen to fit the mobile GPU, where the largest filer number for each  CONV layer is 32, 146, and 28, respectively.
The backbone is initialized with the parameters of the pretrained WDSR model.
Traditional MAE loss is used to measure the differences between the SR image and the ground-truth as the SR loss. The parameter $\gamma$ in the training loss denoted as Eq.~\eqref{eq:loss} is set to 0.01. The first 20 epochs are used for the NAS process, and the following 30 epochs for fine-tuning the searched model. ADAM optimizers with $\beta_{1}{=}0.9$, $\beta_{2}{=}0.999$, and $\epsilon{=}\num{1e-8}$ are used for both model optimization and fine-tuning process. The learning rate is initialized as $\num{1e-4}$ and reduced by half at 10, 16 epochs and at 20, 25 epochs in the NAS and fine-tuning process, respectively. The details of the searched architecture are in Appendix D.

\noindent\textbf{Baseline Methods. } We compare with some traditional human-designed  SR models such as FSRCNN  and EDSR. Besides, some baselines optimizing the speed or hardware with NAS approaches are also covered. For example, TPSR-NOGAN, FALSR-C, ESRN-V optimize the SR efficiency to facilitate the deployment on end devices.   Moreover, we compare with some methods exploring the sparsity in SR models such as DHP, SMSR, and SRPN-L for efficient inference.

\begin{figure*}[t]
  \centering
    \includegraphics[width=0.9\textwidth]{figs/eccv_patch.pdf}
   \caption{Visual Comparisons with other   methods on Urban100/B100 for $\times$4 SR.}
   \label{fig:patch}
  \vspace{-0.2cm}
\end{figure*}

\subsection{Experimental Results}

\noindent\textbf{Comparison with Baselines on SR Performance.}
The comparisons of the models obtained by the proposed framework with state-of-the-art efficient SR works are shown in Table \ref{table:result_sr}. Two commonly used metrics (PSNR and SSIM) are adopted to evaluate image quality. The evaluations are conducted on $\times$2 and $\times$4 scales. For a fair comparison, we start from different low-resolution inputs but the high-resolution outputs are 720p ($1280{\times}  720$). To make a comprehensive study, the latency threshold  $v_T$ is set to different values. Specifically, as real-time execution typically requires at least 25 frames$/$sec (FPS), the latency threshold $v_T$ is set as 40ms to obtain SR models for real-time inference.

For $\times$2 scale, the model obtained with latency threshold $v_T$=100ms outperforms TPSR-NOGAN, LAPSRN, and CARN-M in terms of PSNR and SSIM with fewer parameters and MACs. Compared with FALSR-C, ESRN-V, EDSR, WDSR, SMSR, and SRON-L, our model greatly reduces the model size and computations with a
competitive image quality performance. By setting $v_T$ as 70ms, our model has similar parameters and MACs as MOREMNAS-C, but achieves higher PSNR and SSIM performance.  Similar results can be obtained on the $\times$4 scale.
Furthermore, for both scales, by setting $v_T$ as 40ms, we obtain extremely lightweight models and the models still maintain satisfying PSNR and SSIM performance on all   four datasets. Although SR-LUT  uses look-up tables for efficient SR inference, it suffers from more significant SR performance degradation.

The visual comparisons with  other SR methods for $\times$4 up-scaling task are shown in Fig.~\ref{fig:patch}. Our model can recover the details comparable or even better than other methods by using fewer parameters and computations.

\noindent\textbf{Comparison with Baselines on Speed Performance.}
In general,  our method can achieve real-time SR inference (higher than 25 FPS) for implementing 720p resolution up-scaling with competitive image quality in terms of PSNR and SSIM on mobile platforms (Samsung Galaxy S21). Compared with \cite{wang2021exploring} which also explore the sparsity of SR models, our method can achieve more significant model size and computation reduction (our 11GMACs v.s. 131.6GFLOPs \cite{wang2021exploring}  for $\times$2 scale), leading to faster speed (our 11.3ms v.s. 52ms \cite{wang2021exploring} on Nvidia A100 GPU).

\begin{figure}[t]
\begin{floatrow}
\ffigbox{
  \includegraphics[width=0.9\linewidth]{figs/Human_Design_vs_NAS.pdf}
}{
  \caption{Comparison of $\times2$ SR results between searched models and heuristic models on Set5  with latency measured on the GPU of Samsung Galaxy S21.}
  \label{fig:search_vs_human}
}

\capbtabbox{
\scriptsize\sffamily
    \centering
  \renewcommand{\arraystretch}{1.1}
\begin{adjustbox}{max width=0.45\textwidth}
\begin{tabular}{cc|c|cc|cc}
\toprule
\multicolumn{2}{c|}{Search Method}                & \multirow{2}{*}{ \makecell[c]{Latency \\ (ms)}} & \multicolumn{2}{c|}{Set 5 }    & \multicolumn{2}{c}{Urban100 } \\ \cline{1-2} \cline{4-7}
\multicolumn{1}{c|}{ \makecell[c]{Width \\ Search} } & \makecell[c]{Depth\\ Search} &                               &  PSNR  & SSIM &  PSNR  & SSIM \\
 \hline   \hline
\multicolumn{1}{c|}{\xmark }            &       \xmark      &   150.92 & 37.62 & 0.9589     & 31.03 & 0.9164    \\  \hline
\multicolumn{1}{c|}{\xmark }            &       \cmark      &   111.58 & 37.65 & 0.9591     & 31.10	& 0.9172    \\  \hline
\multicolumn{1}{c|}{\cmark }            &       \xmark      &   108.38 & 37.65 & 0.9591     & 31.02	& 0.9161    \\  \hline
\multicolumn{1}{c|}{\cmark }            &       \cmark      &   \textbf{98.90}  & 37.64 & 0.9591     & 31.08	& 0.9170    \\
\bottomrule
\end{tabular}
\end{adjustbox}
}{
    \caption{Comparison of different search schema for $\times2$ scales. The performance is  evaluated on Set5 and Urban100 datasets}
    \label{table:ablation}
}
\end{floatrow}
\end{figure}

\noindent\textbf{Comparison with Heuristic Models.}
We  compare  our searched models with heuristic models, which are obtained by evenly reducing the depth and width from the WDSR model. Since we do not search per-layer width in heuristic models, the width is the same among all blocks in one heuristic model. For a fair comparison, the same compiler optimization framework is adopted for both searched models and heuristic models.  As shown in Fig.~\ref{fig:search_vs_human}, we can see that the NAS approach can achieve faster inference than the heuristic models under the same PSNR, demonstrating the effectiveness of the search approach.

\noindent\textbf{Compiler Optimization Performance.}
To demonstrate the effectiveness of our compiler optimizations, we implement CARN-M \cite{ahn2018fast}, FSRCNN \cite{dong2016accelerating}, and our searched model with the open-source MNN framework.
By comparing  their PSNR and FPS performance, we find that our model can achieve higher FPS and PSNR  than the baseline models, with detailed results  in Appendix E.  We also compare with the compilation of \cite{huynh2017deepmon}  detailed in Appendix F.

\noindent\textbf{Performance on Various Devices.}
Our main results are trained and tested on the mobile GPU. We highlight that our method can be easily applied to all kinds of devices with their corresponding speed models. To demonstrate this, we perform compiler optimizations for the DSP on the mobile device and train the corresponding speed model. With the new speed model, we   use our method to search an SR model for the DSP, which can achieve 37.34 PSNR on   Set5   with 32.51 ms inference speed for $\times2$ up-scaling task, detailed  in Appendix G.

\subsection{Ablation Study}
For the ablation study, we investigate the influence of depth search and per-layer width search separately for $\times$2 scale task.
Multiple runs are taken for each search method with different latency threshold $v_T$ so that the searched models have similar PSNR and SSIM on Set5 to provide a clear comparison. From the results in Table \ref{table:ablation}, we can see that both depth search only and width search only can greatly reduce the latency with better image quality than non-search case. Specifically, as a missing piece in many prior SR NAS works, depth search provides better PSNR and SSIM performance than width search
on Urban100 with a slightly higher latency, which shows the importance of this search dimension. By combining depth search and width search, we could reach faster inference with similar PSNR and SSIM than conducting either search alone.
\section{Conclusion} \label{sec:conclusion}
We propose a compiler-aware NAS framework to achieve real-time SR on mobile devices. An adaptive WDSR block is introduced to conduct depth search and per-layer width search. The latency is directly taken into the optimization objective with the leverage of a speed model incorporated with compiler optimizations.
With the framework, we achieve real-time SR inference for the implementation of 720p with competitive SR performance on mobile. \\

\noindent\textbf{Acknowledgments.}
The research reported here was funded in whole or in part by the Army Research Office/Army Research Laboratory via grant W911-NF-20-1-0167 to Northeastern University. Any errors and opinions are not those of the Army Research Office or Department of Defense and are attributable solely to the author(s). This research is also partially supported by National Science Foundation CCF-1937500 and CNS-1909172.