\section{Further Examples of Code Transformations}
\input{appendix/comparative-examples}

\section{Pre-training progression for all metrics}

\begin{figure}[!htb]
    \centering
    \begin{subfigure}{0.25\linewidth}
    \centering
    \includegraphics[width=.95\linewidth]{images/rq1/lang_to_Exact_Match.pdf}

    \label{fig:lang_to_accuracy_pretrain_perf}
    \end{subfigure}%
    \begin{subfigure}{0.25\linewidth}
    \centering
    \includegraphics[width=.95\linewidth]{images/rq1/lang_to_Syntax_Match.pdf}

    \label{fig:lang_to_syntax_pretrain_perf}
    \end{subfigure}%
    \begin{subfigure}{0.25\linewidth}
    \centering
    \includegraphics[width=.95\linewidth]{images/rq1/lang_to_DFG_Match.pdf}

    \label{fig:lang_to_dataflow_pretrain_perf}
    \end{subfigure}%
    \begin{subfigure}{0.25\linewidth}
    \centering
    \includegraphics[width=.95\linewidth]{images/rq1/lang_to_CodeBleu.pdf}
    \label{fig:lang_to_codebleu_pretrain_perf}
    \end{subfigure}
    \vspace{-5mm}
    \caption{Progression of Different metrics of different language in Validation dataset over number pre-training steps.}
    \label{fig:pretrain_all_over_step}
\end{figure}

\section{Code Summarization Results}
\input{tables/code_summarization}

\section{Introduction}
\label{sec:intro}
Statistical models of the ``naturalness" of code~\cite{hindle2012naturalness} have proven useful for a range of  Software Engineering tasks~\cite{allamanis2018survey, pradel2021neural}, including code generation~\cite{amodio2017neural}, repair~\cite{chakraborty2020codit, tufano2019empirical},  summarization~\cite{liu2020retrieval},  retrieval~\cite{parvez2021retrieval}, and clone detection~\cite{white2016deep, ding2021contrastive}.
The earlier work in this area trained models directly on tasks, including the early work on  type recovery~\cite{hellendoorn2017deep, allamanis2020typilus}, de-obfuscation~\cite{raychev2014code,vasilescu2017recovering},  repair~\cite{gupta2017deepfix}, and summarization~\cite{iyer2016summarizing, ahmad2020summarization}.
Training on-task requires a lot of labeled data.
While labeled data is abundant for tasks like code completion (where the corpus inherently provides supervision), other tasks like code generation, translation, summarization, repair, etc., require well-curated, high-quality data.
Simply grabbing data from Github might yield poor-quality~\cite{gros2020code}, highly-duplicated data~\cite{allamanis2019adverse}.
With increasing model capacity (hundreds of millions, even billions of parameters, are pretty common; larger models tend to perform better~\cite{chen2021evaluating, wang2021codet5}), this unacceptable disparity between vast model capacity and the limited availability of well-curated, high-quality, labeled data has increased and will likely worsen.

 This shortage of high-quality labeled data for on-task training is not unique to Software Engineering (SE), although it is
 complicated here by the increased, specialized skill required
 for labeling SE data.

To address the issue of training large models in the presence of data scarcity, such models are often pre-trained on some generic tasks, which relate to actual downstream tasks.
 For example, consider two SE tasks: code generation and code translation. Both  tasks require ML models to learn how to generate natural, syntactically, and semantically correct code. This commonality across tasks motivates a quest for better pre-trained models, using a self- (or un-) supervised task
 which transfers well to other downstream tasks.
Such pre-trained models can also learn a generic representation of the input data, which, in turn, transfers to diverse downstream tasks.

A popular approach for dealing with this problem involves derivatives of BERT style models~\cite{devlin2018bert}, \hbox{\emph{e.g.,}}\xspace CodeBERT~\cite{feng2020codebert}, GraphCodeBERT~\cite{guo2020graphcodebert}, etc. These models are good at capturing generic code representations. For code generation tasks, GPT-3 or BART-style models (\emph{e.g.,} Codex, CodeT5,  PL\-BART, SPTCode, etc.~\cite{chen2021evaluating, ahmad2021unified, wang2021codet5, niu2022spt}) are popular. The important insight here
is that independent of final tasks, when \emph{very} high capacity models are trained with huge code corpora to learn simple, self-supervised, ``busy work'', they still \emph{learn general syntactic and semantic constraints of writing code}. Different approaches adopt different techniques to train the model to write code. For instance, GPT-style models (\hbox{\emph{e.g.,}}\xspace Codex) learn to generate code sequentially, mimicking the left-to-right language model. CodeT5 masks out some tokens and asks the model to generate {\em only} those masked tokens. On the other hand,  PLBART and SPT-Code present the model with erroneous code  (with deleted or masked tokens) and ask the model to generate the corrected, complete code. The  models' ability to generate code depends mainly on the pre-training objective that the model is optimized for.

We propose a novel pre-training task: we ask the model to ``naturalize" code, \hbox{\emph{i.e.,}}\xspace take ``weird", synthetic code as input and output semantic equivalent, ``natural" code that a human developer would have written. This is a very demanding pre-training task---the model has to learn both code naturalness \underline{\emph{and}} code semantics. We were inspired by noting the work of human Editors (of books, journals, newspapers): they digest imperfectly written but mostly correct text, understand the intent, and then produce more perfect text with pretty much the same meaning. Editing is \underline{\emph{hard}}: a skilled Editor has to have very high levels of language comprehension, to understand given, potentially badly-written text, \emph{and then} deploy very high-level writing skills to generate well-formed text. If Editing could be used as an at-scale pre-training task, the learned model would presumably have excellent language comprehension and also generate excellent text. However, it's not
obvious how to generate at-scale training data for this ``Editing" task, say, for English.

\input{body/example_code/natual_unnatural_for_into}

But our concern here is code, not natural language. We start with the argument that, because of the bimodal, dual-channel nature of code~\cite{casalnuovo2020theory}, it is indeed possible to generate at-scale training data for the Editing task (a.k.a. refactoring in Software Engineering  terminology). Code has a formal channel, with well-defined semantics; because of this, it's possible to transform code into endless forms, all \emph{meaning-equivalent}. Essentially, we can deploy a set of meaning preserving transformations to \emph{rewrite} existing code from widely-used GitHub projects (which presumably have good-quality code that has passed human code review). These rewrites, (\emph{e.g.,} \Cref{fig:code_for_into}), preserve meaning but will make the code into an artificial, often unnatural form\footnote{Studies, with  human-subjects~\cite{casalnuovo2020does,casalnuovo2020programmers} suggest that humans find such rewritten but semantically identical forms harder to read and understand.}.

Nevertheless, we now have a matched pair of two semantically equivalent forms of code: a ``de-naturalized" form and the original ``natural" form. Furthermore, we can produce these pairs at-scale, and then pre-train on a code ``Naturalization" task. By analogy with human Editors as described above, such pre-training forces the model to learn two hard things: 1) capture the meaning of the input code, and 2) generate an output that more closely resembles human-written code. We hypothesize that the resulting model will both learn better meaning representations, \emph{and also} generate better code.

To this end, we pre-trained our \textsc{NatGen}\xspace model, using ``Code Naturalizing'' task. \textsc{NatGen}\xspace is based on a transformer-based sequence-to-sequence model, and learns
to ``naturalize" artificially generated ``de-naturalized" code back into the form originally written by developers.
We {\em emphasize} that \textsc{NatGen}\xspace learns to generate the whole code; this learned skill  transfers to
downstream fine-tuning tasks that require code generation.
We show that our pre-training objective helps model generate more natural code (complete code, with high syntactic and semantic similarity with the original human-written code). With proper fine-tuning, \textsc{NatGen}\xspace achieves state-of-the-art performance in various downstream fine-tuning tasks, including code generation, code translation, bug fix, that demand code generation. We also show that \textsc{NatGen}\xspace is specially effective when labelled data is scarce.

We summarize our main contributions.

\begin{enumerate}[leftmargin=*]
     \item We introduce the idea of "Code naturalization" as a pre-training task.
     \item Using code from Github, and custom tooling, we have generated and  released a large dataset for pre-training models on the Naturalization task.
     \item We have built and released a large Sequence-to-Sequence model pre-trained on Naturalization.
     \item We show that (when appropriately fine-tuned) \textsc{NatGen}\xspace outperforms SOTA on several settings.
     \end{enumerate}

We publish our source code and data download script for pre-training \textsc{NatGen}\xspace anonymously in  \url{https://github.com/natgen-team/NatGen}.
We also share the pre-trained model in \url{https://bit.ly/3N0NGfG}.

\section{Background \& Problem Formulation}
\label{sec:background}

This section presents the relevant technical background that leads to this work and an overview of the main research questions.

\subsection{\textbf{The Dual Channels of Code}}
Humans can read and write both natural languages and code.
However, unlike natural language, source code involves
\emph{two} channels of information: formal \& natural~\cite{casalnuovo2020does}.
The formal channel, unique to code, affords precise, formal
semantics; interpreters, compilers, etc., use this channel.
On the other hand, the natural channel (perhaps more probabilistic and noisy) relies on variable names, comments, etc., and is commonly used by humans for code comprehension and communication~\cite{casalnuovo2020programmers,casalnuovo2020does}. The formal channel's precision enables semantic preserving code transformation,  which supports static analysis, optimization, obfuscation, \emph{etc}.   For instance, major refactoring of a source code may drastically change the syntactic structure while preserving the semantics~\cite{fowler2018refactoring, ding2021contrastive}. However, not all the semantically equivalent code is ``natural"~\cite{hindle2016naturalness}---the usual way developers write code and thus, amenable to statistical models~\cite{hindle2016naturalness}.
In fact, deviation from such ``naturalness" may lead to unintended bugs~\cite{ray2016naturalness}, and increase difficulty of human comprehension~\cite{casalnuovo2020programmers,casalnuovo2020does}.

We leverage the natural/formal duality for our pre-training objective in this work. We keep the formal channel constant (not changing the meaning) for a given code and modify the syntax by creating ``unnatural'' code. Then we train the model to take the ``unnatural" code as input and do what a human Editor does with natural language text: understand the ``unnatural" code and generate more natural code that a developer would write. Thus, the model simultaneously learns to both
comprehend code, \emph{and}  generate  ``natural'' code.

\subsection{\textbf{{``Naturalizing" \emph{vs.} De-noising}}}
Naturalizing pre-training essentially follows in the tradition of
\emph{denoising pre-training}, although, arguably, the former is more subtle and challenging.
Denoising pre-training~\cite{lewis2019bart, lachaux2020unsupervised, ahmad2021unified} is a well-established pre-training strategy for encoder-decoder models: the encoder is presented with a noised-up input, and the decoder is asked to generate the original, noise-free input. By training the model to identify \& remove ``noise'' in a noisy output,
(in theory) one teaches it to reason about and correctly generate text. Exactly what a model learns largely depends on the noise types. For instance, PLBART~\cite{ahmad2021unified} uses syntactic noise\footnote{Noise that breaks the syntax structure of code}(\hbox{\emph{i.e.,}}\xspace token masking, token deletion, etc.). Thus, denoising pre-training enables  PLBART to learn both about the syntax of input source code, \emph{and} learn to generate syntactically correct code.
Naturalizing pre-training, on the other hand, begins with syntactically correct but artificially-created \emph{unnatural} source code and forces the model to generate correct \emph{semantically equivalent natural} code that is just what a human originally wrote. Such pre-training requires more subtle changes to the code. We hypothesize that this provides a more demanding pre-training setting, which will lead to better on-task code generation performance.

\subsection{Research Questions}
\label{subsec_rqs}

Our hypothesis is that our \emph{naturalizing} task (see \cref{sec:semantic_denoising})
endows our pre-trained model with the ability to generate syntactically {and semantically} correct, \emph{and} natural code. This
leads to several RQs.

\RQ{1}{Does ``Naturalization'' help to improve code generation?} %help to generate more {\em natural} code?}

In contrast to existing de-noising techniques~\cite{ahmad2021unified} that help the model learn lexical \& syntactic structure, the naturalizing task, which is arguably more demanding than de-noising, forces \textsc{NatGen}\xspace generating better code with higher syntactic and semantic correctness.

The pre-training data we use (in \textsc{NatGen}\xspace) challenges the model to naturalize code that was ``de-naturalized" in several ways, such as dead-code inserted, variable renamed, etc. We investigate the relative performance under different naturalization challenges.

\RQ{2}{How do different components in \tool contribute to code generation?} % more {\em natural} code?}

We evaluate the performance under different challenges on a held-out validation dataset. This dataset is sampled with the same distribution of de-naturalizing transforms as the training dataset ($\mathcal{D}_t$); on this set, the model to reconstruct the original code.
Our exploratory investigation reveals that Variable Renaming is the hardest transformation to undo:  the model reconstructs original code with only $~$40\% accuracy. Dead Code, on the other hand, is the easiest with $~$99\% accuracy.

We further investigate \textsc{NatGen}\xspace's performance for downstream source code generation tasks

\RQ{3}{How effective is \tool when fine-tuned for different generative tasks in source code?}

We fine-tune the pre-trained \textsc{NatGen}\xspace on task-specific training dataset for a certain time budget and evaluate the fine-tuned model on the benchmark testing dataset for corresponding task. These tasks include source code (java) generation from text, code translation (from Java to C\# and C\# to Java), and Bug fixing. After fine-tuning, \textsc{NatGen}\xspace achieves the state-of-the-art performance in all these tasks. In addition, we also discover that, code generated by \textsc{NatGen}\xspace are syntactically and semantically more closer to the expected code.

We observe that training a model for a complex task requires sufficient labeled data. However, for most software engineering tasks, finding labeled data is a significant challenge~\cite{ahmad2021avatar}. We investigate potential scenario where size of the training data is extremely small.

\RQ{4}{How well does \tool's pre-training help in tasks where labelled data is scarce?}

We simulate training data scarcity in two different ways -- {\em Zero-shot learning}, and {\em Few-shot learning}.
For ``Zero-shot'' learning, we evaluate the pre-trained \textsc{NatGen}\xspace in different tasks \emph{without} any task specific fine-tuning.
For ``few-shot'' setting, we simulate training data scarcity by sub-sampling the benchmark training datasets.
We fine-tune the pre-trained \textsc{NatGen}\xspace on these limited training examples and measure the performance.
We observe that
\textsc{NatGen}\xspace is very efficient in low-data training. Since \textsc{NatGen}\xspace learns to generate syntactically and semantically correct code as part of pre-training, it faces less burden while learning in low-data training.

\section{SemCode}
\label{sec:method}

In this section, we discuss the methodology of \textsc{NatGen}\xspace. We divide \textsc{NatGen}\xspace's approach into three steps: (i) ``De-Naturalize'' source code to accumulate pre-training data for \textsc{NatGen}\xspace (\S\ref{sec:semantic_denoising}); (ii) pre-train \textsc{NatGen}\xspace using this data for naturalization task (\S\ref{sec:pretrain_train}); (iii) Finetune pre-trained \textsc{NatGen}\xspace with task specific dataset (\S\ref{sec:finetuning}).

\input{figures/approach}

\subsection{De-Naturalizing Source Code}
\label{sec:semantic_denoising}

Our pre-training approach involves generating very large volumes of code-naturalization training data. To de-naturalize code, we consider a set of semantic-preserving,  ``de-naturalizing'' code transformation patterns $\Phi = \{\phi_1, \phi_2,  \phi_3, ...\}$.
Given a code $c_i$, $\phi_j(c_i)$ transforms the code in a way that changes the structure of the code while maintaining the semantic equivalence with the previous code. These transformations are meaning-preserving but artificially transformed code from its original (human-) written form.

\Cref{fig:approach} shows the workflow of generating the pre-traininf dataset. Given a piece of source code ($c_i$), we first extract the AST ($T_{c_i}$), using a parser. We use tree-sitter\footnote{\url{https://tree-sitter.github.io/tree-sitter/}} to parse the source code. For the AST, we extract potential locations for de-naturalization. These locations are nodes ($n_k$) in $T_{c_i}$. While choosing location $n_k$ from $T_{c_i}$, we consult $\Phi$ -- we extract the nodes where at least one of $\phi_j \in \Phi$ is applicable. Once we have a set of such nodes, we filter out the transformation rules that cannot be applied to any node of in $T_{c_i}$. After such a filtration, we have a set of transformations $\Phi_a \subseteq \Phi$. At this stage, we randomly select one transformation pattern $\phi_j \in \Phi_a$ to apply at an application location (AST node) $n_k$.  We apply $\phi_j$ to $n_k$ to get the transformed node $n'_{k}$. We then structurally match $n'_{k}$ with the original AST $T_{c_i}$, specifically $n_k$. We adapt the context of $n_k$ to the transformed node's ($n'_k$) context. In that way, we get the transformed AST ($T'_{c_i}$) -- which we then translate to get the transformed code $c'_i$. We designed the transformation function $\phi_j$ and subsequent context adaptation in such a way that preserves the meaning or functionality of the original code. We use AST analysis and (approximated) data flow analysis on code AST.

\paragraph{\textbf{``De-naturalizing'' Transformations.}}
\label{sec:data_prep}
\input{body/example_code/transformations}
In this section, we describe semantic-preserving, de-naturalizing transformation functions. Casalnuovo \emph{et al}~\cite{casalnuovo2020programmers,casalnuovo2020does} showed that transforming code from what developers' had written to other forms generally makes it less natural (higher in cross-entropy according to language models) and also reduces human comprehension speed, and comprehension accuracy.
Nevertheless, we use six classes of such semantics-preserving,
de-naturalizing transformations.
These transformations are motivated by prior work on functional reasoning about source code~\cite{ding2021contrastive, gopstein2018prevalence, gopstein2020thinking} and semantic bug-seeding~\cite{patra2021semantic}.

\subsubsection{{Loop Transformation.}}
\label{sec:loop_transformation}
This transformation modifies {\tt for} loops into equivalent {\tt while} loop and vice-versa. We rewrite a {\tt while} loop of the form~\texttt{ while ( \highlight{condition} )}~\texttt{\{ loop-body \}} into a {\tt for} loop as ~\texttt{ for ( \highlight{\textbf{;}} \highlight{condition} \highlight{\textbf{;}} )}~\texttt{\{ loop-body \}}. \Cref{fig:loop_transformation} shows one such example. Likewise, to transform a {\tt for} loop into a {\tt while} loop, we move the initializer of the {\tt for} (if any) before the loop, and the update {\em expression} (if any) of the {\tt for} loop as the last {\em statement} in the loop. We also add this update statement before any loop breaking statement (\hbox{\emph{i.e.,}}\xspace \texttt{break}, \texttt{continue}). For example, we transform ``\texttt{for(\highlight{int i = 0;} i < 10; \highlight{i++})\{ if(i)\{ foo(); continue;\}  bar(); \}}'' as ``\texttt{\highlight{int i = 0;} while(i < 10)\{ if(i)\{ foo(); \highlight{i++;} continue;\}  bar(); \highlight{i++;}\}}''.

\subsubsection{{Dead Code Injection.}}
\label{sec:dead_code}
We inject blocks of dead code at random positions in the original code.
By ``dead code" we mean code that appears in the source but is
never executed.
For instance, \Cref{fig:dead_code} shows an example of dead code injection. Here we inject the code block \linecode{if( arr [mid] < key)\{ ...\}} at line 4 of the original code (\cref{fig:original_code}). To add challenge to the model, we transplant these inserted statements from the same input code. To ensure the "death" of inserted code, we put the inserted statements in a block headed by either a loop or a branch,
where  the condition of the loop or the branch can never be satisfied \hbox{\emph{i.e.,}}\xspace the code inside the block will never execute. In \Cref{fig:dead_code}, the condition \highlight{\tt i < i} is always \linecode{false}; and the code in line 5 is quite dead.

\subsubsection{{Block Swap.}}
\label{sec:block_swap}
In this transformation, we swap the ``then" block of a chosen {\tt if} statement with the corresponding {\tt else} block.  To preserve semantic equivalence, we negate the original branching condition. For instance, \cref{fig:block_swap} replaces the {\tt if} block (line 4 in \Cref{fig:original_code}) with the {\tt else} block (line 5 in \cref{fig:original_code}). We negate the original condition (\linecode{arr[mid] == key}) as (\linecode{arr[mid] != key}).

\subsubsection{{Operand Swap.}}
\label{sec:operand_swap}
Here, we swap the operands of binary (i) logical operations, and (ii) arithmetical operation (only addition and multiplication). For instance, we change the statement \linecode{high = mid + 1} with \linecode{high = 1 + mid} in line 6 in \Cref{fig:block_swap}. In case of logical operator, when swapping the operands, we also make sure the modified expression is the logical equivalent to the one before modification. For examples, when we rewrite \linecode{foo >= bar}, we rewrite as \linecode{bar <= foo}.

\subsubsection{{Confusing Code Insertion.}}
\label{sec:confusion_removal}
We introduce confusing code patterns in the code as outlined by \citet{gopstein2018prevalence, gopstein2020thinking}. In particular, we introduce two forms of confusing code. First, we modify the of the form \linecode{\{i = j; j += 1;\}} to \linecode{i = j++;}. Second, we introduce ternary operator wherever applicable. For example, we transform the code \linecode{if (x != 0) \{y = p;\} else \{y = q;\}} to \linecode{y = (x != 0) ? p : q;}.

\subsubsection{{Variable Renaming.}}
\label{sec:var_renaming}
We transform code by renaming some variables to {\tt VAR\_i}. While renaming a variable, we analyze the dataflow of that variable and rename all occurrences of that variable in the entire code. From all the variables used in the code, we change jsut a certain percentage. For instance, in \Cref{fig:var_renaming}, we renamed variable \linecode{arr} to \linecode{var_1}, and variable \linecode{high} to \linecode{var_2}, leaving all other variables unchanged. Note that, unlike other transformations, variable renaming does not create AST of Dataflow graph difference. However, this challenging task~\cite{allamanis2017learning} forces the model to learn to generate natural variable names.

\subsection{Pre-training}
\label{sec:pretrain_train}
Given an developer written code $c_i$, we first make it ``unnatural" using one of the randomly chosen transformation from the applicable transformations as described in \Cref{sec:data_prep}. Formally, we create ``unnatural'' code using a transformation $\phi_j$ as $\phi_j(c_i)$. We then use a neural sequence-to-sequence translation model ($\mathcal{M}$) to reconstruct $c_i$ from from $\phi(c_i)$, that is, we want $\mathcal{M}(\phi_j(c_i))$ to approximate $c_i$ . In particular, given a training dataset $\mathcal{D}_t = \{c_1, c_2, ...\}$ consisting of developers written code, set of "de-naturalizing" transformations $\Phi = \{\phi_1, \phi_2, \phi_3, ...\}$, we optimize the following function to learn $\mathcal{M}$'s optimal parameter $\Theta$.

\begin{equation}
    \Theta = \argmin_{\theta} \sum_{c_i\in \mathcal{D}_t} CrossEntropy\left(\mathcal{M}\left(\phi_j\left(c_i\right)\right), c_i\right)
\end{equation}

\subsection{Finetuning}
\label{sec:finetuning}
The objective of our pre-training is to learn to both comprehend and generate general-purpose source code. However, different tasks related to source code generation (\hbox{\emph{e.g.,}}\xspace text to code generation, code to code translation, bug fixing) call for task specific training of the pre-trained model to cater to the need of that task. This training phase on a pre-trained model is known as fine-tuning~\cite{devlin2018bert}. We consider the fine-tuning in \textsc{NatGen}\xspace as a translation task, we follow the standard procedure for transformer based-machine translation~\cite{vaswani2017attention}. First, the encoder generates the encoded representation $R(X)$ given the input $X= [x_1, x_2, ..., x_n]$. The decoder then sequentially generates the output $Y = [y_1, y_2, ..., y_m]$. While encoding an input token $x_k$, the encoder learn the attention matrix \hbox{\emph{w.r.t.}}\xspace every token in the input, including $x_k$ itself. Such attention matrix is known as {\em self-attention}. While generating an output token $y_m$, the decoder learn the attention matrix with all previsouly generated tokens $[y_1, y_2, ..., y_{m-1}]$ through {\em self-attention} and the encoder generated representation $R(X)$ through {\em cross-attention}. We refer to \citet{vaswani2017attention} for more detail about transformer based translation.

\section{Methodology}
\label{sec:method}

\input{body/example_code/transformations}

Our approach comprises three steps: (i) ``De-Naturalize'' source code to accumulate pre-training data for \textsc{NatGen}\xspace (\S\ref{sec:semantic_denoising}); (ii) pre-train \textsc{NatGen}\xspace using this data for naturalization task (\S\ref{sec:pretrain_train}); (iii) Fine-tune pre-trained \textsc{NatGen}\xspace with task specific dataset (\S\ref{sec:finetuning}).

\subsection{De-Naturalizing Source Code}
\label{sec:semantic_denoising}

For the first step above,
we use six rules to transform a natural code into its unnatural counterpart. These transformations are semantic-preserving but rewrite an original, natural, (human-) written code to an artificial form. Given a natural code element, we deploy an appropriate transformation, based on its AST structure and rewrite the code to ``de-naturalize'' it.

\subsubsection{Designing Transformation Rules.}
\label{sec:data_prep}

We use six classes of de-naturalizing transformations.
These transformations are motivated by prior work on functional reasoning about source code~\cite{ding2021contrastive, gopstein2018prevalence, gopstein2020thinking} and semantic bug-seeding~\cite{patra2021semantic}.
~\Cref{fig:transformations} show the details.

\textbf{Loop Transformation (\Cref{fig:loop_transformation}).}
This rule modifies {\tt for} loops into equivalent {\tt while} loop and vice-versa. We rewrite a {\tt while} loop of the form~\texttt{ while ( \highlight{condition} )}~\texttt{\{ loop-body \}} into a {\tt for} loop as ~\texttt{ for ( \highlight{\textbf{;}} \highlight{condition} \highlight{\textbf{;}} )}~\texttt{\{ loop-body \}}.
Likewise, to transform a {\tt for} loop into a {\tt while} loop, we move the initializer of the {\tt for} (if any) before the loop, and the update {\em expression} (if any) of the {\tt for} loop as the last {\em statement} in the loop. We also add this update statement before any loop breaking statement (\hbox{\emph{i.e.,}}\xspace \texttt{break}, \texttt{continue}). For example, we transform ``\texttt{for(\highlight{int i = 0;} i < 10; \highlight{i++})\{ if(i)\{ foo(); continue;\}  bar(); \}}'' as ``\texttt{\highlight{int i = 0;} while(i < 10)\{ if(i)\{ foo(); \highlight{i++;} continue;\}  bar(); \highlight{i++;}\}}''.

\textbf{Dead Code Injection (\Cref{fig:dead_code}).}
We inject blocks of dead code at random positions in the original code.
By ``dead code" we mean code that appears in the source but is
never executed.
In \Cref{fig:dead_code}, we inject the code block \linecode{high = mid + 1;} at line 4 of the original code (\cref{fig:original_code}). To add challenge to the model, we transplant these inserted statements from the same input code. To ensure the "death" of inserted code, we put the inserted statements in a block headed by either a loop or a branch, guarded by a unsatisfiable condition
so that the code inside the block will never execute. In \Cref{fig:dead_code}, the condition \highlight{\tt i < i} is always \linecode{false}; and the code in line 5 is quite dead.

\textbf{Block Swap (\Cref{fig:block_swap}).}
Here we swap the ``then" block of a chosen {\tt if} statement with the corresponding {\tt else} block.  To preserve semantic equivalence, we negate the original branching condition. For instance, \cref{fig:block_swap} replaces the {\tt if} block (line 4 in \Cref{fig:original_code}) with the {\tt else} block (line 5 in \cref{fig:original_code}). We negate the original condition (\linecode{arr[mid] == key}) as (\linecode{arr[mid] != key}).

\textbf{Operand Swap (\Cref{fig:block_swap}).}
Here, we swap the operands of binary logical operations. For instance, we change the expression \linecode{low <= high} with \linecode{high >= low} in line 2 in \Cref{fig:block_swap}. When swapping the operands of a logical operator, we change the operator to make sure the modified expression is the logical equivalent to the one before modification. In case of asymmetric inequality operators ($>$, $<$, $>=$, $<=$), we change the direction -- keep as is for symmetric operators (\hbox{\emph{i.e.,}}\xspace $==$, $!=$).

\textbf{Confusing Code Insertion (\Cref{fig:confusion}).}
We introduce confusing code patterns in the code as outlined by \citet{gopstein2018prevalence, gopstein2020thinking}. In particular, we introduce two forms of confusing code. First, we modify the of the form \linecode{\{i = j; j += 1;\}} to \linecode{i = j++;}. Second, we introduce ternary operator as applicable. For example, we transform the code \linecode{if (x != 0) \{y = p;\} else \{y = q;\}} to \linecode{y = (x != 0) ? p : q;}.

\textbf{Variable Renaming (\Cref{fig:var_renaming}).}
We rename some variables to {\tt VAR\_i}. While renaming a variable, we analyze the dataflow of that variable and rename all occurrences of that variable in the entire code. From all the variables used in the code, we change just a certain percentage. For instance, in \Cref{fig:var_renaming}, we renamed variable \linecode{arr} to \linecode{var_1}, and variable \linecode{high} to \linecode{var_2}, leaving all other variables unchanged. Note that, unlike other transformations, variable renaming does not create AST of Dataflow graph difference. However, this challenging task~\cite{allamanis2017learning} forces the model to learn to generate natural variable names.
This resembles the de-obfuscation pre-training task of~\cite{roziere2021dobf}.

\input{figures/approach}

\subsubsection{Applying Transformation.}
\label{sec:transform}
Assume a set of transformation rules $\Phi = \{\phi_1, \phi_2,  \phi_3, ...\}$.
Given  original code $c_i$, $\phi_j(c_i)$ transforms the code, changing the structure while preserving semantics. \Cref{fig:approach} shows how to apply such transformation to $c_i$. It works in three steps:
\begin{itemize}[leftmargin=*]
    \item {\em Find Transformation Location.} Given a piece of source code ($c_i$), we first use tree-sitter\footnote{\url{https://tree-sitter.github.io/tree-sitter/}}
    to parse out the AST ($T_{c_i}$).
    From the AST, we extract potential locations for de-naturalization. These locations are nodes ($n_k$) in $T_{c_i}$. While choosing location $n_k$ from $T_{c_i}$, we consult $\Phi$ -- we extract the nodes where at least one of $\phi_j \in \Phi$ is applicable.

    \item {\em Select Transformation Rule.}  Once we have a set of such nodes, we filter out the transformation rules that cannot be applied to any node of in $T_{c_i}$. After such a filtration, we have a set of transformations $\Phi_a \subseteq \Phi$. At this stage, we randomly select one transformation pattern $\phi_j \in \Phi_a$ to apply at an application location (AST node) $n_k$.

    \item {\em Apply Transformation.} We apply $\phi_j$ to $n_k$ to get the transformed node $n'_{k}$. We then structurally match $n'_{k}$ with the original AST $T_{c_i}$, specifically $n_k$. We adapt the context of $n_k$ to the transformed node's ($n'_k$) context. In that way, we get the transformed AST ($T'_{c_i}$), which we then translate to get the transformed code $c'_i$.
\end{itemize}

We designed the transformation function $\phi_j$ and subsequent context adaptation in such a way that preserves the meaning or functionality of the original code. We use AST analysis and (approximated) data flow analysis on code AST.

\subsection{Pre-training}
\label{sec:pretrain_train}
Once we have a pool of  ``unnatural'' code using the transformation in \Cref{sec:semantic_denoising} (\hbox{\emph{i.e.,}}\xspace transform code $c_i$ as `un-natural' code $\phi_j(c_i)$),
we use a neural sequence-to-sequence translation model ($\mathcal{M}$) to reconstruct $c_i$ from $\phi(c_i)$, \hbox{\emph{i.e.,}}\xspace we want $\mathcal{M}(\phi_j(c_i))$ to approximate $c_i$ . In particular, given a training dataset $\mathcal{D}_t = \{c_1, c_2, ...\}$ consisting of developers written code, set of   ``de-naturalizing'' transformations $\Phi = \{\phi_1, \phi_2, \phi_3, ...\}$, we optimize the following function to learn $\mathcal{M}$'s optimal parameter $\Theta$.
\begin{equation}
    \Theta = \argmin_{\theta} \sum_{c_i\in \mathcal{D}_t} CrossEntropy\left(\mathcal{M}\left(\phi_j\left(c_i\right)\right), c_i\right)
\end{equation}

\subsection{Fine-tuning}
\label{sec:finetuning}
The objective of our pre-training is to learn to both comprehend and generate general-purpose source code. However, different tasks related to source code generation (\hbox{\emph{e.g.,}}\xspace text to code generation, code to code translation, bug fixing) call for task-specific training of the pre-trained model. This training phase on a pre-trained model is known as fine-tuning~\cite{devlin2018bert}. We consider the fine-tuning in \textsc{NatGen}\xspace as a translation task and follow the standard transformer based-machine translation procedure~\cite{vaswani2017attention}. First, the encoder generates the encoded representation $R(X)$ given the input $X= [x_1, x_2, ..., x_n]$. The decoder then sequentially generates the output $Y = [y_1, y_2, ..., y_m]$. While encoding an input token $x_k$, the encoder learns the attention matrix \hbox{\emph{w.r.t.}}\xspace every token in the input, including  $x_k$.
Such attention matrix is known as {\em self-attention}. While generating an output token $y_m$, the decoder learns  the attention matrix with all previously generated tokens $[y_1, y_2, ..., y_{m-1}]$ through {\em self-attention} and the encoder generated representation $R(X)$ through {\em cross-attention}. We refer to \citet{vaswani2017attention} for more detail about transformer-based translation.

\section{Experimental Setup}
\label{sec:experiemnts}

This section details the experimental design of \textsc{NatGen}\xspace.

\paragraph{\textbf{Pre-training data.}}
Following prior works~\cite{feng2020codebert, guo2020graphcodebert, wang2021codet5}, we primarily use CodeSearchNet~\cite{husain2019codesearchnet} dataset for the pre-training purpose. CodeSerachNet is a publicly available dataset with six languages: Java, Python, Go, JavaScript, Ruby, and PHP. In addition to CodeSearchNet, CodeT5 uses additional data for C and C\#.
We also use 1M functions each for C and C\#. For these two additional languages, we collected 5000 active projects from GitHub and randomly selected 1M functions considering the maximum sequence length of the model.
\input{tables/finetune_dataset_stat}

\paragraph{\textbf{Fine-tuning data.}}
We evaluate different variations of three benchmark tasks related to source code generation. The first task is {\em Text to Code generation}, where the input is an NL description of a Java method, and the output is the code. The second task is {\em Code Translation} between {\em Java to C\#} and {\em C\# to Java}. For this task, we evaluate Java-C\# parallel dataset proposed by ~\citet{CodeXGLUE}. The third and final task is {\em Bug Fix}, where the given a buggy code and a summary of the fix model generates the fixed code. For this task, we used the two different versions of the dataset (small, with less than 50 tokens and medium with up to 100 tokens) proposed by \citet{tufano2019learning}. Note that, similar to MODIT~\cite{chakraborty2021on}, we evaluate on {\em concrete} version of the refinement datasets.\Cref{tab:finetune_data_stat} shows the datasets and their statistics. For Text to Code Generation and Code Translation, we reuse the same split from CodeXGLUE~\cite{CodeXGLUE}, and for Bug Fix, we reuse the same split as MODIT.

\paragraph{\textbf{Pre-training Model Configurations.}}
\label{sec:pretrain_exp}
We use 12 layer transformers with 12 attention heads on both encoder and decoder following the CodeT5~\cite{wang2021codet5} architecture. As discussed in \Cref{sec:method}, we use de-naturalization generative objectives for pre-training.
We initialize our model with CodeT5's~\cite{wang2021codet5} released parameters.
In particular, we initialize \textsc{NatGen}\xspace with ``CodeT5-base'' model.
We pre-train \textsc{NatGen}\xspace on 2 Nvidia GeForce RTX 3090 GPUs for 25K steps, maintaining the effective batch size at 1080 with learning rate 5e-5. We train \textsc{NatGen}\xspace for approximately 168 hours.

\paragraph{\textbf{Evaluation Metric.}}
\label{sec:metric}
Throughout the experiments in this work, we  evaluate accuracies w.r.t. exact match (EM), Syntax match (SM), Dataflow match (DM), and CodeBLEU (CB)~\cite{ren2020codebleu}.
SM is the proportion of matching subtrees between output code and tadget code's ASTs \hbox{\emph{w.r.t.}}\xspace number of all possible subtrees in the target code's AST.
DM is the percentage of matched (with target code) anonymized dataflow edge (def-use edge) of output code \hbox{\emph{w.r.t.}}\xspace all dataflow edges in the target code.
Note that, both the SM and DM are components of CB. We explicitly evaluate these for understanding the syntactic and semantic correctness of generated code. We reuse Microsoft CodeXGLUE tool~\cite{codebleu_calculator} to compute SM, DM, and CB.

\paragraph{\textbf{Baselines.}}
\label{sec:common_baselines}
While comparing the evaluation results for different tasks, we compare with large scale pre-trained models, including GPT-2~\cite{radford2019language}, CodeGPT~\cite{CodeXGLUE}, PLBART~\cite{ahmad2021unified}, SPT-Code~\cite{niu2022spt} and CodeT5~\cite{wang2021codet5}. Most of our fine-tuning evaluation is on benchmarked dataset; thus, we report the available results from CodeXGLUE leaderboard~\cite{cxg_leaderboard}. There are some task specific baselines, which we discuss while describing corresponding task.

\section{Empirical Results}
\label{sec:result}

We evaluate \textsc{NatGen}\xspace on (i) pre-training and (ii) three fine-tuning tasks. We also check \textsc{NatGen}\xspace's effectiveness in zero-shot and few-shot settings.

\input{body/RQs/rq1}
\input{body/RQs/rq2}
\input{body/RQs/rq3}
\input{body/RQs/rq4}

\section{Limitations \& Threats}
\label{sec:discussion}

\paragraph{\textbf{Bias introduced by `de-naturalizing' transformations.}}
In \Cref{sec:semantic_denoising}, we described our six transformations to ``de-naturalize" source code. The \textsc{NatGen}\xspace model learns to revert one transformation at a time. In fact, we found empirically that, when given code with more than one `de-naturalization' transformation applied, the model reverses only one of them. There is thus a threat our limited
application of de-naturalization limits the ability of
our \textsc{NatGen}\xspace.
Regardless, we consider \textsc{NatGen}\xspace as a proof-of-concept and the first work towards teaching a model to write natural code. We leave the investigation more natural code patterns and their effect on code generation as a potential future work.

\paragraph{\textbf{Knowledge retention from CodeT5.}}
As mentioned in \Cref{sec:pretrain_exp}, we start \textsc{NatGen}\xspace's pre-training from CodeT5-base model~\cite{wang2021codet5}. Starting further pre-training from an existing pre-trained checkpoint is very common in large-scale pre-training. For instance, GraphCodeBERT~\cite{guo2020graphcodebert}  is pre-trained based on CodeBERT~\cite{feng2020codebert} model, which was pre-trained based on RoBERTa~\cite{liu2019roberta} model. Both the Open AI-CodeX~\cite{chen2021evaluating} and Github Copilot~\cite{copilot} models are further pre-trained in OpenAI-GPT3~\cite{brown2020language}. Nevertheless, when we further train a pre-trained model on different tasks, it is subject to ``{\em catastrophic forgetting}''~\cite{kirkpatrick2017overcoming} of the knowledge learned in the base model. In order to test whether \textsc{NatGen}\xspace is forgetting CodeT5's knowledge about natural language generation, we also evaluate \textsc{NatGen}\xspace for Code summarization. Here the input is source code, and the output is Natural language. After fine-tuning \textsc{NatGen}\xspace's overall BLEU in 19.547 while CodeT5's was 19.551, suggesting that \textsc{NatGen}\xspace mostly retains CodeT5's capacity to generate NL.

\paragraph{\textbf{Fair Comparison with CodeT5.}}
We initialize \textsc{NatGen}\xspace with pre-trained checkpoint from CodeT5 (already pre-trained 75K steps with their objective) and train \textsc{NatGen}\xspace for 25K steps with `natural-code' writing objective. A skeptic reader would want to know what happens when we pre-train CodeT5 for 25K more steps with their training objective. We argue that since the pre-training objective does not explicitly account for generating code (See section 3.2 of CodeT5's original paper), further training with the CodeT5 objective does not necessarily increase its code generation capacity. We do acknowledge CodeT5's ability to understand and reason about {\em input}. Since the pre-training large model is extremely expensive~(\S\ref{sec:experiemnts})\footnote{CodeT5 was pre-trained on 16 NVIDIA A100s, with 40G memory each, for 12 days! One might reasonably assume it was already
well-trained on the original objective}; we leverage such knowledge by initializing \textsc{NatGen}\xspace from CodeT5's publicly available pre-trained model. Moreover, CodeT5 release neither their code for pre-training (only for fine-tuning), nor any earlier or later checkpoints for us to carry out further investigation.

\paragraph{\textbf{``Naturalization'' with program-analysis.}}
\textsc{NatGen}\xspace is a prototype of a generative pre-trained model with ``Naturalization'' task, trained to revert six classes of de-naturalization transformations (see
\Cref{fig:transformations}). However, perfect performance \hbox{\emph{w.r.t.}}\xspace these transformation is {\bf not} the main objective of this research. Tools to accomplish ``naturalization" could surely be built using traditional refactoring methods;
however, our goal is to train \textsc{NatGen}\xspace so that it learns to generate natural code with the help of this ``Naturalization'' task.

\paragraph{\textbf{\textsc{NatGen}\xspace as ``Code-Refactoring'' tool.}}
\textsc{NatGen}\xspace suggests the promise of neural transformers to
build meaning-preserving code-refactoring tools. However,
to realize a more accurate and powerful neural re-factoring
tool,  more training data, with a larger variety of transformations, would be required. We leave this as future work.

\section{Related Works}
\label{sec:related}
The approach of pre-training large Transformers without human labels started in NLP
domain with BERT~\cite{devlin2018bert}, which introduces two pre-training objectives (i.e., Mask Language Modeling and Next Sentence Prediction). Later, Liu et al. show that RoBERTa~\cite{liu2019roberta} outperforms BERT only using Mask Language Modeling (MLM) with new training strategies and hyper-parameter tuning. MLM is a self-supervised task that the model randomly masks or modifies a certain number of tokens and tries to recover them.

Following the success of the pre-trained model in the NLP domain, researchers applied these models to code related tasks.
CodeBERT is one of the earliest that was specially trained for code and relevant natural language descriptions. It is pre-trained with two objectives (i.e., MLM and Replaced Token Detection~\cite{clark2020electra}) and demonstrated pre-training's effectiveness for code. Later, an architecturally equivalent model, GraphCodeBERT, was introduced; it improved over CodeBERT on most tasks by incorporating data-flow information.

Though CodeBERT~\cite{feng2020codebert} \& GraphCodeBERT~\cite{guo2020graphcodebert} do well at code understanding tasks, these models are not as good at generative tasks. Both models are encoder-only and have to start with an untrained decoder in fine-tuning for generative tasks, such as code repair, code generation, code summarization, and code translation. To address this limitation, Ahmad et al. introduced PLBART~\cite{ahmad2021unified}, pre-trained as a generative denoising autoencoder. A specific set of noises is introduced to code and relevant natural language description and used as the input to the model. The model's objective is to encode the noisy input in the encoder and generate noise-free code or text in the decoder. PLBART (builds on BART~\cite{lewis2019bart}) outperforms both CodeBERT~\cite{feng2020codebert} and GraphCodeBERT~\cite{guo2020graphcodebert} on both understanding and generative tasks with a pre-trained encoder and decoder~\cite{ahmad2021unified}. DOBF~\cite{roziere2021dobf} uses de-obfuscation (recovering variable names) as their pre-training task; however, rather than generating code, they just generate a dictionary
of recovered names.

CodeT5~\cite{wang2021codet5} (based T5~\cite{raffel2019exploring}) is the latest denoising model. CodeT5 uses the developer-assigned identifiers in code, adding two code-specific pre-training objectives to the original T5, identifier tagging and masked identifier prediction. CodeT5 is an encoder-decoder model and excels at both understanding and generative tasks compared to other models. Similar to CodeT5, \cite{phan2021cotext,mastropaolo2021studying} are also built based on T5 architecture and perform reasonably well in the different downstream tasks. \textsc{NatGen}\xspace has a similar architecture to CodeT5; but rather than CodeT5's pre-training objectives, we ``de-naturalize" code, using the formal channel of code to inject meaning-preserving transforms, and then force \textsc{NatGen}\xspace to recreate, the original, ``natural" code. Rewriting semantically equivalent code requires semantic understanding, and that can be applied to code only because of its dual-channel nature. Our evaluation shows that rewriting semantically equivalent programs in the pre-training stage results in
performance gains in at least three popular Software Engineering tasks.

\section{Threats to Validity}
\label{sec:threats}

\paragraph{Internal Validity.}
In \Cref{sec:semantic_denoising}, we choose one of six transformation patterns to ``de-naturalize" source code. Thus, the model learns to ``naturalize" one transformation at a time. In fact, we observed examples, where we give the model with more than one `un-natural' code-patterns, and model reverses only one pattern. There is a threat that these patterns create biases towards learning to write ``natural'' code, since all natural code are not bound by these simple patterns. Regardless, we consider \textsc{NatGen}\xspace as a proof-of-concept and the first work towards teaching a model to write natural code. We leave the investigation more natural code patterns and their effect on code generation as a potential future work.

\paragraph{Construct Validity.}
We initialize \textsc{NatGen}\xspace with pre-trained checkpoint from CodeT5 (already pre-trained 75K steps with their objective) and train \textsc{NatGen}\xspace for 25K steps with `natural-code' writing objective. A skeptic reader would want to know what happens when we pre-train CodeT5 for 25K more steps with their training objective. We argue that since the pre-training objective does not explicitly account for generating code (we refer readers to section 3.2 of CodeT5's original paper), any further training with the CodeT5 objective does not necessarily increase its code generation capacity. We do acknowledge CodeT5's ability to understand and reason about {\em input}. Since the pre-training large model is extremely expensive~(\S\ref{sec:experiemnts}), we leverage such knowledge by initializing \textsc{NatGen}\xspace from CodeT5's publicly available pre-trained model. Moreover, CodeT5 did not release their code for pre-training, nor any earlier or later checkpoints for us to carry out further investigation.

\subsection{External Threat.}

\section{Conclusion}
\label{sec:conclusion}
We introduce the ``Code-Naturalization'' pre-training objective for generative models of code.
As proof-of-concept  we pre-trained our \textsc{NatGen}\xspace to write `natural' source code from `un-natural' counterpart.
With this pre-training, \textsc{NatGen}\xspace learns to write code syntactically and semantically closer to developers' written code.
We ``de-naturalize''  existing developers'  code, using six kinds of ``semantic-preserving'' transformations.
We further fine-tune the \textsc{NatGen}\xspace on different variations of three downstream tasks that require code generation.
\textsc{NatGen}\xspace achieves state-of-the-art performance in these downstream tasks, and \textsc{NatGen}\xspace's generated code are syntactically and semantically closer to the target code.
Our pre-training on the `naturalizing' task is especially effective in resource-constrained setting \hbox{\emph{i.e.,}}\xspace zero-shot, and few-shot transfer learning.

\subsection{Effectiveness of Semantic Denoising on Pretraining}
\subsection{\textsc{NatGen}\xspace's Effectiveness on pre-training}
\label{sec:rq1}

\RQrepeat{1}{Does ``Naturalization'' help to improve code generation?} %help to generate more {\em natural} code?}

\paragraph{\underline{Motivation.}}
We investigate whether pre-training on naturalizing task helps the model generate  correct and natural code
(code that is syntactically and semantically similar to the original code).

\paragraph{\underline{Experimental Setup.}}
We compare three large scale pre-trained models: (i) CodeT5~\cite{wang2021codet5}, (ii) PLBART~\cite{ahmad2021unified}, and (iii) \textsc{NatGen}\xspace.
Note that, since PLBART is only pre-trained on Java and Python, we compare PLBART only for those languages, with the corresponding results of other models.
We ask each of these models to reconstruct developers' written code from its de-naturalized (but semantically identical, see \S\ref{sec:semantic_denoising} \& \S\ref{sec:data_prep}) variants. We use the held-out validation data from our training procedure for this evaluation.
We evaluate the models for generating the Exact Match (EM), Syntax Match (SM) and Dataflow Match (DM).

\input{tables/pretrain_evaluation}

\paragraph{\underline{Results.}}
\Cref{tab:pretrain_eval_full} shows the evaluation results.

\noindent
$\boldsymbol{\cdot}$~\textit{Syntax Match.} We find that the code generated by PLBART and \textsc{NatGen}\xspace are mostly syntactically correct. However, CodeT5's does not always generate syntactically valid code, suggesting an advantage for naturalization pre-training.
For instance, \Cref{fig:example_code_for_pretraining} shows code generated by different models from the given input.
As we can see, CodeT5 generates a syntactically erroneous fragment. In contrast, PLBART made a minor edit on the input code, just removing the \linecode{protected} keyword. Both PLBART and \textsc{NatGen}\xspace are pre-trained to generate complete code rather than fragments (which is the case of CodeT5~\cite{raffel2019exploring}); thus, the former two generally do better at generating syntactically correct code.

\input{figures/rq0/example_code}

\noindent
$\boldsymbol{\cdot}$~\textit{Semantic Match.}
\textsc{NatGen}\xspace is effective at recovering developers' written
code from its de-naturalized semantic variants---around 70\% of the generated code (CodeBlue = 97\%) {\em exactly matches} the original code.
PLBART, which deploys syntactic denoising, is at the second position in terms of CodeBlue.

\textsc{NatGen}\xspace also dominates the other two models in generating syntactically (SM) \& semantically (DM) valid code.
While PLBART appears to generate syntactically correct code, it mostly copies code from the input---median edit distance from PLBART's input and the generated code is 3 (see \Cref{tab:pretrain_eval_full}). In fact, in 7.05\% of cases, PLBART just copies the input! By contrast, \textsc{NatGen}\xspace learns to generate \emph{variants} of the input code, with only 0.01\% direct copy and a median edit distance of 10. Since PLBART is trained to remove syntax errors from the input, we conjecture that it does not inherently learn the semantic variation of the code. By contrast, we expose \textsc{NatGen}\xspace to semantic code variations, forcing it to learn to generate code that is both more natural \underline{\emph{and}} semantically equivalent.

$\boldsymbol{\cdot}$~\textit{Closer look into CodeT5.}
Unlike \textsc{NatGen}\xspace and PLBART, CodeT5 is not explicitly trained to generate complete code. During pre-training, CodeT5 learned to ``unmask'' masked token sequences. Thus, to better measure CodeT5's generation capacity, we conduct another experiment where we replaced all occurrences of some of the variable names in code with a special {\tt MASK1}, {\tt MASK2} tokens and asked CodeT5 to generate. This is one of the objectives (masked identifiers prediction) CodeT5 is pre-trained to optimize. We take the CodeT5's output and identify all potential identifiers~\footnote{we use regex \linecode{"[A-Za-z\_]+[A-Za-z0-9_]*"} to find identifiers.}. Surprisingly, in {\em only} 0.27\% of the cases, could CodeT5  generate {\em all} the variables, and in 0.61\% of cases {\em half} of the masked variables., while \textsc{NatGen}\xspace successfully translates 40.45\% of those examples back to its original code, including correctly predicting the replaced variable names. In addition, CodeT5's generated token sequence contained a lot of other tokens than the variable names (\cref{fig:example_code_for_pretraining}.4, for example).

\RS{1}{

    Naturalization enables \textsc{NatGen}\xspace to reason about code semantics and thus help generate more natural code variants than existing pre-training models and pre-training objectives.
}
\subsection{Evaluation of \textsc{NatGen}\xspace's components}

We also did an ablation study evaluating the effect of \textsc{NatGen}\xspace's different components on the results.

\RQrepeat{2}{How do different components in \tool contribute to code generation?} % more {\em natural} code?}
\paragraph{\underline{Motivation.}}
\label{sec:rq2}

In this RQ, we study how different transformation rules (see \S\ref{sec:semantic_denoising}
contribute to learn generating natural code from  different semantic variants .
We also evaluate how well \textsc{NatGen}\xspace learns that in different programming languages over training time.

\paragraph{\underline{Experimental Setup.}}  While pre-training, we checkpoint the \textsc{NatGen}\xspace model every 1k training steps,  for a full run of 25k steps. At each checkpoint, we evaluate the naturalization task performance. Before training, we held out 0.1\% of the total data as validation data. Note that, since our goal in this experiment is to understand \textsc{NatGen}\xspace's pre-training better, we ``de-naturalized" the validation data using the same training data distribution. This setting gives us a controlled environment for experimentation.

\input{figures/rq1/transformation_performances}

\paragraph{\underline{Results.}}

\Cref{fig:transfromation_performance} shows \textsc{NatGen}\xspace's performance under different types of semantic variants.
Results show that \textsc{NatGen}\xspace  has most trouble recreating the original code (just 40\% Exact Match) with the variable renaming task.
Variable renaming is challenging even for human developers~\cite{allamanis2015suggesting}---different developers may propose different names for the same object. Nevertheless, on this task, \textsc{NatGen}\xspace achieves good syntax and dataflow match (99\% and 92\% respectively), indicating that \textsc{NatGen}\xspace preserves syntax or semantics in most cases while generating code with renamed variables.

On the other hand, \textsc{NatGen}\xspace can eliminate Dead Code
with 99\% accuracy.  This result may be an artifact of our specific implementation of this transformation. Our dead-code insertion rule is simple, and formulaic; so the \textsc{NatGen}\xspace quickly learns to identify and remove such dead code. A more complex pattern of dead code may challenge the model more, and help make it more robust; we leave this for future work. For naturalizing other transformations, \textsc{NatGen}\xspace achieves more than 80\% exact match accuracy for Block swap and Confusion removing, and more than 75\% exact match accuracy for the rest. In all cases, syntax match, dataflow match, and CodeBLEU are well above 90\%.

\input{figures/rq1/codebleu_per_language}

 \Cref{fig:pretrain_perf_over_step} shows how validation performance improves for different languages, with more training steps. Across all the languages the performance rapidly increases over the first few thousand training steps. In fact, at the beginning of (step 0) of \textsc{NatGen}\xspace's pre-training, the overall exact match is 0, syntax match is 13.93\%, dataflow match is 19.86\%  and CodeBLEU is 9.74\% (see \Cref{tab:pretrain_eval_full} for details\footnote{
 \textsc{NatGen}\xspace's pre-training start from CodeT5-base. Thus,
 CodeT5-base is \textsc{NatGen}\xspace's checkpoint at step 0.}). However, after just 1000 steps of training, the exact match rises to 61\%, syntax match to 97\%, dataflow match to 94\%, and CodeBLEU to 95\%. These metrics continue improving as  training progresses. These results confirm that across all the languages \textsc{NatGen}\xspace gradually learns to generate more natural code from semantic variants.

\RS{2}{
pre-training performance depends on the types of semantic variants---while variable renaming seems the most difficult ($\sim$40\% accuracy), dead-code elimination appears to be an easier task ($\sim$99\% accuracy) to learn.
}
\subsection{\textsc{NatGen}\xspace in Downstream Tasks}.
\subsection{\textsc{NatGen}\xspace's Effectiveness on Fine-tuning Tasks}
\label{sec:rq3}

This section evaluates \textsc{NatGen}\xspace's performance on three benchmark source code generative tasks.

\RQrepeat{3}{How effective is \tool when fine-tuned for different generative tasks in source code?}

\input{tables/code-generation}

\paragraph{\underline{Baselines.}}
In addition to the baselines discussed in \Cref{sec:common_baselines}, for the {\em Text to Java Code generation} task, we compare with a group of baselines with no pre-training involved.
These baselines include LSTM based Sequence to sequence models, \citet{guo-etal-2019-coupling}'s, and \citet{iyer2018mapping}'s proposed techniques.
We also report our reproduced version of CodeT5 results in different tasks, slightly different from what they reported.
For both the {\em Bug Fix} task, we compare with the reported results of MODIT~\cite{chakraborty2021on} and our reproduced CodeT5 result.
\input{figures/rq3/zero-shot-boxplot}

\paragraph{\underline{Results.}}~~\\

\vspace{-3mm}
\noindent\textbf{Text to Code Generation.}
\Cref{tab:code_generation} shows evaluation results for text to code generation.
We trained for 30 epochs. We stopped the training is the validation performance does not increase for more than three(3) consecutive epochs. For both CodeT5 and \textsc{NatGen}\xspace, we report the performance of final model after the fine-tuning terminated ($\mathcal{M}_{last}$) and the performance of the model with best validation perfomance ($\mathcal{M}_{best}$).
Interestingly, for both CodeT5 and \textsc{NatGen}\xspace, the $\mathcal{M}_{last}$ model performs better than the corresponding $\mathcal{M}_{best}$ model.
The result shows that \textsc{NatGen}\xspace's generated code are more syntactically and semantically closer to the target code.
The $\mathcal{M}_{last}$ model of \textsc{NatGen}\xspace outperforms CodeT5's $\mathcal{M}_{last}$ model by 2.8\% in SM, 5.28\% in DM and 4.74\% in CB.
We conjecture that \textsc{NatGen}\xspace's pre-training with ``naturalization'' help generate more natural code.

\input{tables/code_translation}

\noindent\textbf{Code Translation.}
\Cref{tab:code_translation} shows the results of \textsc{NatGen}\xspace and different baselines for Code Translation.
For Java to C\# translation, \textsc{NatGen}\xspace achieves exact match accuracy of 66.2\% while CodeT5's accuracy is 65.9\%.
In C\# to Java translation, \textsc{NatGen}\xspace achieves 67.3\% exact match accuracy, which CodeT5 achieves 66.0\%.
In addition, the syntactic match (SM), Dataflow match, and CodeBLEU are also higher than that of CodeT5.

\input{tables/code_refinement}

\input{figures/rq3/few-shot-boxplot}
\input{figures/rq3/few-shot-progress/few-shot-progress-big-figure}

\noindent\textbf{Bug Fix.}
Similar to MODIT, we evaluate the top-1 accuracy of the generated fixed code. We also evaluate uni-modal settings, where the fix description is unavailable, and multi-modal settings, where we have access to the fix description.
\Cref{tab:code_refinement} shows the results of Bug Fix.
For the BugFix$_{small}$ dataset, \textsc{NatGen}\xspace outperforms both CodeT5 and MODIT in both unimodal and multi-modal settings.
For For the BugFix$_{medium}$ dataset, \textsc{NatGen}\xspace performs better than CodeT5 and MODIT in unimodal setting and slightly worse than CodeT5 in the multi-modal setting.

\RS{3}{
\textsc{NatGen}\xspace performs better than most of the existing baselines.
\textsc{NatGen}\xspace's improvement in Syntax match and Dataflow match signifies \textsc{NatGen}\xspace's ability to generate code syntactically and semantically closer to target code.}

\subsection{\textsc{NatGen}\xspace in limited-data setting.}

\RQrepeat{4}{How well does \tool's pre-training help in tasks where labelled data is scarce?}
\label{sec:rq4}

\paragraph{\underline{Motivation.}} Learning to generate code usually
requires a large amount of annotated training data.
A lot of time and effort goes into curating high-quality training data~\cite{ahmad2021avatar, lachaux2020unsupervised}. Unsupervised pre-training endows machine learning models with necessary domain knowledge about the task~\cite{erhan2010does}. In practice, this knowledge appears to transfer across multiple tasks. Such pre-training reduces the effort to learn each different task. We therefore study the effectiveness of \textsc{NatGen}\xspace's domain knowledge about source code syntax and semantics. In particular, we {\em stress test} whether the knowledge \textsc{NatGen}\xspace learned during pre-training is useful for downstream tasks, by limiting
available task-specific training data.

\paragraph{\underline{Experimental Setup.}} We evaluate \textsc{NatGen}\xspace's over
several data-limited tasks: {\em Text to Code generation}, {\em Code Translation}, and {\em Bug Fix}. We consider two different settings. First, we consider zero-shot~\cite{romera2015embarrassingly, xian2018zero} evaluation. Here we evaluate different pre-trained models {\em without} any task-specific training. Naturally, we don't see good performance in this setting. Nevertheless, this stress-test measures the code generation ability of models. Second, we try few-shot learning~\cite{wang2020generalizing, ravi2016optimization, sun2019meta}. We randomly choose a few training examples for each task and fine-tune the pre-trained models on those examples, and evaluate their performance. We gradually increase the number of training examples over several few-shot settings.

\paragraph{\underline{Results.}} \Cref{fig:zero-shot-boxplot} shows the \textsc{NatGen}\xspace's and CodeT5's zero-shot performance. Lacking task-specific training, we can see here how much transferable knowledge each model learned just during pre-training. There are large differences in all the tasks between \textsc{NatGen}\xspace and CodeT5 across Syntax Match and Dataflow Match. It signifies \textsc{NatGen}\xspace learns to generate both syntactically and semantically correct code during pre-training, which CodeT5 rarely can do.
\Cref{fig:few-shot-boxplot} shows the performance of \textsc{NatGen}\xspace and CodeT5 when trained on 200 training examples. \textsc{NatGen}\xspace also has an advantage over CodeT5 here.

We note a larger performance gap in the Translation tasks (\Cref{fig:java-cs-zero-shot} \& \ref{fig:cs-java-zero-shot}) and Bug Fix (\Cref{fig:refine-zero-shot}) tasks, compared to Text to Code Generation task (\Cref{fig:concode-zero-shot}) in both the zero-shot and the few shot (\Cref{fig:few-shot-boxplot}) experiments. We conjecture that such discrepancy is the artifact of the nature of the tasks. The cross-lingual alignment between NL and Java code is the key factor in generating text to code. In contrast, both the input and output are the programming language in the translation and bug fix task. Thus, we hypothesize that \textsc{NatGen}\xspace leverages its shared knowledge across different programming languages learned during the pre-training.

We further stress test \textsc{NatGen}\xspace's with few-shot learning; we gradually increased the number of training examples and trained both CodeT5 and \textsc{NatGen}\xspace. \Cref{fig:few-shot-progress} shows the performance progress as the number of training examples increase.
For all four tasks, \textsc{NatGen}\xspace significantly improves over CodeT5 when the number of training examples is minimal. With increasing training examples, the performance gap gradually decreases. Arguably, with enough {\em labeled} data and enough resources, all high-capacity models will get better at generating source code. Nevertheless, we learn two critical lessons from \textsc{NatGen}\xspace's better performance in zero-shot and few-shot learning. First, \textsc{NatGen}\xspace's better performance across all tasks suggests  that that the coding knowledge it learns from the naturalization task is more generic and transferable. Second, for any pre-trained model to be effective in code generation, especially in a limited training data scenario, the pre-training should explicitly teach the model how to write code. Otherwise, we hypothesize that a big chunk of fine-tuning resources will be spent on the models' learning to write code.

\RS{4}{
\textsc{NatGen}\xspace is very effective in source code generative tasks when minimal training resource is available. Since \textsc{NatGen}\xspace explicitly learns to generate code during pre-training, it can avoid learning such during fine-tuning saving fine-tuning resource.
}

\section{Introduction}
\label{sec:intro}
Statistical models that capture the ``naturalness" of code~\cite{hindle2012naturalness} have proven to be quite valuable in a variety of tasks~\cite{allamanis2018survey}, including  code repair, code summarization, code retrieval,
and code clone detection. The earlier work in this area used models directly trained  on tasks, including the early work on  type recovery~\cite{hellendoorn2017deep}, de-obfuscation~\cite{raychev2014code,vasilescu2017recovering}, code repair~\cite{gupta2017deepfix, chakraborty2020codit}, and code summarization~\cite{iyer2016summarizing, ahmad2020summarization}.
 Training on-task requires a large amount of labeled data, so that the enormous number of parameters that come with modern neural models can be deployed  and configured effectively for maximum performance.
 While labeled data is abundant for tasks like code completion (where the corpus inherently provides supervision), other tasks like summarization, code repair, etc. require careful curating to ensure that
 the data is of high quality. Simply grabbing data from Github might lead to poor quality~\cite{gros2020code}, and high levels of duplication~\cite{allamanis2019adverse}.
 As model capacity has increased (capacities in the range of tens of  billions of parameters are now routine) this unacceptable disparity between vast model capacity and the limited availability of well-curated, high-quality, labeled data has increased, and will likely  grow worse.

This shortage, of high-quality labeled data for on-task training is not a problem peculiar to software engineering (although it is more acute here, since the required manual labeling effort might demand a much higher level of specialized skill than with natural language tasks).
In NLP research, the strategy currently in vogue for dealing with this problem involve various offshoots of BERT~\cite{devlin2018bert}. The key insight here is that independent of final tasks, when \emph{very} high capacity models are trained, over enormous data sets, on simple, self-supervised, ``busy work'' tasks,  they  still \emph{learn representations that are useful on \underline{actual} tasks}. Furthermore, these cheaply-learned representations can then be used to train highly performant models \emph{even} with relatively small amounts of high-quality on-task training data. These two steps, known as ``pre-training" and ``fine-tuning" now yield SOTA performance on a wide range of tasks.

For Natural Language, the ``busy work'' pre-training tasks include such tasks as:  guessing masked words in context; guessing the next sentence in context; and de-noising a given, corrupted sentence. Using harder pre-training tasks appears to

  lead to better fine-tuned on-task performance. This is achieved in two different ways: first, using harder pre-training tasks forces models to 1) learn \emph{better internal representations of
  the given inputs} ({\hbox{\emph{e.g.,}}\xspace} being forced to fill in several blanks forces
  the model to learn better statistics of text patterns); 2) making the generation part of pre-training harder (\emph{e.g.,} T5 forces a complex generation task, and NXP in BERT forces next sentence generation) forces the models to become \emph{better at the generation of sequential text}.

 We propose a novel kind of ``busy work" task that places stronger demands during pre-training. Our idea is inspired by noting
  the kind of work that human Editors
  (of books, journals, newspapers) perform: they digest imperfectly written, but mostly correct text, understand the intent, and then produce more perfect text, with pretty-much the same meaning. Editing is \underline{\emph{hard}}: a skilled Editor has to have very high levels of language comprehension, to understand given, potentially badly-written text, \emph{and then} deploy very high-level writing
  skills to generate well-formed text.
  If Editing could be used as an at-scale pre-training task, the learned model would presumably have very good language comprehension, and also generate excellent text. But it's not at all obvious how to generate at-scale training data   for this ``Editing" task, say, for English.

  But our concern here is code, not natural language. We start with the argument that, because of
  bimodal, dual-channel nature of code~\cite{casalnuovo2020theory}, it is indeed possible to generate at-scale training data for the Editing task.
  Code has a formal channel, with well-defined semantics; because of this,
  it's possible to transform code into different forms, while
  \emph{preserving meaning}. Essentially, we can deploy a set of meaning preserving transformations to \emph{rewrite} existing code from widely-used
  github projects (which presumably have good-quality code that has passed human code review). These rewrites preserve meaning, but will make the code into an artificial, often unnatural form\footnote{Studies, with human-subjects~\cite{casalnuovo2020does,casalnuovo2020programmers} show that humans find such rewritten but semantically identical forms harder to read and
  understand.}. We now have a matched pair, of two semantically forms of code: an ``unnatural" form, and the original form; furthermore we can produce these pairs, at scale. Using large numbers of such pairs, we can pre-train on a challenging code ``Naturalization" task, which then, by analogy with human Editors as described above, forces the pre-trained model to learn to do two hard things: 1) first, capture the meaning of the input code, and 2) generate an output that more closely resembles the code that human coders write. Our hypothesis that the resulting model will both learn better meaning representations, \emph{and also} generate better code.

  We can now summarize our main contributions

  \begin{enumerate}
      \item We introduce the idea of "Code naturalization" as a pre-training task.
     \item Using code from Github, and custom tooling, we have generated (and will release) a large dataset for pre-training models on the Naturalization task.
     \item We have built (and will release) a large CodeT5~\cite{wang2021codet5} model pre-trained on Naturalization.
     \item We show that (when appropriately fine-tuned) this \textsc{NatGen}\xspace outperforms SOTA on several tasks.
  \end{enumerate}

We publish our source code and data download sript for pretraining \textsc{NatGen}\xspace anonymously in  \url{https://github.com/natgen-team/NatGen}. We also share all the trained models in \url{https://drive.google.com}.

\medskip

\section{0pt}{8pt plus 4pt minus 2pt}{4pt plus 2pt minus 2pt}

\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%

\newcommand{\rom}[1]{\uppercase\expandafter{\romannumeral #1\relax}}

\newcommand{\hbox{\emph{cf.}}\xspace}{\hbox{\emph{cf.}}\xspace}
\newcommand{\ldots [deletia] \ldots}{\ldots [deletia] \ldots}
\newcommand{\hbox{\emph{et al.}}\xspace}{\hbox{\emph{et al.}}\xspace}
\newcommand{\hbox{\emph{e.g.,}}\xspace}{\hbox{\emph{e.g.,}}\xspace}
\newcommand{\hbox{\emph{i.e.,}}\xspace}{\hbox{\emph{i.e.,}}\xspace}
\newcommand{\hbox{\emph{s.t.,}}\xspace}{\hbox{\emph{s.t.,}}\xspace}
\newcommand{\hbox{\emph{w.r.t.}}\xspace}{\hbox{\emph{w.r.t.}}\xspace}
\newcommand{\hbox{\emph{viz.}}\xspace}{\hbox{\emph{viz.}}\xspace}
\newcommand{\hbox{\emph{v.s.}}\xspace}{\hbox{\emph{v.s.}}\xspace}
\newcommand{\hbox{\emph{etc.}}\xspace}{\hbox{\emph{etc.}}\xspace}

\newtheorem{theorem}{Theorem}[section]
\newcommand{\defref}[1]{Definition~\ref{#1}}
\newtheorem{definition}[theorem]{Definition}

\newcommand{\varname}[1]{{\small\texttt{#1}}\xspace}
\newcommand{\ch}[1]{{\color{black} #1}\xspace}

\def\superscript{\S}{\superscript{\S}}
\def$^{\dag}${$^{\dag}$}
\def\superscript{\ddag}{\superscript{\ddag}}
\def$^{\natural}${$^{\natural}$}

\newlength\Linewidth
\def\findlength{\setlength\Linewidth\linewidth
\addtolength\Linewidth{-4\fboxrule}
\addtolength\Linewidth{-3\fboxsep}
}
\newenvironment{examplebox}{\par\begingroup
  \setlength{\fboxsep}{3pt}\findlength
  \setbox0=\vbox\bgroup\noindent
  \hsize=0.90\linewidth
  \begin{minipage}{0.90\linewidth}\normalsize}
    {\end{minipage}\egroup
    \textcolor{gray20}{\fboxsep1.2pt\fbox
     {\fboxsep5pt\colorbox{gray05}{\normalcolor\box0}}}
    \endgroup\par\noindent
    \normalcolor\ignorespacesafterend}
\let\Examplebox\examplebox
\let\endExamplebox\endexamplebox

\newcounter{RQCounter}
\newcounter{RQACounter}

\usetikzlibrary{shadows}
\usepackage{graphics}
\newmdenv[
    tikzsetting= {fill=blueish},
    skipabove=0.33em,
    skipbelow=0.33em,
    linewidth=1pt,
    innerleftmargin=4pt,
    innerrightmargin=4pt,
    innertopmargin=2pt,
    innerbottommargin=2pt,
    linecolor=gray95,
    roundcorner=2pt,
    shadow=true,
    shadowsize=4pt,
    shadowcolor=gray95
]{questionbox}

\newmdenv[
    tikzsetting= {fill=greenish},
    skipabove=0.33em,
    skipbelow=0.33em,
    linewidth=1pt,
    innerleftmargin=4pt,
    innerrightmargin=4pt,
    innertopmargin=2pt,
    innerbottommargin=2pt,
    linecolor=gray95,
    roundcorner=2pt,
    shadow=true,
    shadowsize=4pt,
    shadowcolor=gray95
]{answerbox}

\newmdenv[
    skipabove=0.33em,
    skipbelow=0.33em,
    innerleftmargin=4pt,
    innerrightmargin=4pt,
    innertopmargin=2pt,
    innerbottommargin=2pt,
]{lessonbox}

\usepackage{tikz}
\newcommand*\circled[1]{%
    \tikz[baseline=(char.base)]{%
        \node[shape=circle,draw,innersep=5pt] (char) {#1};%
    }%
}%

\renewcommand\tabularxcolumn[1]{m{#1}}

\newenvironment{lesson}
{
    \begin{lessonbox}
}
{\end{lessonbox}}

\newenvironment{result}
{\begin{answerbox}}
{\end{answerbox}}

\newenvironment{question}
{\begin{questionbox}}
{\end{questionbox}}

\definecolor{javared}{rgb}{0.6,0,0}
\definecolor{javagreen}{rgb}{0.25,0.5,0.35}
\definecolor{javapurple}{rgb}{0.5,0,0.35}
\definecolor{javadocblue}{rgb}{0.25,0.35,0.75}

\lstdefinestyle{basejava}{
  language=java,
  showstringspaces=false,
  basicstyle=\small\ttfamily,
  keywordstyle=\bfseries\color{javapurple},
  commentstyle=\itshape\blue,
  identifierstyle=\blue,
  frame=none,
  backgroundcolor=\color{white},
}

\lstdefinestyle{CustomJava}{
  belowcaptionskip=\baselineskip,
  breaklines=true,
  xleftmargin=\parindent,
  language=java,
  showstringspaces=false,
  basicstyle=\scriptsize\ttfamily,
  keywordstyle=\bfseries\color{javapurple},
  commentstyle=\itshape\blue,
  identifierstyle=\blue,
  belowskip=1pt,
  numbers=left,
  gobble=0
}

\lstdefinestyle{CustomJavaWoNumbers}{
  belowcaptionskip=0.5\baselineskip,
  breaklines=true,
  xleftmargin=\parindent,
  language=java,
  showstringspaces=false,
  basicstyle=\scriptsize\ttfamily,
  keywordstyle=\bfseries\color{javapurple},
  commentstyle=\itshape\blue,
  identifierstyle=\blue,
  belowskip=0.5pt,
  numbers=none,
  gobble=0
}

\lstdefinestyle{codit}{
  belowcaptionskip=\baselineskip,
  breaklines=true,

  language=java,
  showstringspaces=false,
  basicstyle=\scriptsize\ttfamily,
  keywordstyle=\bfseries\color{javapurple},
  commentstyle=\itshape\blue,
  identifierstyle=\blue,
}

\lstset{escapechar=@,style=customjava}
\lstset{escapechar=@,style=codit}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}