\section{Introduction}
Single-document summarization methods can be divided into two categories: extractive and abstractive. Extractive summarization systems form summaries by selecting and copying text snippets from the document, while abstractive methods aim to generate concise summaries with paraphrasing. This work is primarily concerned with extractive summarization. Though abstractive summarization methods have made strides in recent years, extractive techniques are still very attractive as they are simpler, faster, and more reliably yield semantically and grammatically correct sentences.

Many extractive summarizers work by selecting sentences from the input document \citep{tra_ext1_luhn1958automatic,tra_ext3_mihalcea2004textrank,tra_ext5_wong2008extractive,ext1_kaageback2014extractive,ext2_2015Yin,ext3_cao2015learning,yasunaga2017graph}. Furthermore, a growing trend is to frame this sentence selection process as a sequential binary labeling problem, where binary inclusion/exclusion labels are chosen for sentences one at a time, starting from the beginning of the document, and decisions about later sentences may be conditioned on decisions about earlier sentences. Recurrent neural networks may be trained with stochastic gradient ascent to maximize the likelihood of a set of ground-truth binary label sequences \citep{ext4_cheng2016neural,ext5_summarunner}.
However, this approach has two well-recognized disadvantages. First, it suffers from exposure bias, a form of mismatch between training and testing data distributions which can hurt performance \citep{Ranzato2015SequenceLT, rl1_bahdanau+al-2017-actorcritic-iclr, abs5_paulus2017deep}. Second, extractive labels must be generated by a heuristic, as summarization datasets do not generally include ground-truth extractive labels; the ultimate performance of models trained on such labels is thus fundamentally limited by the quality of the heuristic.

An alternative to maximum likelihood training is to use reinforcement learning to train the model to directly maximize a measure of summary quality, such as the ROUGE score between the generated summary and a ground-truth abstractive summary \citep{DBLP:conf/aaai/WuH18}. This approach has become popular because it avoids exposure bias, and directly optimizes a measure of summary quality. However, it also has a number of downsides. For one, the search space is quite large: for a document of length $T$, there are $2^T$ possible extractive summaries. This makes the exploration problem faced by the reinforcement learning algorithm during training very difficult. Another issue is that due to the sequential nature of selection, the model is inherently biased in favor of selecting earlier sentences over later ones, a phenomenon which we demonstrate empirically in Section \ref{sec:discussion}. The first issue can be resolved to a degree using either a cumbersome maximum likelihood-based pre-training step (using heuristically-generated labels) \citep{DBLP:conf/aaai/WuH18}, or placing a hard upper limit on the number of sentences selected. The second issue is more problematic, as it is inherent to the sequential binary labeling setting.

In the current work, we introduce \textsc{BanditSum}, a novel method for training neural network-based extractive summarizers with reinforcement learning. This method does away with the sequential binary labeling setting, instead formulating extractive summarization as a contextual bandit. This move greatly reduces the size of the space that must be explored, removes the need to perform supervised pre-training, and prevents systematically privileging earlier sentences over later ones. Although the strong performance of Lead-3 indicates that good sentences often occur early in the source article, we show in Sections \ref{sec:results} and \ref{sec:discussion} that the contextual bandit setting greatly improves model performance when good sentences occur late without sacrificing performance when good sentences occur early.

Under this reformulation, \textsc{BanditSum } takes the document as input and outputs an \textit{affinity} for each of the sentences therein. An affinity is a real number in $[0, 1]$ which quantifies the model's propensity for including a sentence in the summary. These affinities are then used in a process of repeated sampling-without-replacement which does not privilege earlier sentences over later ones. \textsc{BanditSum } is free to process the document as a whole before yielding affinities, which permits affinities for different sentences in the document to depend on one another in arbitrary ways. In our technical section, we show how to apply policy gradient reinforcement learning methods to this setting.

The contributions of our work are as follows:
\begin{itemize}
\item We propose a theoretically grounded method, based on the contextual bandit formalism, for training neural network-based extractive summarizers with reinforcement learning. Based on this training method, we propose the \textsc{BanditSum } system for extractive summarization.

\item We perform experiments demonstrating that \textsc{BanditSum } obtains state-of-the-art performance on a number of datasets and requires significantly fewer update steps than competing approaches.

\item We perform human evaluations showing that in the eyes of human judges, summaries created by \textsc{BanditSum } are less redundant and of higher overall quality than summaries created by competing approaches.

\item We provide evidence, in the form of experiments in which models are trained on subsets of the data, that the improved performance of \textsc{BanditSum } over competitors stems in part from better handling of summary-worthy sentences that come near the end of the document (see Section~\ref{sec:discussion}).
\end{itemize}

\section{Related Work\label{sec:related-work}}
Extractive summarization has been widely studied in the past. Recently, neural network-based methods have been gaining popularity over classical methods \citep{tra_ext1_luhn1958automatic,tra_ext2_gong2001generic, tra_ext4_conroy2001text,tra_ext3_mihalcea2004textrank,tra_ext5_wong2008extractive}, as they have demonstrated stronger performance on large corpora. Central to the neural network-based models is the encoder-decoder structure. These models typically use either a convolution neural network \cite{cnn1_kalchbrenner2014convolutional,cnn2_kim2014convolutional,ext2_2015Yin,ext3_cao2015learning}, a recurrent neural network \cite{rnn2_chung2014gru,ext4_cheng2016neural,ext5_summarunner}, or a combination of the two \cite{DBLP:Narayan/2018,DBLP:conf/aaai/WuH18} to create sentence and document representations, using word embeddings \cite{we1_mikolov2013efficient,we2_pennington2014glove} to represent words at the input level. These vectors are then fed into a decoder network to generate the output summary.

The use of reinforcement learning (RL) in extractive summarization was first explored by \citet{rl_ryang2012frameworkRL}, who proposed to use the TD($\lambda$) algorithm to learn a value function for sentence selection. \citet{rl_rioux2014fear} improved this framework by replacing the learning agent with another TD($\lambda$) algorithm. However, the performance of their methods was limited by the use of shallow function approximators, which required performing a fresh round of reinforcement learning for every new document to be summarized. The more recent work of \citet{abs5_paulus2017deep} and \citet{DBLP:conf/aaai/WuH18} use reinforcement learning in a sequential labeling setting to train abstractive and extractive summarizers, respectively, while \citet{chen2018abstractive} combines both approaches, applying abstractive summarization to a set of sentences extracted by a pointer network \citep{vinyals2015pointer} trained via REINFORCE.
However, pre-training with a maximum likelihood objective is required in all of these models.

The two works most similar to ours are \citet{yao2018deep} and \citet{DBLP:Narayan/2018}. \citet{yao2018deep} recently proposed an extractive summarization approach based on deep Q learning, a type of reinforcement learning. However, their approach is extremely computationally intensive (a minimum of 10 days before convergence), and was unable to achieve ROUGE scores better than the best maximum likelihood-based approach. \citet{DBLP:Narayan/2018} uses a cascade of filters in order to arrive at a set of candidate extractive summaries, which we can regard as an approximation of the true action space. They then use an approximation of a policy gradient method to train their neural network to select summaries from this approximated action space. In contrast, \textsc{BanditSum } samples directly from the true action space, and uses exact policy gradient parameter updates.

\section{Extractive Summarization as a Contextual Bandit \label{sec:technical}}
Our approach formulates extractive summarization as a contextual bandit which we then train an agent to solve using policy gradient reinforcement learning. A bandit is a decision-making formalization in which an agent repeatedly chooses one of several actions, and receives a reward based on this choice. The agent's goal is to quickly learn which action yields the most favorable distribution over rewards, and choose that action as often as possible. In a \textit{contextual} bandit, at each trial, a context is sampled and shown to the agent, after which the agent selects an action and receives a reward; importantly, the rewards yielded by the actions may depend on the sampled context. The agent must quickly learn which actions are favorable in which contexts. Contextual bandits are a subset of Markov Decision Processes in which every episode has length one.

Extractive summarization may be regarded as a contextual bandit as follows. Each document is a context, and each ordered subset of a document's sentences is a different action. Formally, assume that each context is a document $d$ consisting of sentences $s = (s_1, \dots, s_{N_d})$, and that each action is a length-$M$ sequence of unique sentence indices $i = (i_1, \dots, i_M)$ where $i_t \in \{1, \dots, N_d\}$, $i_t \neq i_{t'}$ for $t \neq t'$, and $M$ is an integer hyper-parameter. For each $i$, the extractive summary induced by $i$ is given by $(s_{i_1}, \dots, s_{i_M})$. An action $i$ taken in context $d$ is given a reward $R(i, a)$, where $a$ is the gold-standard abstractive summary that is paired with document $d$, and $R$ is a scalar reward function quantifying the degree of match between $a$ and the summary induced by $i$.

A policy for extractive summarization is a neural network $p_\theta(\cdot | d)$, parameterized by a vector $\theta$, which, for each input document $d$, yields a probability distribution over index sequences. Our goal is to find parameters $\theta$ which cause $p_\theta(\cdot|d)$ to assign high probability to index sequences that induce extractive summaries that a human reader would judge to be of high-quality. We achieve this by maximizing the following objective function with respect to parameters $\theta$:
\begin{align}
    J(\theta) = E\left[R(i, a)\right]\label{eqn:objective}
\end{align}
where the expectation is taken over documents $d$ paired with gold-standard abstractive summaries $a$, as well as over index sequences $i$ generated according to $p_\theta(\cdot | d)$.

\subsection{Policy Gradient Reinforcement Learning}
Ideally, we would like to maximize \eqref{eqn:objective} using gradient ascent. However, the required gradient cannot be obtained using usual techniques (e.g. simple backpropagation) because $i$ must be discretely sampled in order to compute $R(i, a)$.

Fortunately, we can use the likelihood ratio gradient estimator from reinforcement learning and stochastic optimization \cite{williams1992simple, sutton2000policy}, which tells us that the gradient of this function can be computed as:
\begin{align}
    \nabla_\theta J(\theta) = E\left[ \nabla_\theta \log p_\theta(i | d) R(i, a) \right] \label{eqn:gradient}
\end{align}
where the expectation is taken over the same variables as \eqref{eqn:objective}.

Since we typically do not know the exact document distribution and thus cannot evaluate the expected value in \eqref{eqn:gradient}, we instead estimate it by sampling. We found that we obtained the best performance when, for each update, we first sample one document/summary pair $(d, a)$, then sample $B$ index sequences $i^1, \dots, i^B$ from $p_\theta(\cdot | d)$, and finally take the empirical average:
\begin{align}
    \nabla_\theta J(\theta) \approx \frac{1}{B} \sum_{b=1}^{B}\nabla_\theta \log p_\theta(i^b | d) R(i^b, a) \label{eqn:estimated-gradient}
\end{align}
This overall learning algorithm can be regarded as an instance of the REINFORCE policy gradient algorithm \citep{williams1992simple}.

\subsection{Structure of $p_\theta(\cdot|d)$}
There are many possible choices for the structure of $p_\theta(\cdot | d)$; we opt for one that avoids privileging early sentences over later ones. We first decompose $p_\theta(\cdot | d)$ into two parts: $\pi_\theta$, a deterministic function which contains all the network's parameters, and $\mu$, a probability distribution parameterized by the output of $\pi_\theta$. Concretely:
\begin{align}
    p_\theta(\cdot | d) = \mu(\cdot | \pi_\theta(d))
\end{align}
Given an input document $d$, $\pi_\theta$ outputs a real-valued vector of \textit{sentence affinities} whose length is equal to the number of sentences in the document (i.e. $\pi_\theta(d) \in \mathbb{R}^{N_d}$) and whose elements fall in the range $[0, 1]$. The $t$-th entry $\pi(d)_t$ may be roughly interpreted as the network's propensity to include sentence $s_t$ in the summary of $d$.

Given sentence affinities $\pi_\theta(d)$, $\mu$ implements a process of repeated sampling-without-replacement. This proceeds by repeatedly normalizing the set of affinities corresponding to sentences that have not yet been selected, thereby obtaining a probability distribution over unselected sentences, and sampling from that distribution to obtain a new sentence to include. This normalize-and-sample step is repeated $M$ times, yielding $M$ unique sentences to include in the summary.

At each step of sampling-without-replacement, we also include a small probability $\epsilon$ of sampling uniformly from all remaining sentences. This is used to achieve adequate exploration during training, and is similar to the $\epsilon$-greedy technique from reinforcement learning.

Under this sampling scheme, we have the following expression for $p_\theta(i|d)$:
\begin{align}
\prod_{j=1}^M \left( \frac{\epsilon}{N_d - j + 1} + \frac{(1-\epsilon)\pi(d)_{i_j}}{z(d) - \sum_{k=1}^{j-1} \pi(d)_{i_k}} \right)
\end{align}
where $z(d) = \sum_t\pi(d)_t$. For index sequences that have length different from $M$, or that contain duplicate indices, we have $p_\theta(i|d) = 0$. Using this expression, it is straightforward to use automatic differentiation software to compute $\nabla_\theta \log p_\theta(i|d)$, which is required for the gradient estimate in \eqref{eqn:estimated-gradient}.

\subsection{Baseline for Variance Reduction} \label{subsec:baseline}
Our sample-based gradient estimate can have high variance, which can slow the learning. One potential cause of this high variance can be seen by inspecting \eqref{eqn:estimated-gradient}, and noting that it basically acts to change the probability of a sampled index sequence to an extent determined by the reward $R(i, a)$. However, since ROUGE scores are always positive, the probability of every sampled index sequence is increased, whereas intuitively, we would prefer to decrease the probability of sequences that receive a comparatively low reward, even if it is positive. This can be remedied by the introduction of a so-called baseline which is subtracted from all rewards.

Using a baseline $\overline{r}$, our sample-based estimate of $\nabla_\theta J(\theta)$ becomes:
\begin{align}
    \frac{1}{B} \sum_{i=1}^{B}\nabla_\theta \log p_\theta(i^b | d) (R(i^b, a) - \overline{r})\label{eqn:estimated-gradient-baseline}
\end{align}
It can be shown that the introduction of $\overline{r}$ does not bias the gradient estimator and can significantly reduce its variance if chosen appropriately \citep{sutton2000policy}.

There are several possibilities for the baseline, including the long-term average reward and the average reward across different samples for one document-summary pair. We choose an approach known as self-critical reinforcement learning, in which the test-time performance of the current model is used as the baseline \citep{Ranzato2015SequenceLT,rennie2017self,abs5_paulus2017deep}. More concretely, after sampling the document-summary pair $(d, a)$, we greedily generate an index sequence using the current parameters $\theta$:
\begin{align}
    i_\textit{greedy} = \argmax_i p_\theta(i | d)
\end{align}
and calculate the baseline for the current update as $\overline{r} = R(i_\textit{greedy}, a)$. This baseline has the intuitively satisfying property of only increasing the probability of a sampled label sequence when the summary it induces is better than what would be obtained by greedy decoding.

\subsection{Reward Function}
A final consideration is a concrete choice for the reward function $R(i, a)$. Throughout this work we use:
\begin{multline}
R(i, a) = \frac{1}{3} (\text{ROUGE-1}_f(i, a) + {} \\
                     \text{ROUGE-2}_f(i, a) +
                     \text{ROUGE-L}_f(i, a)).
\end{multline}
The above reward function optimizes the average of all the ROUGE variants \cite{eva1_lin:2004:ACLsummarization} while balancing precision and recall.

\section{Model\label{sec:model}}

In this section, we discuss the concrete instantiations of the neural network $\pi_\theta$ that we use in our experiments. We break $\pi_\theta$ up into two components: a document encoder $f_{\theta1}$, which outputs a sequence of sentence feature vectors $(h_1, \dots, h_{N_d})$ and a decoder $g_{\theta2}$ which yields sentence affinities:
\begin{align}
    h_1, \dots, h_{N_d} &= f_{\theta1}(d)\\
    \pi_\theta(d) &= g_{\theta2}(h_1, \dots, h_{N_d})
\end{align}

\noindent \textbf{Encoder.} Features for each sentence in isolation are first obtained by applying a word-level Bidirectional Recurrent Neural Network (BiRNN) to the embeddings for the words in the sentence, and averaging the hidden states over words. A separate sentence-level BiRNN is then used to obtain a representations $h_i$ for each sentence in the context of the document.

\noindent \textbf{Decoder.} A multi-layer perceptron is used to map from the representation $h_t$ of each sentence through a final sigmoid unit to yield sentence affinities $\pi_\theta(d)$.

The use of a bidirectional recurrent network in the encoder is crucial, as it allows the network to process the document as a whole, yielding representations for each sentence that take all other sentences into account. This procedure is necessary to deal with some aspects of summary quality such as redundancy (avoiding the inclusion of multiple sentences with similar meaning), which requires the affinities for different sentences to depend on one another. For example, to avoid redundancy, if the affinity for some sentence is high, then sentences which express similar meaning should have low affinities.

\section{Experiments\label{sec:experiments}}
In this section, we discuss the setup of our experiments. We first discuss the corpora that we used and our evaluation methodology. We then discuss the baseline methods against which we compared, and conclude with a detailed overview of the settings of  the model parameters.
\subsection{Corpora}
Three datasets are used for our experiments: the CNN, the Daily Mail, and combined CNN/Daily Mail \citep{data1_hermann2015teaching,data2_nallapati2016abstractive}. We use the standard split of \citet{data1_hermann2015teaching} for training, validating, and testing and the same setting without \textit{anonymization} on the three corpus as \citet{abs4_SeeLM17}. The Daily Mail corpus has 196,557 training documents, 12,147 validation documents and 10,397 test documents; while the CNN corpus has 90,266/1,220/1,093 documents, respectively.

\subsection{Evaluation}
The models are evaluated based on ROUGE \cite{eva1_lin:2004:ACLsummarization}.
We obtain our ROUGE scores using the standard pyrouge
package\footnote{\url{https://pypi.python.org/pypi/pyrouge/0.1.3}} for the test set evaluation and a faster python implementation of the ROUGE metric\footnote{We use the modified version based on \url{https://github.com/pltrdy/rouge}}
for training and evaluating on the validation set. We report the F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L, which compute the uniform, bigram, and longest common subsequence overlapping with the reference summaries.

\subsection{Baselines}
We compare \textsc{BanditSum } with other extractive methods including: the Lead-3 model, SummaRuNNer \citep{ext5_summarunner}, Refresh \citep{DBLP:Narayan/2018}, RNES \citep{DBLP:conf/aaai/WuH18}, DQN \citep{yao2018deep}, and NN-SE \citep{ext4_cheng2016neural}.  The Lead-3 model simply produces the leading three sentences of the document as the summary.

\subsection{Model Settings}
We use 100-dimensional Glove embeddings \citep{we2_pennington2014glove} as our embedding initialization. We do not limit the sentence length, nor the maximum number of sentences per document. We use one-layer BiLSTM for word-level RNN, and two-layers BiLSTM for sentence-level RNN. The hidden state dimension is 200 for each direction on all LSTMs. For the decoder, we use a feed-forward network with one hidden layer of  dimension 100.

During training, we use Adam \citep{adam_kingma2014adam} as the optimizer with the learning rate of $5e^{-5}$, beta parameters $(0, 0.999)$, and a weight decay of $1e^{-6}$, to maximize the objective function defined in equation \eqref{eqn:objective}. We employ gradient clipping of 1 to regularize our model. At each iteration, we sample $B = 20$ times to estimate the gradient defined in equation \ref{eqn:estimated-gradient}. For our system, the reported performance is obtained within two epochs of training.

At the test time, we pick sentences sorted by the predicted probabilities until the length limit is reached. The full-length ROUGE F1 score is used as the evaluation metric. For $M$, the number of sentences selected per summary, we use a value of 3, based on our validation results as well as on the settings described in \citet{ext5_summarunner}.

\section{Experiment Results\label{sec:results}}
In this section, we present  quantitative results from the ROUGE evaluation and qualitative results based on human evaluation. In addition, we demonstrate the stability of our RL model by comparing the validation curve of \textsc{BanditSum } with SummaRuNNer \citep{ext5_summarunner} trained with a maximum likelihood objective.
\subsection{Rouge Evaluation}

\begin{table}[!h]
\footnotesize
\centering
\begin{tabular}{|l|l|l|l|}
\hline
Model & \multicolumn{3}{l|}{ROUGE} \\
\hline
 & 1 & 2 & L \\
 \hline
Lead\citep{DBLP:Narayan/2018} & 39.6 & 17.7 & 36.2 \\
Lead-3(ours) & 40.0 & 17.5 & 36.2 \\
SummaRuNNer & 39.6 & 16.2 & 35.3  \\
DQN & 39.4	& 16.1 &	35.6 \\
Refresh & 40.0 &18.2 &36.6\\
RNES w/o coherence & 41.3 & \textbf{18.9} & \textbf{37.6}\\
\hline
\textsc{BanditSum }  & \textbf{41.5} & 18.7 & \textbf{37.6} \\
\hline
\end{tabular}
\caption{Performance comparison of different extractive summarization models on the combined CNN/Daily Mail test set using full-length F1. }
\label{table:results_cnn}
\end{table}

\begin{table}[!h]
\small
\centering
\setlength\tabcolsep{4 pt}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
Model & \multicolumn{3}{l|}{CNN} & \multicolumn{3}{l|}{Daily Mail} \\ \hline
 & 1 & 2 & L & 1 & 2 & L   \\ \hline
Lead-3 & 28.8 & 11.0 & 25.5 & 41.2 & 18.2 & 37.3 \\
NN-SE & 28.4 & 10.0 & 25.0 & 36.2 & 15.2 & 32.9 \\
Refresh & 30.4 & \textbf{11.7} & 26.9& 41.0 & 18.8 & 37.7 \\
\textsc{BanditSum } & \textbf{30.7} & 11.6 & \textbf{27.4} & \textbf{42.1} & \textbf{18.9} &  \textbf{38.3}\\
\hline
\end{tabular}
\caption{The full-length ROUGE F1 scores of various extractive models on the CNN and the Daily Mail test set separately.}
\label{table:daily_mail}
\end{table}

We present the results of comparing \textsc{BanditSum } to several baseline algorithms\footnote{
Due to different pre-processing methods and different numbers of selected sentences, several papers report different Lead scores \citep{DBLP:Narayan/2018,abs4_SeeLM17}.
We use the test set provided by \citet{DBLP:Narayan/2018}. Since their Lead score is a combination of Lead-3 for CNN and Lead-4 for Daily Mail, we recompute the Lead-3 scores for both CNN and Daily Mail with the preprocessing steps used in \citet{abs4_SeeLM17}. Additionally, our results are not directly comparable to results based on the anonymized dataset used by \citet{ext5_summarunner}.}
on the CNN/Daily Mail corpus in Tables~\ref{table:results_cnn} and \ref{table:daily_mail}.
Compared to other extractive summarization systems, \textsc{BanditSum } achieves performance that is significantly better than two RL-based approaches, Refresh \citep{DBLP:Narayan/2018} and DQN \citep{yao2018deep}, as well as SummaRuNNer, the state-of-the-art maximum liklihood-based extractive summarizer \cite{ext5_summarunner}. \textsc{BanditSum } performs a little better than RNES \citep{DBLP:conf/aaai/WuH18} in terms of ROUGE-1 and slightly worse in terms of ROUGE-2. However, RNES requires pre-training with the maximum likelihood objective on heuristically-generated extractive labels; in contrast, \textsc{BanditSum } is very light-weight and converges significantly faster. We discuss the advantage of framing the extractive summarization based on the contextual bandit (\textsc{BanditSum}) over the sequential binary labeling setting (RNES) in the discussion Section~\ref{sec:discussion}.

We also noticed that different choices for the policy gradient baseline (see Section~\ref{subsec:baseline}) in \textsc{BanditSum } affect learning speed, but do not significantly affect asymptotic performance. Models trained with an average reward baseline learned most quickly, while models trained with three different baselines (greedy, average reward in a batch, average global reward) all perform roughly the same after training for one epoch. Models trained without a baseline were found to under-perform other baseline choices by about 2 points of ROUGE score on average.

\subsection{Human Evaluation}
We also conduct a qualitative evaluation to understand the effects of the improvements introduced in \textsc{BanditSum } on human judgments of the generated summaries. To assess the effect of training with RL rather than maximum likelihood, in the first set of human evaluations we compare \textsc{BanditSum } with the state-of-the-art maximum likelihood-based model SummaRuNNer. To evaluate the importance of using an exact, rather than approximate, policy gradient to optimize ROUGE scores, we perform another human evaluation comparing \textsc{BanditSum } and Refresh, an RL-based method that uses the an approximation of the policy gradient.

We follow a human evaluation protocol similar to the one used in \citet{DBLP:conf/aaai/WuH18}. Given a set of $N$ documents, we ask $K$ volunteers to evaluate the summaries extracted by both systems. For each document, a reference summary, and a pair of randomly ordered extractive summaries (one generated by each of the two models) is presented to the volunteers. They are asked to compare and rank the extracted summaries along three dimensions: overall, coverage, and non-redundancy.

\begin{table}[h!]
\small
\begin{center}
\begin{tabularx}{\columnwidth}{|l|X|X|X|}
\hline
Model & Overall & Coverage & Non-Redundancy \\ \hline
SummaRuNNer & 1.67 &\textbf{1.46}  &1.70  \\ \hline
\textsc{BanditSum } & \textbf{1.33} & 1.54 & \textbf{1.30} \\ \hline
\end{tabularx}
\end{center}
\caption{Average rank of human evaluation based on 5 participants who expressed 57 pairwise preferences between the summaries generated by SummaRuNNer and \textsc{BanditSum}. The model with the lower score is better.}
\label{table:human_eval_summarunner}
\end{table}

\begin{table}[h!]
\small
\begin{center}
\begin{tabularx}{\columnwidth}{|l|X|X|X|}
\hline
Model & Overall & Coverage & Non-Redundancy \\ \hline
Refresh & 1.53 &\textbf{1.34}  & 1.55  \\ \hline
\textsc{BanditSum } & \textbf{1.50} & 1.58 & \textbf{1.30} \\ \hline
\end{tabularx}
\end{center}
\caption{Average rank of manual evaluation with 4 participants who expressed 20 pairwise preferences between the summaries generated by Refresh and our system. The model with the lower score is better.}
\label{table:human_eval_refresh}
\end{table}

To compare with SummaRuNNer, we randomly sample 57 documents from the test set of DailyMail and ask 5 volunteers to evaluate the extracted summaries.
While comparing with Refresh, we use the 20 documents (10 CNN and 10 DailyMail) provided by \citet{DBLP:Narayan/2018} to 4 volunteers. Tables \ref{table:human_eval_summarunner} and \ref{table:human_eval_refresh} show the results of human evaluation in these two settings. \textsc{BanditSum } is shown to be better than Refresh and SummaRuNNer in terms of overall quality and non-redundancy. These results indicate that the use of the true policy gradient, rather than the approximation used by Refresh, improves overall quality. It is interesting to observe that, even though \textsc{BanditSum } does not have an explicit redundancy avoidance mechanism, it actually outperforms the other systems on non-redundancy.

\subsection{Learning Curve}
Reinforcement learning methods are known for sometimes being unstable during training. However, this seems to be less of a problem for \textsc{BanditSum}, perhaps because it is formulated as a contextual bandit rather than a sequential labeling problem. We show this by comparing the validation curves generated by \textsc{BanditSum } and the state-of-the-art maximum likelihood-based model -- SummaRuNNer \citep{ext5_summarunner} (Figure~\ref{fig:train_efficiency}).

\begin{figure}[!h]
  \includegraphics[width=0.5\textwidth]{figures/dm_nll_rl.png}
  \caption[Caption for LOF]{Average of ROUGE-1,2,L F1 scores on the Daily Mail validation set within one epoch of training on the Daily Mail training set. The x-axis (multiply by 2,000) indicates the number of data example the algorithms have seen. The supervised labels in SummaRuNNer are used to estimate the \texttt{upper\_bound}.}
  \label{fig:train_efficiency}
\end{figure}

From Figure~\ref{fig:train_efficiency}, we observe that \textsc{BanditSum } converges significantly more quickly to good results than SummaRuNNer. Moreover, there is less variance in the performance of \textsc{BanditSum}. One possible reason is that extractive summarization does not have well-defined supervised labels. There exists a mismatch between the provided labels and human-generated abstractive summaries. Hence, the gradient, computed from the maximum likelihood loss function, is not optimizing the evaluation metric of interest. Another important message is that both models are still far from the estimated upper bound\footnote{The supervised labels for the \texttt{upper\_bound} estimation are obtained using the heuristic described in \citet{ext5_summarunner}.}, which shows that there is still significant room for improvement.

\subsection{Run Time}
On CNN/Daily mail dataset, our model's time-per-epoch is about 25.5 hours on a TITAN Xp. We trained the model for 3 epochs, which took about 76 hours in total. For comparison, DQN took about 10 days to train on a GTX 1080 \citep{yao2018deep}. Refresh took about 12 hours on a single GPU to train \citep{DBLP:Narayan/2018}. Note that this figure does not take into account the significant time required by Refresh for pre-computing ROUGE scores.

\section{Discussion: Contextual Bandit Setting Vs. Sequential Full RL Labeling \label{sec:discussion}}

We conjecture that the contextual bandit (CB) setting is a more suitable framework for modeling extractive summarization than the sequential binary labeling setting, especially in the cases when good summary sentences appear later in the document. The intuition behind this is that models based on the sequential labeling setting are affected by the order of the decisions, which biases towards selecting sentences that appear earlier in the document. By contrast, our CB-based RL model has more flexibility and freedom to explore the search space, as it samples the sentences without replacement based on the affinity scores. Note that although we do not explicitly make the selection decisions in a sequential fashion, the sequential information about dependencies between sentences is implicitly embedded in the affinity scores, which are produced by bi-directional RNNs.

\newcommand{\mathit{D}_{\textit{early}}}{\mathit{D}_{\textit{early}}}
\newcommand{\mathit{D}_{\textit{late}}}{\mathit{D}_{\textit{late}}}

We provide empirical evidence for this conjecture by comparing \textsc{BanditSum } to the sequential RL model proposed by~\citet{DBLP:conf/aaai/WuH18} (Figure \ref{fig:model_comparison}) on two subsets of the data: one with good summary sentences appearing early in the article, while the other contains articles where good summary sentences appear late. Specifically, we construct two evaluation datasets by selecting the first 50 documents ($\mathit{D}_{\textit{early}}$, i.e., best summary occurs early) and the last 50 documents ($\mathit{D}_{\textit{late}}$, i.e., best summary occurs late) from a sample of 1000 documents that is ordered by the average extractive label index \textit{$\overline{idx}$}. Given an article with $n$ sentences indexed from $1,\ldots, n$ and a greedy extractive labels set with three sentences $(i,j,k)$\footnote{For each document, a length-$3$ extractive summary with near-optimal ROUGE score is selected following the heuristic proposed by \citet{ext5_summarunner}.},  the average index for the extractive label is computed by \textit{$\overline{idx}$}$ = (i+j+k)/3n$.

\begin{figure}[!h]
  \includegraphics[width=0.5\textwidth]{figures/data_comparision.png}
  \caption{Model comparisons of the average value for ROUGE-1,2,L F1 scores ($\overline{f}$) on $\mathit{D}_{\textit{early}}$ and $\mathit{D}_{\textit{late}}$. For each model, the results were obtained by averaging $\overline{f}$  across ten trials with 100 epochs in each trail. $\mathit{D}_{\textit{early}}$ and  $\mathit{D}_{\textit{late}}$ consist of 50 articles each, such that the good summary sentences appear early and late in the article, respectively. We observe a significant advantage of \textsc{BanditSum } compared to RNES and RNES3 (based on the sequential binary labeling setting) on $\mathit{D}_{late}$.}
  \label{fig:model_comparison}
\end{figure}

Given these two subsets of the data, three different models (\textsc{BanditSum}, RNES and RNES3) are trained and evaluated on each of the two datasets without extractive labels. Since the original sequential RL model (RNES) is unstable without supervised pre-training, we propose the RNES3 model that is limited to select no more then three sentences. Starting with random initializations without supervised pre-training, we train each model ten times for 100 epochs and plot the learning curve of the average ROUGE-F1 score computed based on the trained model in Figure \ref{fig:model_comparison}. We can clearly see that \textsc{BanditSum } finds a better solution more quickly than RNES and RNES3 on both datasets. Moreover, it displays a significantly speed-up in the exploration and finds the best solution when good summary sentences appeared later in the document ($\mathit{D}_{\textit{late}}$).

\section{Conclusion \label{sec:conclusion}}
In this work, we presented a contextual bandit learning framework, \textsc{BanditSum }, for extractive summarization, based on neural networks and reinforcement learning algorithms. \textsc{BanditSum } does not require sentence-level extractive labels and optimizes ROUGE scores between summaries generated by the model and abstractive reference summaries. Empirical results show that our method performs better than or comparable to state-of-the-art extractive summarization models which must be pre-trained on extractive labels, and converges using significantly fewer update steps than competing approaches. In future work, we will explore the direction of adding an extra coherence reward \citep{DBLP:conf/aaai/WuH18} to improve the quality of extracted summaries in terms of sentence discourse relation.

\section*{Acknowledgements}
The research was supported in part by Natural Sciences and Engineering Research Council of Canada (NSERC). The authors would like to thank Compute Canada for providing the computational resources.

\section{Credits}

This document has been adapted from the instructions for earlier ACL
and NAACL proceedings. It represents a recent build from
\url{https://github.com/acl-org/acl-pub}, with modifications by Micha
Elsner and Preethi Raghavan, based on the NAACL 2018 instructions by
Margaret Michell and Stephanie Lukin, 2017/2018 (NA)ACL bibtex
suggestions from Jason Eisner, ACL 2017 by Dan Gildea and Min-Yen Kan,
NAACL 2017 by Margaret Mitchell, ACL 2012 by Maggie Li and Michael
White, those from ACL 2010 by Jing-Shing Chang and Philipp Koehn,
those for ACL 2008 by Johanna D. Moore, Simone Teufel, James Allan,
and Sadaoki Furui, those for ACL 2005 by Hwee Tou Ng and Kemal
Oflazer, those for ACL 2002 by Eugene Charniak and Dekang Lin, and
earlier ACL and EACL formats.  Those versions were written by several
people, including John Chen, Henry S. Thompson and Donald
Walker. Additional elements were taken from the formatting
instructions of the {\em International Joint Conference on Artificial
  Intelligence} and the \emph{Conference on Computer Vision and
  Pattern Recognition}.

\section{Introduction}

The following instructions are directed to authors of papers submitted
to \confname{} or accepted for publication in its proceedings. All
authors are required to adhere to these specifications. Authors are
required to provide a Portable Document Format (PDF) version of their
papers. \textbf{The proceedings are designed for printing on A4
paper.}

\section{General Instructions}

Manuscripts must be in two-column format.  Exceptions to the
two-column format include the title, authors' names and complete
addresses, which must be centered at the top of the first page, and
any full-width figures or tables (see the guidelines in
Subsection~\ref{ssec:first}). {\bf Type single-spaced.}  Start all
pages directly under the top margin. See the guidelines later
regarding formatting the first page.  The manuscript should be
printed single-sided and its length
should not exceed the maximum page limit described in Section~\ref{sec:length}.
Pages are numbered for  initial submission. However, {\bf do not number the pages in the camera-ready version}.

By uncommenting {\small\verb|\aclfinalcopy|} at the top of this
 document, it will compile to produce an example of the camera-ready formatting; by leaving it commented out, the document will be anonymized for initial submission.

The review process is double-blind, so do not include any author information (names, addresses) when submitting a paper for review.
However, you should maintain space for names and addresses so that they will fit in the final (accepted) version.  The \confname{} \LaTeX\ style will create a titlebox space of 2.5in for you when {\small\verb|\aclfinalcopy|} is commented out.

The author list for submissions should include all (and only) individuals who made substantial contributions to the work presented. Each author listed on a submission to \confname{} will be notified of submissions, revisions and the final decision. No authors may be added to or removed from submissions to \confname{} after the submission deadline.

\subsection{The Ruler}
The \confname{} style defines a printed ruler which should be presented in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document without the provided
style files, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment the {\small\verb|\aclfinalcopy|} command in the document preamble.)

Reviewers: note that the ruler measurements do not align well with
lines in the paper -- this turns out to be very difficult to do well
when the paper contains many figures and equations, and, when done,
looks ugly. In most cases one would expect that the approximate
location will be adequate, although you can also use fractional
references ({\em e.g.}, the first paragraph on this page ends at mark $108.5$).

\subsection{Electronically-available resources}

\conforg{} provides this description in \LaTeX2e{} ({\small\tt
  emnlp2018.tex}) and PDF format ({\small\tt emnlp2018.pdf}), along
with the \LaTeX2e{} style file used to format it ({\small\tt
  emnlp2018.sty}) and an ACL bibliography style ({\small\tt
  acl\_natbib\_nourl.bst}) and example bibliography ({\small\tt
  emnlp2018.bib}).  These files are all available at
\url{http://emnlp2018.org/downloads/emnlp18-latex.zip}; a Microsoft
Word template file ({\small\tt emnlp18-word.docx}) and example
submission pdf ({\small\tt emnlp18-word.pdf}) is available at
\url{http://emnlp2018.org/downloads/emnlp18-word.zip}.  We strongly
recommend the use of these style files, which have been appropriately
tailored for the \confname{} proceedings.

\subsection{Format of Electronic Manuscript}
\label{sect:pdf}

For the production of the electronic manuscript you must use Adobe's
Portable Document Format (PDF). PDF files are usually produced from
\LaTeX\ using the \textit{pdflatex} command. If your version of
\LaTeX\ produces Postscript files, you can convert these into PDF
using \textit{ps2pdf} or \textit{dvipdf}. On Windows, you can also use
Adobe Distiller to generate PDF.

Please make sure that your PDF file includes all the necessary fonts
(especially tree diagrams, symbols, and fonts with Asian
characters). When you print or create the PDF file, there is usually
an option in your printer setup to include none, all or just
non-standard fonts.  Please make sure that you select the option of
including ALL the fonts. \textbf{Before sending it, test your PDF by
  printing it from a computer different from the one where it was
  created.} Moreover, some word processors may generate very large PDF
files, where each page is rendered as an image. Such images may
reproduce poorly. In this case, try alternative ways to obtain the
PDF. One way on some systems is to install a driver for a postscript
printer, send your document to the printer specifying ``Output to a
file'', then convert the file to PDF.

It is of utmost importance to specify the \textbf{A4 format} (21 cm
x 29.7 cm) when formatting the paper. When working with
{\tt dvips}, for instance, one should specify {\tt -t a4}.
Or using the command \verb|\special{papersize=210mm,297mm}| in the latex
preamble (directly below the \verb|\usepackage| commands). Then using
{\tt dvipdf} and/or {\tt pdflatex} which would make it easier for some.

Print-outs of the PDF file on A4 paper should be identical to the
hardcopy version. If you cannot meet the above requirements about the
production of your electronic submission, please contact the
publication chairs as soon as possible.

\subsection{Layout}
\label{ssec:layout}

Format manuscripts two columns to a page, in the manner these
instructions are formatted. The exact dimensions for a page on A4
paper are:

\begin{itemize}
\item Left and right margins: 2.5 cm
\item Top margin: 2.5 cm
\item Bottom margin: 2.5 cm
\item Column width: 7.7 cm
\item Column height: 24.7 cm
\item Gap between columns: 0.6 cm
\end{itemize}

\noindent Papers should not be submitted on any other paper size.
 If you cannot meet the above requirements about the production of
 your electronic submission, please contact the publication chairs
 above as soon as possible.

\subsection{Fonts}

For reasons of uniformity, Adobe's {\bf Times Roman} font should be
used. In \LaTeX2e{} this is accomplished by putting

\begin{quote}
\begin{verbatim}
\usepackage{times}
\usepackage{latexsym}
\end{verbatim}
\end{quote}
in the preamble. If Times Roman is unavailable, use {\bf Computer
  Modern Roman} (\LaTeX2e{}'s default).  Note that the latter is about
  10\% less dense than Adobe's Times Roman font.

\begin{table}[t!]
\begin{center}
\begin{tabular}{|l|rl|}
\hline \bf Type of Text & \bf Font Size & \bf Style \\ \hline
paper title & 15 pt & bold \\
author names & 12 pt & bold \\
author affiliation & 12 pt & \\
the word ``Abstract'' & 12 pt & bold \\
section titles & 12 pt & bold \\
document text & 11 pt  &\\
captions & 10 pt & \\
abstract text & 10 pt & \\
bibliography & 10 pt & \\
footnotes & 9 pt & \\
\hline
\end{tabular}
\end{center}
\caption{\label{font-table} Font guide. }
\end{table}

\subsection{The First Page}
\label{ssec:first}

Center the title, author's name(s) and affiliation(s) across both
columns. Do not use footnotes for affiliations. Do not include the
paper ID number assigned during the submission process. Use the
two-column format only when you begin the abstract.

{\bf Title}: Place the title centered at the top of the first page, in
a 15-point bold font. (For a complete guide to font sizes and styles,
see Table~\ref{font-table}) Long titles should be typed on two lines
without a blank line intervening. Approximately, put the title at 2.5
cm from the top of the page, followed by a blank line, then the
author's names(s), and the affiliation on the following line. Do not
use only initials for given names (middle initials are allowed). Do
not format surnames in all capitals ({\em e.g.}, use ``Mitchell'' not
``MITCHELL'').  Do not format title and section headings in all
capitals as well except for proper names (such as ``BLEU'') that are
conventionally in all capitals.  The affiliation should contain the
author's complete address, and if possible, an electronic mail
address. Start the body of the first page 7.5 cm from the top of the
page.

The title, author names and addresses should be completely identical
to those entered to the electronical paper submission website in order
to maintain the consistency of author information among all
publications of the conference. If they are different, the publication
chairs may resolve the difference without consulting with you; so it
is in your own interest to double-check that the information is
consistent.

{\bf Abstract}: Type the abstract at the beginning of the first
column. The width of the abstract text should be smaller than the
width of the columns for the text in the body of the paper by about
0.6 cm on each side. Center the word {\bf Abstract} in a 12 point bold
font above the body of the abstract. The abstract should be a concise
summary of the general thesis and conclusions of the paper. It should
be no longer than 200 words. The abstract text should be in 10 point font.

{\bf Text}: Begin typing the main body of the text immediately after
the abstract, observing the two-column format as shown in

the present document. Do not include page numbers.

{\bf Indent}: Indent when starting a new paragraph, about 0.4 cm. Use 11 points for text and subsection headings, 12 points for section headings and 15 points for the title.

\begin{table}
\centering
\small
\begin{tabular}{cc}
\begin{tabular}{|l|l|}
\hline
{\bf Command} & {\bf Output}\\\hline
\verb|{\"a}| & {\"a} \\
\verb|{\^e}| & {\^e} \\
\verb|{\`i}| & {\`i} \\
\verb|{\.I}| & {\.I} \\
\verb|{\o}| & {\o} \\
\verb|{\'u}| & {\'u}  \\
\verb|{\aa}| & {\aa}  \\\hline
\end{tabular} &
\begin{tabular}{|l|l|}
\hline
{\bf Command} & {\bf  Output}\\\hline
\verb|{\c c}| & {\c c} \\
\verb|{\u g}| & {\u g} \\
\verb|{\l}| & {\l} \\
\verb|{\~n}| & {\~n} \\
\verb|{\H o}| & {\H o} \\
\verb|{\v r}| & {\v r} \\
\verb|{\ss}| & {\ss} \\\hline
\end{tabular}
\end{tabular}
\caption{Example commands for accented characters, to be used in, {\em e.g.}, \BibTeX\ names.}\label{tab:accents}
\end{table}

\subsection{Sections}

{\bf Headings}: Type and label section and subsection headings in the
style shown on the present document.  Use numbered sections (Arabic
numerals) in order to facilitate cross references. Number subsections
with the section number and the subsection number separated by a dot,
in Arabic numerals.
Do not number subsubsections.

\begin{table*}[t!]
\centering
\begin{tabular}{lll}
  output & natbib & previous \conforg{} style files\\
  \hline
  \citep{Gusfield:97} & \verb|\citep| & \verb|\cite| \\
  \citet{Gusfield:97} & \verb|\citet| & \verb|\newcite| \\
  \citeyearpar{Gusfield:97} & \verb|\citeyearpar| & \verb|\shortcite| \\
\end{tabular}
\caption{Citation commands supported by the style file.
  The citation style is based on the natbib package and
  supports all natbib citation commands.
  It also supports commands defined in previous \conforg{} style files
  for compatibility.
  }
\end{table*}

{\bf Citations}: Citations within the text appear in parentheses
as~\cite{Gusfield:97} or, if the author's name appears in the text
itself, as Gusfield~\shortcite{Gusfield:97}.
Using the provided \LaTeX\ style, the former is accomplished using
{\small\verb|\cite|} and the latter with {\small\verb|\shortcite|} or {\small\verb|\newcite|}. Collapse multiple citations as in~\cite{Gusfield:97,Aho:72}; this is accomplished with the provided style using commas within the {\small\verb|\cite|} command, {\em e.g.}, {\small\verb|\cite{Gusfield:97,Aho:72}|}. Append lowercase letters to the year in cases of ambiguities.
 Treat double authors as
in~\cite{Aho:72}, but write as in~\cite{Chandra:81} when more than two
authors are involved. Collapse multiple citations as
in~\cite{Gusfield:97,Aho:72}. Also refrain from using full citations
as sentence constituents.

We suggest that instead of
\begin{quote}
  ``\cite{Gusfield:97} showed that ...''
\end{quote}
you use
\begin{quote}
``Gusfield \shortcite{Gusfield:97}   showed that ...''
\end{quote}

If you are using the provided \LaTeX{} and Bib\TeX{} style files, you
can use the command \verb|\citet| (cite in text)
to get ``author (year)'' citations.

If the Bib\TeX{} file contains DOI fields, the paper
title in the references section will appear as a hyperlink
to the DOI, using the hyperref \LaTeX{} package.
To disable the hyperref package, load the style file
with the \verb|nohyperref| option: \\{\small
\verb|\usepackage[nohyperref]{acl2018}|}

\textbf{Digital Object Identifiers}: As part of our work to make ACL
materials more widely used and cited outside of our discipline, ACL
has registered as a CrossRef member, as a registrant of Digital Object
Identifiers (DOIs), the standard for registering permanent URNs for
referencing scholarly materials. \conforg{} has \textbf{not} adopted the
ACL policy of requiring camera-ready references to contain the appropriate
  DOIs (or as a second resort, the hyperlinked ACL Anthology
  Identifier). But we certainly encourage you to use
  Bib\TeX\ records that contain DOI or URLs for any of the ACL
  materials that you reference. Appropriate records should be found
for most materials in the current ACL Anthology at
\url{http://aclanthology.info/}.

As examples, we cite \cite{P16-1001} to show you how papers with a DOI
will appear in the bibliography.  We cite \cite{C14-1001} to show how
papers without a DOI but with an ACL Anthology Identifier will appear
in the bibliography.

\textbf{Anonymity:} As reviewing will be double-blind, the submitted
version of the papers should not include the authors' names and
affiliations. Furthermore, self-references that reveal the author's
identity, {\em e.g.},
\begin{quote}
``We previously showed \cite{Gusfield:97} ...''
\end{quote}
should be avoided. Instead, use citations such as
\begin{quote}
``\citeauthor{Gusfield:97} \shortcite{Gusfield:97}
previously showed ... ''
\end{quote}

Preprint servers such as arXiv.org and workshops that do not
have published proceedings are not considered archival for purposes of
submission. However, to preserve the spirit of blind review, authors
are encouraged to refrain from posting until the completion of the
review process. Otherwise, authors must state in the online submission
form the name of the workshop or preprint server and title of the
non-archival version. The submitted version should be suitably
anonymized and not contain references to the prior non-archival
version. Reviewers will be told: ``The author(s) have notified us that
there exists a non-archival previous version of this paper with
significantly overlapping text. We have approved submission under
these circumstances, but to preserve the spirit of blind review, the
current submission does not reference the non-archival version.''

\textbf{Please do not use anonymous citations} and do not include
 when submitting your papers. Papers that do not
conform to these requirements may be rejected without review.

\textbf{References}: Gather the full set of references together under
the heading {\bf References}; place the section before any Appendices,
unless they contain references. Arrange the references alphabetically
by first author, rather than by order of occurrence in the text.
By using a .bib file, as in this template, this will be automatically
handled for you. See the \verb|\bibliography| commands near the end for more.

Provide as complete a citation as possible, using a consistent format,
such as the one for {\em Computational Linguistics\/} or the one in the
{\em Publication Manual of the American
Psychological Association\/}~\cite{APA:83}. Use of full names for
authors rather than initials is preferred. A list of abbreviations
for common computer science journals can be found in the ACM
{\em Computing Reviews\/}~\cite{ACM:83}.

The \LaTeX{} and Bib\TeX{} style files provided roughly fit the
American Psychological Association format, allowing regular citations,
short citations and multiple citations as described above.

\begin{itemize}
\item Example citing an arxiv paper: \cite{rasooli-tetrault-2015}.
\item Example article in journal citation: \cite{Ando2005}.
\item Example article in proceedings, with location: \cite{borsch2011}.
\item Example article in proceedings, without location: \cite{andrew2007scalable}.
\end{itemize}
See corresponding .bib file for further details.

Submissions should accurately reference prior and related work, including code and data. If a piece of prior work appeared in multiple venues, the version that appeared in a refereed, archival venue should be referenced. If multiple versions of a piece of prior work exist, the one used by the authors should be referenced. Authors should not rely on automated citation indices to provide accurate references for prior and related work.

{\bf Appendices}: Appendices, if any, directly follow the text and the
references (but see above).  Letter them in sequence and provide an
informative title: {\bf Appendix A. Title of Appendix}.

\subsection{URLs}

URLs can be typeset using the \verb|\url| command. However, very long
URLs cause a known issue in which the URL highlighting may incorrectly
cross pages or columns in the document. Please check carefully for
URLs too long to appear in the column, which we recommend you break,
shorten or place in footnotes. Be aware that actual URL should appear
in the text in human-readable format; neither internal nor external
hyperlinks will appear in the proceedings.

\subsection{Footnotes}

{\bf Footnotes}: Put footnotes at the bottom of the page and use 9
point font. They may be numbered or referred to by asterisks or other
symbols.\footnote{This is how a footnote should appear.} Footnotes
should be separated from the text by a line.\footnote{Note the line
separating the footnotes from the text.}

\subsection{Graphics}

{\bf Illustrations}: Place figures, tables, and photographs in the
paper near where they are first discussed, rather than at the end, if
possible.  Wide illustrations may run across both columns.  Color
illustrations are discouraged, unless you have verified that
they will be understandable when printed in black ink.

{\bf Captions}: Provide a caption for every illustration; number each one
sequentially in the form:  ``Figure 1. Caption of the Figure.'' ``Table 1.
Caption of the Table.''  Type the captions of the figures and
tables below the body, using 11 point text.

\subsection{Accessibility}
\label{ssec:accessibility}

In an effort to accommodate people who are color-blind (as well as those printing
to paper), grayscale readability for all accepted papers will be
encouraged.  Color is not forbidden, but authors should ensure that
tables and figures do not rely solely on color to convey critical
distinctions. A simple criterion: All curves and points in your figures should be clearly distinguishable without color.

\section{Translation of non-English Terms}

It is also advised to supplement non-English characters and terms
with appropriate transliterations and/or translations
since not all readers understand all such characters and terms.
Inline transliteration or translation can be represented in
the order of: original-form transliteration ``translation''.

\section{Length of Submission}
\label{sec:length}

The \confname{} main conference accepts submissions of long papers and
short papers.
 Long papers may consist of up to eight (8) pages of
content plus unlimited pages for references. Upon acceptance, final
versions of long papers will be given one additional page -- up to nine (9)
pages of content plus unlimited pages for references -- so that reviewers' comments
can be taken into account. Short papers may consist of up to four (4)
pages of content, plus unlimited pages for references. Upon
acceptance, short papers will be given five (5) pages in the
proceedings and unlimited pages for references.

For both long and short papers, all illustrations and tables that are part
of the main text must be accommodated within these page limits, observing
the formatting instructions given in the present document. Supplementary
material in the form of appendices does not count towards the page limit; see appendix A for further information.

However, note that supplementary material should be supplementary
(rather than central) to the paper, and that reviewers may ignore
supplementary material when reviewing the paper (see Appendix
\ref{sec:supplemental}). Papers that do not conform to the specified
length and formatting requirements are subject to be rejected without
review.

Workshop chairs may have different rules for allowed length and
whether supplemental material is welcome. As always, the respective
call for papers is the authoritative source.

\section*{Acknowledgments}

The acknowledgments should go immediately before the references.  Do
not number the acknowledgments section. Do not include this section
when submitting your paper for review. \\

\noindent {\bf Preparing References:} \\

Include your own bib file like this:
{\small\verb|\bibliographystyle{acl_natbib_nourl}|
\verb|


\section{Introduction}
The H~{\sc i} proximity effect has been widely studied since it was
discovered. One thing of particular
interest is that the study of the H~{\sc i} proximity
effect provides a method of measuring the intensity of the MUVB
at the H~{\sc i} edge. However, the H~{\sc i} proximity effect
does not reveal information on the spectral shape of the MUVB.\par

We demonstrate that the proximity effect
on metal elements may provide a method for constraining
the spectral shape of the MUVB. Songalia and Cowie (1996)
\cite{SonC96} reported that
metal elements are detected in 75\% of clouds with $N_{\hbox{$\scriptstyle\rm HI$}} >3.0\times 10^{14}
\,{\rm cm^{-2}}$. The lack of data for studying the proximity effect
on metal elements is going to overcome. If future observations
detect what we expect for the proximity effect on metal elements,
it will confirm that the proximity effect is an ionization effect
and photoionization is the dominant ionization mechanism for metal
ions in which the proximity effect is seen.
\section{Results}
The electronic configurations of metal elements are complicated
and a large number of ionization stages can coexist simultaneously.
In general, a simple feature does not exist for metal proximity effect,
unlike for the H~{\sc i} proximity effect. Therefore, we may study the
dependence of the column density ratios of two ions in successive
ionization stages on $\omega$.\par
We found that [O~{\sc iv}/O~{\sc iii}]$_{\hbox{$\scriptstyle\rm Q/\rm B\ $}}$ is the most optimum
indicator of the spectral shape of the MUVB (we use [X]$_{\hbox{$\scriptstyle\rm Q/\rm B\ $}}$ to represent
the ratio of the value of X in the vicinity of the QSO to that
in background case). The HST does provide
the possibility to observe O~{\sc iv} and O~{\sc iii} simultaneously
\cite{TriL97} and \cite{VogR95}.
It is clear from Figure 1 that [O~{\sc iv}/O~{\sc iii}]$_{\hbox{$\scriptstyle\rm Q/\rm B\ $}}$
strongly depends on the assumed spectral shape of the MUVB
and less sensitive to the assumed $U_{\hbox{$\scriptstyle\rm B$}}$ (the ionization parameter
in background case)
in small $\omega$ region. Therefore, [O~{\sc iv}/O~{\sc iii}]$_{\hbox{$\scriptstyle\rm Q/\rm B\ $}}$
is good for constraining the spectral shape even if $U_{\hbox{$\scriptstyle\rm B$}}$ is unknown.
\begin{figure}
\centerline{\vbox{
\psfig{figure=liujmF1.ps,height=7.cm}
}}
\caption[]{[O~{\sc iv}/O~{\sc iii}]$_{\hbox{$\scriptstyle\rm Q/\rm B\ $}}$ vs $\omega$.
The heavy curves are for $U_{\hbox{$\scriptstyle\rm B$}}=10^{-2.7}$; the light
curves are for $U_{\hbox{$\scriptstyle\rm B$}}=10^{-1.5}$.
The solid curves are for $\alpha_{\hbox{$\scriptstyle\rm B$}}=\alpha_{\hbox{$\scriptstyle\rm Q$}}=1.5$ and $b=1$;
the dotted curves are for $\alpha_{\hbox{$\scriptstyle\rm B$}}=0.5$,
$b=12.5$, and $\alpha_{\hbox{$\scriptstyle\rm Q$}}=1.5$; the dashed curves are for
$\alpha_{\hbox{$\scriptstyle\rm B$}}=0.3$, $b=100$, and $\alpha_{\hbox{$\scriptstyle\rm Q$}}=1.5$.
}
\end{figure}
The ratio Si~{\sc iv}/C~{\sc iv}, contrast to previous thought, is
not a good indicator to the spectral shape, because it
strongly depends on both the ionization parameter $U_{\hbox{$\scriptstyle\rm B$}}$ and
the spectral shape of the MUVB. A small uncertainty on either of them
must results in a big error in determination of the other.
We also found that the extra ionizing radiation from QSOs does not
always reduce the ratio C~{\sc iv}/H~{\sc i}, but enhances
C~{\sc iv}/H~{\sc i} in the case of $U_{\hbox{$\scriptstyle\rm B$}}\lower.5ex\hbox{\ltsima} 10^{-2}$.
So, the observed excess of C~{\sc iv} systems in luminous
QSOs may be a photoionization effect,
but not a gravitational lensing effect \cite{VanQ96}.

\acknowledgements{We are grateful to Dr Avery Meiksin for his comments.}

\begin{iapbib}{99}{
\bibitem{LiuM97} Liu, J.M., \& Meiksin, A., 1997, in preparation
\bibitem{SonC96} Songaila, A., \& Cowie, L.L. 1996, AJ, 112, 335
\bibitem{TriL97} Tripp, T.M., Lu, L., \& Savage, B.D. 1997,
	[astro-ph/9703080]
\bibitem{VanQ96} Vanden Berk, D.E., Quashnock, J.M., York, D.G.,
	\& Yanny, B. 1996, 469, 78
\bibitem{VogR95} Vogel, S., \& Reimers, D. 1995, A\&A, 294, 377
}
\end{iapbib}
\vfill
\end{document}


\section{Introduction}\label{sec1}

The temporal and spatial mixing of strong fundamental frequency ($\omega$) and second harmonic ($2\omega$) laser fields in gas-phase medium can produce a ultrashort terahertz (THz) pulse \cite{Cook00}. Given the absence of emitter damage and absorption, the two-color scheme is particularly promising for intense and broadband THz radiation \cite{Koulouklidis2020, Mitrofanov2020,Jang2019}. The wonderful characteristics of the THz source has already been used in time-resolved terahertz spectroscopy \cite{Pashkin2011,Valverde-Chavez2015,Wang2016} and transient absorption spectroscopy \cite{Chen2016}. The bright broadband THz radiation has potential applications in broadband spectroscopy \cite{Cossel17}, atomic and molecular ultrafast imaging \cite{Zhang2018a}, advanced accelerator technique \cite{Zhang2018b}, etc.

Although the THz generation in two-color scheme has already been a widely-used, well-established technique, yet the attempts to improve the technique are never abandoned. In experimental setups, multiple control knobs have been tuned to increase the THz conversion efficiency and bandwidth. Firstly, the appropriate parameters of driving lasers, including the wavelength and pulse duration, can optimize the THz generation. The wavelength scaling investigations \cite{Clerici2013,Nguyen2019} exhibited that the driving laser with longer wavelength strongly enhances the THz down-conversion efficiency by one or two orders of magnitude comparing to THz generation driven by 800 nm pulses at same input energies. In practical experiments, mid-infrared fields at 3.9 $\mu$m delivered by optical parametric amplifiers have been used to produce extremely strong THz field above 100 MV/cm \cite{Koulouklidis2020, Mitrofanov2020,Jang2019}. Simultaneously, tuning the wavelength ratio of two-color fields \cite{Vvedenskii2014,Kostin2016,Zhang2017} can manipulate the central wavelength and effectively extend the THz bandwidth \cite{Thomson2010,Babushkin2011,Balciunas2015}. If more than two-color fields are involved in the process, multiple wavelength fields can also improve the THz performance \cite{Martinez2015}.
Short pulses by means of pulse compression technique significantly broadens the bandwidth of THz pulse, yielding  supercontinuum radiation of bandwidth $>$100 THz by using 7 fs driving pulse \cite{Thomson2010,Matsubara2012,Blank2013}.
In addition, controlling the polarization of two-color fields is also necessary. It has been well acknowledged that the relative polarization of two-color lasers should be optimized in THz conversion process \cite{Xie2006,Zhang2020}. And two-color co-rotating circularly-polarized fields can further amplify THz emission by a factor of 5 comparing to linearly polarized fields \cite{Meng2016}.
Finally, the plasma profile also plays a role on THz generation. By manipulating the focal length \cite{Oh2013}, beam wavefront  \cite{Kuk2016,Zhang2018c,Sheng2021}, gas pressure and species  \cite{Yoo2019}, the spatiotemporal dynamics in plasma formation and pulse propagation have been modified to apparently influence THz radiation.

In the two-color scheme, the temporal and spatial overlapping of $\omega$ and $2\omega$ beams is a basic premise in beam alignment, where the two plasmas induced by the $\omega$ and $2\omega$ beams should spatially merge into a big plasma (overlapped plasma) acting as the best experimental condition. However, we found, when the two-color plasmas are concentrically separated along the propagation axis into cascading plasmas, the characteristics of THz radiation is significantly improved. Therefore, we introduce a new control knob, the distance between the two-color foci, to considerably increase the yield and bandwidth of THz emission. In this paper, the THz strength and spectral profile generated by linearly and circularly-polarized two-color fields, are measured as a function of the distance, which shows that the maximum conversion efficiency exceeds $10^{-3}$ with bandwidth $>$100 THz by using 800 nm, 35 fs driving lasers. The theory involving laser pulse propagation equation and photocurrent model indicates that the spatial redistribution of laser intensity in cascading plasmas balances THz generation and absorption in the plasma channel, leading to the maximum THz output.

\section{Experiment}\label{sec2}

\begin{figure}[h]%
\centering
\includegraphics[width=0.9\textwidth]{Geometry.jpg}
\caption{Experimental schematics of ultra-broadband THz amplification. The $\omega$ and 2$\omega$ beams are respectively focused by two identical lenses with 10 cm focus length. In the experiment, we fix the lens position of $\omega$ beam, and the position of the 2$\omega$ lens is longitudinally moved along the propagation direction. The profile of the plasma channel strongly depends on the displacement $d$ of the 2$\omega$ lens, which is introduced as a new knob to amplify THz radiation.}\label{fig1}
\end{figure}

The experiment is implemented on a Ti:sapphire femtosecond amplification system. The $\omega$ beam with center wavelength of 800 nm is converted to the 2$\omega$ beam by a $\beta$-barium borate (BBO) crystal. The $\omega$ and 2$\omega$ beams respectively pass through two arms of a Michelson interferometer, and the polarization of the two beams can be individually controlled by the waveplates in the two arms. The lengths of the two arms are actively stabilized, and the relative phase delay $\tau$ of the $\omega$-2$\omega$ pulses is tunable with the accuracy of sub-femtosecond. The commonly-used focusing geometry, using one mirror (or lens) to focus two-color beams, is not applied in our setup, whereas a complicated focusing geometry is utilized. We place two identical lenses with 10 cm focus length in two arms of the interferometer to respectively focus the $\omega$ and 2$\omega$ beams, and the position of the lens can be moved along the propagation direction to change the focusing condition and plasma channel profile. As shown in Fig.~\ref{fig1}, the foci of the $\omega$ and 2$\omega$ beams are spatially separated to form cascading plasmas. The strength and bandwidth of the THz radiation strongly depends on the distance between the foci of $\omega$ and 2$\omega$ beams, which is controlled by the lens displacement $d$ of the 2$\omega$ beam. The THz strength as a function of $d$ is calibrated with electro-optical sampling (EOS) and pyroelectric detector, and the bandwidth is measured with a Fourier transform spectroscopy. The setup of THz generation and detection is referred to \textit{Method Section}.

\section{Experimental Results}\label{sec3}

\begin{figure}
\centering
\includegraphics[width=0.8\textwidth]{IntensityTHz.pdf}
\caption{Terahertz amplification in cascading plasmas. (a) The profile of cascading plasmas versus lens displacement $d$. The $\omega$ and 2$\omega$ beams produce the plasmas respectively. The two plasmas are spatially overlapped when $d=0 \ \mathrm{mm}$, and concentrically separated when $d$ is varied.  The plasmas produced by 400 nm and 800 nm beams are marked. (b) The THz power versus $d$ and input laser pulse energy $I_{\mathrm{THz}} (d, I_{\omega},I_{\mathrm{2}\omega})$ measured with pyroelectric detector. The \textit{x} axis is lens displacement $d$, and \textit{y} axis is THz power. The THz pulse is generated by the co-rotating circularly-polarized two-color fields. Surprisingly, the cascading plasmas at $d=2\ \mathrm{mm}$ radiates the most intense THz pulse, which is significantly larger than that at $d=0\ \mathrm{mm}$. (c) The \textit{s}-polarized THz electric field versus $d$ and input laser pulse energy $\boldsymbol{E}_{\mathrm{THz}} (d, I_{\omega},I_{\mathrm{2}\omega})$, which is measured with electro-optical sampling.}
\label{fig2}
\end{figure}

Firstly, we measured the THz intensity $I_{\mathrm{THz}}$ versus the lens displacement $d$. In our setup, both the $\omega$ and 2$\omega$ beams produce plasmas, and the distance between the plasmas can be changed. The plasma profiles versus $d$ is recorded by a CCD camera, as shown in Fig.~\ref{fig2}(a). In commonly used setups, two plasmas are spatially overlapped to form a single plasma, where we define as $d=0\ \mathrm{mm}$. The third harmonic generation \cite{He2021} versus $d$ is used to precisely calibrate the position $d=0\ \mathrm{mm}$.  When changing $d$ the big plasma is separated along the propagation direction into two cascading plasmas, and the distance between the cascading plasmas approximately equals the lens displacement $d$. The brightness of 2$\omega$ plasma is obviously varied versus $d$. When the 2$\omega$ plasma approaches the $\omega$ plasma, the brightness of 2$\omega$ plasma suddenly decreases, reflecting the complex process in plasma channel formation.

After the spatiotemporal optimization of the two-color beams, the THz power versus $d$ and input laser power $I_{\mathrm{THz}}(d, I_{\omega},I_{\mathrm{2}\omega})$ is calibrated by a commercial pyroelectric detector, shown in Fig.~\ref{fig2}(b). Here, the co-rotating circularly-polarized two-color fields are used to obtain maximum conversion efficiency, and the THz emissions $I_{\mathrm{THz}}(d)$ in linearly polarized two-color fields have the similar $d$-dependent behavior. It is surprising that the THz radiation is significantly amplified by over one order of magnitude at $d=2\ \mathrm{mm}$ when the two plasmas are spatially well separated along propagation direction. The maximum conversion efficiency is obtained when the plasma formed by $\omega$ field is spatially ahead of the 2$\omega$ plasma. When decreasing input laser power, the maximum $I_{\mathrm{THz}}(d, I_{\omega},I_{\mathrm{2}\omega})$ approaches $d=0\ \mathrm{mm}$. The observation conflicts with the empirical awareness that the spatial overlapping of two-color foci is a prerequisite for the most efficient terahertz generation.

Here, we can approximately estimate the conversion efficiency in cascade plasmas. Before the pyroelectric detector, two silicon wafers block the scattering light with $\sim$0.2 transmittance of THz beam. And the chopper blocks half of the input laser energy. Thus, the THz pulse energy at $d=2\ \mathrm{mm}$ exceeds 1 $\mu$J, and the conversion efficiency approaches 0.002, which is one order of magnitude higher than the analogous configuration with efficiency of $10^{-4}$. The estimation can also be confirmed in Fig.~\ref{fig2}(b), where $I_{\mathrm{THz}}(d=2\ \mathrm{mm})$ is $\sim$10 times higher than $I_{\mathrm{THz}}(d=0\ \mathrm{mm})$.

To further confirm the amplification in cascading plasmas, the THz electric field $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}(d, I_{\omega},I_{\mathrm{2}\omega})$ is measured with EOS, shown in Fig.~\ref{fig2}(c), which exhibits the similar behavior as $I_{\mathrm{THz}}(d, I_{\omega},I_{\mathrm{2}\omega})$. Here, only \textit{s}-polarized $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}$ is shown in the main text, and \textit{p}-polarized $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}$ has a similar manner (\textit{Supplementary Information}). Comparing to the intensity measurement with pyroelectric detector, the EOS measurement shows two different manners: (1) The $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}$ is sensitive to phase delay $\tau$ of $\omega$-2$\omega$ fields, but $I_{\mathrm{THz}}$ is not $\tau$ dependent. By scanning $\tau$, $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}$ is periodically modulated versus $\tau$, which is presented in \textit{Supplementary Information}. (2) The maximum efficiencies appear at different $d$. The maximum $I_{\mathrm{THz}}$ appears at $d=2\ \mathrm{mm}$, whereas $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}$ is optimized at $d=1.25\ \mathrm{mm}$. The discrepancy probably originates from the detection bandwidth of the two methods. The EOS only has response below 3 THz, while the pyroelectric detector has linear and relatively flat response function in the range of terahertz to infrared frequency. A possible explanation for unpronounced $\tau$-dependent $I_{\mathrm{THz}}$ is that, the high-frequency contributions from different spatial positions are averaged in the far field, which washes out the yield modulation versus $\tau$.

\begin{figure}
\centering
\includegraphics[width=0.8\textwidth]{THzfrequency.pdf}
\caption{Terahertz bandwidth versus $d$ between two-color plasmas. The spectral features of THz generation $\tilde{E}(\nu)$ in linearly polarized two-color fields are presented at $d=-2\ \mathrm{mm}$ (red), $d=-1\ \mathrm{mm}$ (blue) and $d=0\ \mathrm{mm}$ (green). The spectral intensities are represented on linear scale. When separating the two-color plasmas, $\tilde{E}(\nu)$ is significantly broadened and shifted to high frequency region. Inset: The interferograms obtained by Fourier transform spectroscopy.}
\label{fig3}
\end{figure}

The THz spectral features $\tilde{I}_{\mathrm{THz}}(\nu)$ at $I_{\omega} = 870\ \mathrm{\mu J},I_{\mathrm{2}\omega} = 460\ \mathrm{\mu J}$ are measured with Fourier transform spectroscopy, shown in Fig.~\ref{fig3}. The home-built Fourier transform spectrometer has been calibrated with a commercial optical parametric amplifier. Due to the noise, the very low-frequency region ($< 5$ THz) is not credible, which however can be measured with EOS. The spectral measurement shows that, when varying $d$ between two-color plasmas, the THz bandwidth is significantly broadened above 100 THz. The bandwidth broadening can be validated by the interferograms (Fig.~\ref{fig3} Inset). The temporal cycles emitted from cascading plasmas is much narrower than that in overlapped plasma.
The spectrum measurement exhibits that, our new method can not only boost the THz strength, which can achieve comparable conversion efficiency as strong terahertz generation by tilted wave front excitation in lithium niobate crystal, but can also span THz spectral bandwidth up to mid-infrared region.

\section{Theory}\label{sec4}

In order to understand the conversion efficiency enhancement in cascading plasmas, the THz radiation is numerically investigated with a (2D+1) laser pulse propagation equation combined with photocurrent model. In simulation, the free electron ensemble in cascading plasmas accelerated by asymmetric $\omega$-2$\omega$ fields induces residual photocurrent, which leads to the THz radiation. The absorption of the THz wave in plasma channel is also included in the model (\textit{Method Section}).

\begin{figure}
\centering
\includegraphics[width=0.75\textwidth]{Theory.pdf}
\caption{Comparison between measurement and theory. (a) THz generation $I_{\mathrm{THz}}(d)$ (blue dots) and  $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}(d)$ (yellow stars) when $I_{\omega} = 870\ \mathrm{\mu J},I_{\mathrm{2}\omega} = 460\ \mathrm{\mu J}$ in linearly polarized $\omega$-2$\omega$ fields, and theoretical prediction (red line). (b) (c) Simulated spatial distributions of electron density in plasma at $d=0\ \mathrm{mm}$ and $d= 2\ \mathrm{mm}$. In panel (c) at $d= 2\ \mathrm{mm}$, the $\omega$ and 2$\omega$ foci are spatially separated into cascading plasmas. The left plasma is formed by the $\omega$ beam, and the right plasma is formed by the 2$\omega$ beam. The simulation well reproduces the fluorescence images of plasmas in Fig.~\ref{fig2} (a), which reflect electron density distributions in plasmas. (d) (e) The theoretical results of spatial distributions of THz generation. (f) (g) THz generation minus absorption (net THz emission). The dense plasma obviously attenuates THz emission. All the values in contour maps are logarithmically scaled and normalized to the maximum.}
\label{fig4}
\end{figure}

The THz radiation $I_{\mathrm{THz}}(d)$ and  $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}(d)$ at $I_{\omega} = 870\ \mathrm{\mu J},I_{\mathrm{2}\omega} = 460\ \mathrm{\mu J}$ in linearly polarized $\omega$-2$\omega$ fields are compared to the theoretical simulation, as shown in Fig.~\ref{fig4}(a). In simulation, we estimate that the waists of $\omega$ and 2$\omega$ beams at the foci are $20\ \mathrm{\mu m}$ and $10\ \mathrm{\mu m}$, and peak powers of $\omega$ and 2$\omega$ fields are $100\ \mathrm{TW/cm^2}$ and $150\ \mathrm{TW/cm^2}$.
Fig.~\ref{fig4}(a) shows the simulated THz strength versus $d$ at 1 THz, which approximately agrees with the measured $d$-dependent behaviors of $I_{\mathrm{THz}}(d)$ and  $\ensuremath{\boldsymbol{E}}_{\mathrm{THz}}(d)$.

We investigate electron density distribution, THz generation and absorption in space, and give a preliminary explanation for the THz enhancement in cascading plasmas. As shown in Fig.~\ref{fig4}(b) and (c), when the foci of $\omega$ and 2$\omega$ spatially overlap at $d=0\ \mathrm{mm}$, the two-color fields produce very dense plasma, which is confined to a small volume with high electron density and strong gradient.
In the center of the plasma volume, the electron density is estimated as $n_e \sim 10^{18}\ /\mathrm{cm}^3$.
Comparatively, when the two-color foci are separated, the plasma is stretched into cascading plasmas with larger length and lower electron density. The electron density in cascade plasmas $n_e \sim 10^{16}\ /\mathrm{cm}^3$, which is  more homogeneously spatially distributed than the overlapped plasma.

The THz generation is highly relevant to electron density. Although the spatial density of THz emission in the cascading plasmas (Fig.~\ref{fig4}(e)) is smaller than that in overlapped plasma (Fig.~\ref{fig4}(d)) due to low electron density, yet, it is compensated after full space integration of plasma volume. Therefore, the overlapped plasma and cascading plasmas have similar throughput of THz emission. It can be further verified by the residual photocurrents at spatial samples at $z = 1.5,\  2.5,\  3.5\ \mathrm{mm}$  (\textit{Supplementary Information}), indicating the spatial THz generation in plasma.

The net THz emission from plasma depends on both THz generation and absorption in plasma. Here, we define absorption length $L_a$ to describe how long the THz wave is able to propagate in plasma filament. In the center of the overlapped plasma, the electron density $n_e \sim 10^{18} /\mathrm{cm}^3$, corresponds to absorption length $L_a \sim 1\ \mathrm{\mu m}$, which is far less than the plasma length. Hence, the THz wave is mostly depleted in plasma volume, as shown in (Fig.~\ref{fig4}(f)). Comparatively, the absorption length in cascading plasmas can be estimated as $L_a \sim 6.8\ \mathrm{mm}$ according to average electron density of plasma volume. It indicates that the THz absorption in cascading plasmas is much weaker than that in overlapped plasma (Fig.~\ref{fig4}(e)). Thus, by manipulating spatial redistribution of laser input energy, the THz generation and absorption are self optimized in cascading plasmas, finally achieving one order of magnitude enhancement of conversion efficiency.

The THz spectral broadening can hardly be explained with the plasma absorption. We tentatively attribute spectral broadening to spatiotemporal reshaping of driving fields when the ultrashort pulse propagates in cascading plasmas. The dispersion and highly nonlinear effects in plasma filament lead to the temporal distortion and compression of two-color fields. The pulse reshaping would induce the rapidly time-varying photocurrent, leading to high-frequency THz component. Since the nonlinear interaction length in cascading plasmas is longer than the length of the overlapped plasma, the spectral broadening in cascading plasmas is more pronounced than that in overlapped plasma. More convincing explanations need further theoretical investigation, which may involve highly complex and rich spatiotemporal dynamics during pulse propagation in plasma filament.

\section{Conclusion}\label{sec5}

We introduce a new control knob, the distance between the two-color cascading plasmas, to promote THz radiation in two-color scheme. With the distance appropriately optimized, the THz conversion efficiency reaches $\sim10^{-3}$ and the bandwidth can be broadened $>$100 THz with 800 nm, 35 fs laser pulse. The conversion efficiency is one order of magnitude higher and the bandwidth is more than 2 times broader than the counterpart configuration of overlapped plasma. The new proposed geometry can achieve considerable brightness and supercontinuum bandwidth with fairly simple setup, which avoids sophisticated optical parametric amplifier and pulse compression technique.

The first-ever proposed method may also be applicable in long wavelength driving THz generation for further enhancement conversion efficiency, which may break the current records of the strength and bandwidth of THz ultrashort pulses. The ultra-broadband feature of THz radiation has potential applicability to study structure and ultrafast dynamics of complex systems, whereas the strength feature can be applied in nonlinear optics, strong field physics and accelerator technique. The propagation equation combined with photocurrent model indicates that the THz amplification originates from the interplay between THz generation and plasma absorption in cascading plasmas. The complex mechanism of THz radiation in cascade plasma channel is still an open question calling for further theoretical exploration.

\section{Methods
\subsection{Experimental setup}
The experimental setup is shown in \textit{Supplementary Information}. A Ti:sapphire laser delivers \textit{p}-polarized femtosecond pulse with 35 fs, $\sim$1.8 mJ/pulse, centered at 800 nm with 60 nm bandwidth.
The $\omega$ beam passes through a 200 $\mu$m type-I {$\beta$}-barium borate (BBO) crystal, and a \textit{s}-polarized 2$\omega$ beam is generated (conversion efficiency $\sim$30\%). The co-propagating $\omega$-2$\omega$ beams are separated by a dichroic mirror (DM-2) into the two arms of a Michelson interferometry. The polarization of the $\omega$-2$\omega$ beams can be arbitrarily controlled by quarter-waveplates.

The phase jitter between $\omega$-2$\omega$ beams are suppressed by introducing a actively stabilized Michelson interferometry. To stabilize the relative phase of two arms of Michelson interferometry, a continuous green laser (532 nm) co-propagates with the $\omega$-2$\omega$ beams and interferes. The interference fringes are monitored by a CCD camera as a feedback signal. A mirror fixed on a piezo actuator provides a real-time feedback to keep the interference fringes stable.
After stabilization, the relative phase fluctuation in the system is smaller than $0.02\pi$ during data acquisition. Due to the difference of refractive indices between the $\omega$-2$\omega$ fields in air, the phase delay $\tau$ can be tuned with sub-femtosecond accuracy by changing the distance between the BBO and air plasma.

The $\omega$-2$\omega$ beams are respectively focused in two arms of the interferometry with L-1 and L-2 lenses with focal length of 10 cm. The L-2 lens is fixed, and L-1 lens is installed on a translation stage. The L-1 position $d$ can be moved along propagation direction. The $\omega$-2$\omega$ beams are combined with DM-3 and focused into atmospheric air to produce plasma. The plasma profile and THz conversion efficiency highly depend on $d$. And the plasma profile can be recorded by a CCD camera.

The THz strength is measured by two methods, electro-optical sampling (EOS) and pyroelectric detector. In EOS, a 800 $\mu$m thickness silicon wafer blocks the $\omega$ and 2$\omega$ beam.
The THz beam is aligned by two parabolic mirrors and focused on a 1 mm thickness ZnTe crystal. A metal wire-grid polarizer filters out polarized THz beam, and the ZnTe crystal is fixed at the special orientation, which has the same responses for \textit{s}- and \textit{p}-polarized components. The EOS has detection bandwidth of $< 3$ THz and the polarization can be resolved. The THz intensity is also measured with a pyroelectric detector (THZ9B-BL-DZ, GENTEC-EO) with 25 Hz chopper frequency. Here, an 400 $\mu$m thickness silicon wafer is glued on the detector to block scattering light. The THz bandwidth is obtained with a home built Fourier transform spectrometer, where the pyroelectric detector is used as the detector. The wavelength calibration of the spectrometer in mid-infrared region is implemented with optical parametric amplifier.

\subsection{Theoretical Model}

The linearly polarized laser pulse propagation model consisting of the absorption loss of ionization, nonlinear Kerr effect and plasma defocusing is described as a three-dimensional (2D+1) Maxwell's wave equation \cite{Geissler1999}
\begin{equation}
\bigtriangledown ^2 E(r,z,t)-\frac{1}{c^2}\frac{\partial ^2 E(r,z,t)}{\partial t^2}=\mu_0 \frac{\partial J(r,z,t)}{\partial t}+\frac{\omega^2_0}{c^2}(1-\eta ^2_{\mathrm{eff}}) E(r,z,t),
\end{equation}
where $E(r,z,t)$ denotes the transverse electric field at radial $r$ and axial $z$, and $\mu_0$, $\omega_0$ and $c$ are the permeability of vacuum, central frequency of electric field and speed of light in vacuum. In source terms, the absorption loss of ionization is given as \cite{Gaarde08}  $J(r,z,t)=\frac{W(t) n_{e}(t) I_p E(r,z,t)}{\vert E(r,z,t) \vert ^{2}}$, where $I_p$ , $W(t)$ and $n_e(t)$ are the ionization potential, ionization rate and free electron density. The effective refractive index in source term is written as $\eta_{\mathrm{eff}}=\eta_0 +\eta_2 I(r,z,t) - \omega^2_p /2\omega^2_0 $ , where the refraction and absorption of the neutral gas $\eta_0 \sim 1$, the nonlinear Kerr index in atmospheric air $\eta_2=3.2 \times 10^{-19} \mathrm{cm^2/W}$, the plasma defocusing $\omega^2_p / 2\omega^2_0 =n_e /2 n_c  $, $n_c[\mathrm{/cm^3}]\sim 1.1 \times 10^{21}/\lambda^2 [\mathrm{\mu m}]$ is the critical density of the laser with wavelength of $ \lambda $. The electron density of the photoionization induced plasma is calculated with the empirical Ammosov-Delone-Krainov (ADK) formula \cite{Tong05}
\begin{equation}
\frac{d n_e(t)}{dt}=W(t)n_a(t),
\label{eq:IonizationRate}
\end{equation}
where $W(t)$ is the instantaneous tunnel ionization rate. When the first ionization dominates,
the time-dependent neutral density is written as $n_a(t)=n_0-n_e(t)$, $n_0$ is initial neutral gas density. As the major ingredient of air, the nitrogen is used to estimate the ionization rate of air.

The transient photocurrent model \cite{Kre06} is employed to calculate THz generation in the two-color fields $E=E_\omega (r,z,t)+E_{2\omega}(r,z,t)$. The THz generation at a certain position is estimated as $E_{\mathrm{THz}} \propto \frac{dJ(t)}{dt}$. The transient current which is formed by  acceleration of free electrons in two-color fields can be expressed as
\begin{equation}
J(t)=-e\int _ 0 ^ t \upsilon (t^{ \prime}, t) dn_e(t^{ \prime}).
\label{eq:NetCurrent}
\end{equation}
According to Eq. \ref{eq:IonizationRate}, the increment of electron density $dn_e$ depends on ionization rate. The transverse velocity that the electron acquires from $t^{\prime}$ to $t$ is given by
\begin{equation}
\upsilon (t^{\prime} ,t) = -\frac{e}{m}\int_{t^{\prime}}^t E(\xi)d\xi,
\label{eq:NetVelocity}
\end{equation}
where free electrons are assumed to be born with initial velocity $\upsilon(t^{\prime}) = 0$. The transverse velocity depends on the laser waveform between $t^{\prime}$ and $t$.
According to Eq. \ref{eq:NetCurrent}, the residual current at the end of laser pulse can be expressed as $J(t)=-e\int _0 ^ \infty \upsilon_d (t^{ \prime}) dn_e(t^{ \prime})$. Considering Eq. \ref{eq:NetVelocity}, the transverse velocity at the end of laser pulse, i.e., the drift velocity, is written as $\upsilon_d (t^{\prime}) = -\frac{e}{m}\int_{t^{\prime}}^\infty E(\xi)d\xi$. According to integration formula, the drift velocity can be written as $\upsilon_d (t^{\prime}) = -\frac{e}{m}(\int_0 ^\infty E(\xi)d\xi-\int_0 ^{t^{\prime}} E(\xi)d\xi)=-\frac{e}{m}(A(\infty)-A(t^{\prime}))$. The two vector potentials $A(\infty)$ and $A(t^{\prime})$ are determined by asymmetry of the two-color fields.

When THz wave propagates in a dense plasma, the THz emission with frequency $\Omega$ depends on absorption length of the plasma medium. Because the coherence length between $\omega$ and $2\omega$ is far longer than absorption length for gaseous plasma, the absorption effect dominates \cite{Constant1999}.
The THz intensity without the dependence of phase shift is calculated by
\begin{equation}
I_{\mathrm{THz}}=\vert \iint E_{\mathrm{THz}}(\Omega)\exp( -\frac{L-z}{L_a} ) drdz \vert ^2,
\end{equation}
where the $L$ refers to plasma length. The absorption length depends on the electron–ion collisional frequency $\nu$ and the plasma frequency $\omega_p$, which is written as $L_a\sim 2c(\Omega^2+\nu^2)/(\omega^2_p \nu)$. The electron–ion collisional frequency is estimated as $\nu \sim 2.9 \times 10^{-6}Z^2N_i [\mathrm{/cm^3}] \ln{\Lambda _{ei}} (T_{\mathrm{eff}}[\mathrm{eV}])^{-3/2}$,
where $N_i$ is the ion density, $\ln {\Lambda}$ is the Coulomb logarithm, the effective temperature $T_{\mathrm{eff}}=\kappa_{B}T_{e}+2U_p/3$. The ponderomotive potential is written as $U_p=\frac{1}{4}\frac{e^2 E^2_0}{m \omega^2_0} \sim 9.33 \times 10^{-14} I_0 (\mathrm{W/cm^2}) \lambda^2_0 (\mu m) $. In the simulation, the local fluctuation of electron temperature is neglected, and the average temperature $T_e$ is given by an average electron–ion collisional frequency $\tilde{\nu} \sim 3 \times 10^{12}\ \mathrm{Hz}$ in the gaseous plasma when laser intensity is $\sim 100\ \mathrm{TW/cm^2}$. In the inverse bremsstrahlung heating regime, when the heating time scale $\tau^{\ast}=(1/\tilde{\nu})^{3/10} \sim 90\  \mathrm{fs} $  is larger than the
pulse duration $\tau \sim 35\ \mathrm{fs}$, the electron temperature in short-pulse regime \cite{Durfee1995} is estimated by $\kappa_{B}T_{e} \sim 2U_p \tau/(5\tau^{\ast}) \sim 1.7\ \mathrm{eV}$.

\bmhead{Acknowledgments}
The work is supported by National Natural Science Foundation of China (NSFC) (12174284, 11827806, 11874368, 11864037, 91850209). We also acknowledge the support from Shanghai-XFEL beamline project (SBP) and Shanghai High repetition rate XFEL and Extreme light facility (SHINE).