\section{Introduction}
The Internet of Things (IoT) is poised to transform our lives and unleash enormous economic benefit. However, inadequate data security and trust are seriously limiting its adoption. Blockchain, a distributed tamper-resistant database, has native resistance to data tamper by which  it  is  practically impossible to modify the data retroactively once a transaction has been recorded. We believe such tamper-resistance property can be of significant value in ensuring trust in IoT systems.

\; We aim at demonstrating the tamper-resistant capability of Blockchain in securing trust in IoT systems. An IoT testbed was constructed in our lab, and Ethereum based Blockchain was built in the testbed. A number of tamper-resistance experiments have been carried out and analyzed to examine the process of block validation in Blockchain on both full node and light node. Our demonstrations reveal that Blockchain has a dedicated tamper-resistant capability, which can be applied to IoT to ensure trusted data collection and sharing.
With this features used within P2P network, precisely by design, a Blockchain-based database can therefore constitute a trust-free decentralized consensus system. Note that trust-free means conventional \engordnumber{3} party as an arbitral body is filled in by common cryptographical theorems. This promotes Blockchain to be a suitable role for data recording, storage and identity management, especially for those sensitive data [2].

The connection between Blockchain and IoT has no longer been futuristic [4-9]. With the research running, scientists have handed out Blockchain-based IoT.
There is no doubt that Blockchain and IoT are the two hot tags in the field of science and technology. IoT, including sensors, vehicles and other moving objects, basically contains any use of embedded electronic components with the outside world communication equipment, in particular, it makes use of the IP protocol,
as shown in Fig. 1.
\begin{figure}[htbp]
\centering
  \includegraphics[height=2.8cm]{111.pdf}
  \caption{Trusted digital Supply Chain}
  \label{fgr:example}
\end{figure}

\section{Core Concept of Tamper-resistant}
\subsection{Key components}

We now explain the key components of tamper-resistant.

\; First of all, we point out the difference between the so-called 51\% attack and chain reorganization. Nodes will never import a block that fails validation. A 51\% attack consists of a 51\% of the miners forking off from a block in the past and creating a new chain that eventually beats the current canonical chain in total difficulty.

\; A reorganization is done only if a block being imported on a side fork  leads  to  a  higher  total  difficulty  on  that particular fork than the canonical fork. The blocks nonetheless still need to be valid. Note that triggering re-organization function is based on TotalDifficult in Proof-of-Work [2].

\; Synchronization happens all the time across each node starting from an identical genesis block in a chain. Each node runs a validation function to  validate each incoming block. No block will be accepted  unless passing the validation.
\subsection{Validation procedures}
Throughout the validation procedures, the follow conditions are evaluated:
\begin{compactitem}
\item if \emph{StateRoot} $\in$ local levelDB, throw errors;
\item if \emph{ParentBlock} $\notin$ local levelDB, throw errors;
\item if \emph{StateRoot}$|$$_{\emph{ParentBlock}}$  $\notin$ local levelDB, throw errors;
\item if \emph{Validate(Header)} where \{\emph{nonce, difficulty, mixDigest\ldots}\} $\in$ \emph{Header} not passed, throw errors;
\item if \emph{Validate(UncleHeader)} not passed, throw errors;
\item if \emph{Validate(GasUsed)} not passed $||$  \emph{Validate(bloom)} not passed, throw errors;
\item if \emph{TxHash} $\not=$ \emph{Hash(Txs)} $||$ \emph{ReceiptHash} $\not=$ \emph{Hash(Receipts)}, throw errors;
\item if \emph{StateRoot} $\not=$
\emph{StateRoot}$|$ $ \xLongleftarrow{Txs} \emph{CurrentStateRoot}$, throw errors.
\end{compactitem}

  The last validation points out that since there is no transaction sent to those normal nodes, the state root after a state change will never equal to the state root of the header of the incoming block coming from an abnormal node.

A simple fraud of database does not account for any PoW computation. However, not only \emph{difficulty}, \emph{epochDataset} and \emph{mixhash}, but also the \emph{HeaderHash} is involved in calculating the targeting \emph{nonce}. It means arbitrary but sufficient amount of computation should be carried out on the tampered block. Even if sufficient amount of computation is satisfied, the record can be recovered by the canonical chain with the fastest speed effort, unless the hacker has control over 51\% computational power among the whole network.

\section{Tamper-resistance Demonstrations}
\subsection{System setup}
\subsubsection*{Hardware setup (shown in Fig.1)}
\begin{compactitem}
\item two workstations as mining nodes, shown in Fig.1(a);
\item three Raspberry Pi 3 B+, shown in Fig.1(b), attached with IoT sensors as end-point nodes, which are only allowed to look up and upload data without mining.
\begin{figure}[htb]
\centering
\subfloat[]{%
  \includegraphics[scale=0.17]{testbed2.pdf}} \hfill
\subfloat[]{%
  \includegraphics[width=.24\textwidth]{testbed1.pdf}}
  \caption{Overview of testbed}
  \label{fgr:example}
\end{figure}
\end{compactitem}
\vspace{1mm}
\subsubsection*{Software setup}
\begin{compactitem}
\item Ubuntu 14.04 Trusty and Mac OS on mining nodes;
\item Raspbian on Raspberry Pi;
\item Golang Ethereum 1.5.9 for Blockchain [3];
\item Golang on hacking, tampering and logging;
\item Python on data receiving and encapsulation;
\item Javascript on Blockchain processing via web3 API [3].
\end{compactitem}
\vspace{2mm}
\subsubsection*{System Implementation}

\

The Raspberry Pi IoT devices equipped with temperature sensors measure room temperatures in the lab every 30 minutes. An Ethereum based Blockchain is built in the testbed. The IoT measurement data are encapsulated and uploaded to a pre-built contract in the Blockchain.
\subsection{Hacking scenarios and analysis}
We demonstrate several distinct scenarios where blocks are tampered with fake solution to PoW. Scenario analysis and experimental results are presented.

\vspace{1mm}
\noindent\emph{1) \ Non-mining node hacked:}
\

When a non-mining node is hacked, and the total difficulty is smaller than that of a normal block, the canonical chain always chooses the block with larger total difficulty in the context of a valid block. Thus this tampered block will be seen as an uncle block, the canonical chain will be recovered back by those normal blocks from other normal nodes.

\
When the total difficulty is greater than that of a normal block, other normal nodes will not accept the incoming blocks since the tampered block fails to pass the PoW validation during the synchronization. Once a future block whose total difficulty transcends that of the tampered block is generated by one   of the normal mining nodes, this tampered block will be seen as an uncle block, the canonical chain will be recovered back by those normal blocks from other normal nodes.

\vspace{1mm}
\noindent\emph{2) \ Mining node hacked:}
\

When a mining node hacked, and the total difficulty is smaller than that of a normal block, the results are the same as that of the non-mining node hacked case. Once this node starts mining, the uncle block will be broadcast at the same time for validation, contributing to errors throwing on other normal nodes shown as in Fig.2.
\begin{figure}[htbp]
\centering
  \includegraphics[scale=0.28]{222.pdf}
  \caption{Bad Block with tampered uncle blocks}
  \label{fgr:example}
\end{figure}

\
When the total difficulty is greater than that of a normal block, it ends up being insufficient computational power. As a result, this tampered block will not be accepted by any other adversaries, and this mining node will be removed as a bad peer by other normal nodes, shown in Fig.3.
\begin{figure}[htbp]
\centering
  \includegraphics[height=1cm]{333.pdf}
  \caption{Insufficient computation leading to incorrect nonce}
  \label{fgr:example}
\end{figure}

\noindent\emph{3) \ Tampered Block on light node:}
\

We now investigate the scenario that a light node is hacked, and the PoW verification passes and the height of this light node is greater than that of those mining nodes. In this scenario, there will be no suitable peers available for this light node and any transactions  sent  from local will be broadcast to null, until the height is transcended by canonical chain, shown in Fig.4.

\begin{figure}[htb]
\centering
\subfloat[No suitable peer available when fetching data]{%
  \includegraphics[scale=0.29]{hack_light.pdf}}
  \\
\subfloat[No suitable peer available when sending transactions]{%
  \includegraphics[scale=0.3]{hack_morelightsendtx.pdf}}
\end{figure}

\section{A Practical Demonstration}
\; We now demonstrate the effectiveness of the tamper-resistant property of Blockchain in protecting the IoT data records. In our Blockchain secured IoT testbed, one Raspberry Pi device was hacked, its temperature record was changed from the original measure of 34$^\circ$C to -4$^\circ$C.
\par As soon as the tampering happened, the blockchain noticed this anomaly with a broken chain in the hacked node, which signals a tampering action.
\par Next, when the blockchain is synchronized, the abnormal block is  automatically  recovered  back  to the major one through the canonical chain. As a result the tampered record of -4$^\circ$C has been replaced by the original record of 34$^\circ$C. This is the chain reorganization process of the Blockchain. A log is generated to record which content had been changed unexpectedly, and this log is automatically uploaded onto Blockchain for future reference.
\par This demonstrated that the Blockchain can be applied to IoT to secure data records.


\section{acknowledgment}

\begin{description}
\item  The work of O. Z. was supported by International Science Education
Program (ISEP), grant No. QSU082068.
\end{description}


\section{Introduction}

Short text matching (STM) is generally regarded as a task of paraphrase identification or sentence semantic matching.
Given a pair of sentences, the goal of matching models is to predict their semantic similarity.
It is widely used in question answer systems \cite{liu2018improved} and dialogue systems~\cite{gao2019neural,yu2014cognitive}.

Recent years have seen great progress in deep learning methods for text matching \cite{mueller2016siamese,gong2017natural,chen2017enhanced,lan2018neural}.
However, almost all of these models are initially proposed for English text matching. For Chinese language tasks, early work utilizes Chinese characters as input to the model, or first segment each sentence into words, and then take these words as input tokens. Although character-based models can overcome the problem of data sparsity to some degree~\cite{li2019word}, the main drawback is that explicit word information is not fully utilized, which has been demonstrated to be useful for semantic matching \cite{li2019enhancing}.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\columnwidth]{lattice.pdf}
    \caption{An example of word segmentation and the potential word ambiguity.}
    \label{fig:example}
\end{figure}

However, a large number of Chinese words are polysemous, which brings great difficulties to semantic understanding~\cite{xu2016improve}.
Word polysemy in short text is more an issue than that in long text because short text usually has less contextual information, so it is extremely hard for models to capture the correct meaning. As shown in Fig. \ref{fig:example}, the word in red in sentence-1 actually has two meanings: one is to describe bragging (\emph{exaggeration}) and another is \emph{moisture}.
Intuitively, if other words in the context have similar or related meanings, the probability of them will increase.
To integrate semantic information of words, we introduce HowNet~\cite{dong2003hownet} as an external knowledge base. In the view of HowNet, words may have multiple senses/meanings and each sense has several sememes to represent it. For instance, the first sense \emph{exaggeration} indicates some boast information in his words. Therefore, it has sememes {\tt information} and {\tt boast}. Similarly, we can also find the sememe {\tt boast} describing the sense \emph{brag} which belongs to the word ``ChuiNiu (bragging)'' in sentence-2. In this way, model can better determine the sense of words and perceive that two sentences probably have the same meaning.

Furthermore, word-based models often encounter some potential issues caused by word segmentation. If the word segmentation fails to output ``ChuiNiu (bragging)'' in sentence-2, we will lose useful sense information. In Chinese, ``Chui (blowing)'' ``Niu (cattle)'' is a bad segmentation, which deviates the correct meaning of ``ChuiNiu (bragging)''.
To tackle this problem, many researchers propose word lattice graphs~\cite{lai2019lattice,li2020flat,chen2020neural}, where they retain words existing in the word bank so that various segmentation paths are kept. It has been shown that multi-granularity information is important for text matching.

In this paper, we propose a Linguistic knowledge Enhanced graph Transformer (LET) to consider both semantic information and multi-granularity information. LET takes a pair of word lattice graphs as input. Since keeping all possible words will introduce a lot of noise, we use several segmentation paths to form our lattice graph and construct a set of senses according to the word. Based on HowNet, each sense has several sememes to represent it. In the input module, starting from the pre-trained sememe embeddings provided by OpenHowNet~\cite{qi2019openhownet}, we obtain the initial sense representation using a multi-dimensional graph attention transformer (MD-GAT, see Sec. 3.1). Also, we get the initial word representation by aggregating features from the character-level transformer encoder using an Att-Pooling (see Sec. 4.1). Then it is followed by SaGT layers (see Sec. 4.2), which fuse the information between words and semantics. In each layer, we first update the nodes' sense representation and then updates word representation using MD-GAT. As for the sentence matching layer (see Sec. 4.3), we convert word representation to character level and share the message between texts. Moreover, LET can be combined with pre-trained language models, e.g. BERT~\cite{devlin2019bert}. It can be regarded as a method to integrate word and sense information into pre-trained language models during the fine-tuning phase.

 Contributions in this work are summarized as: a) We propose a novel enhanced graph transformer using linguistic knowledge to moderate word ambiguity. b) Empirical study on two Chinese datasets shows that our model outperforms not only typical text matching models but also the pre-trained model BERT as well as some variants of BERT. c) We demonstrate that both semantic information and multi-granularity information are important for text matching modeling, especially on shorter texts.

\section{Related Work}

{\bf Deep Text Matching Models} based on
deep learning have been widely adopted for short text matching. They can fall into two categories: representation-based methods~\cite{he2016text,lai2019lattice} and interaction-based methods~\cite{wang2017bilateral,chen2017enhanced}. Most representation-based methods are based on Siamese architecture, which has two symmetrical networks (e.g. LSTMs and CNNs) to extract high-level features from two sentences. Then, these features are compared to predict text similarity. Interaction-based models incorporate interactions features between all word pairs in two sentences. They generally perform better than representation-based methods. Our proposed method belongs to interaction-based methods.

{\bf Pre-trained Language Models}, e.g. BERT, have shown its powerful performance on various natural language processing (NLP) tasks including text matching. For Chinese text matching, BERT takes a pair of sentences as input and each Chinese character is a separated input token. It has ignored word information. To tackle this problem, some Chinese variants of original BERT have been proposed, e.g. BERT-wwm~\cite{cui2019pre} and ERNIE~\cite{sun2019ernie}. They take the word information into consideration based on the whole word masking mechanism during pre-training. However,
the pre-training process of a word-considered BERT requires a lot of time and resources. Thus, Our model takes pre-trained language model as initialization and utilizes word information to fine-tune the model.

\section{Background}

In this section, we introduce graph attention networks (GATs) and HowNet, which are the basis of our proposed models in the next section.
\subsection{Graph Attention Networks}

Graph neural networks (GNNs)~\cite{scarselli2008graph} are widely applied in various NLP tasks, such as text classifcation~\cite{yao2019graph}, text generation~\cite{zhao2020line}, dialogue policy optimization~\cite{ chen2018structured,chen2018policy,chen2019agentgraph,chen2020distributed} and dialogue state tracking~\cite{chen2020schema,zhu2020efficient}, etc.
GAT is a special type of GNN that operates on graph-structured data with attention mechanisms. Given a graph $G = (\mathcal{V}, \mathcal{E})$, where $\mathcal{V}$ and $\mathcal{E}$ are the set of nodes $x_i$ and the set of edges, respectively. $\mathcal{N}^{+}(x_i)$ is the set including the node $x_i$ itself and the nodes which are directly connected by $x_i$.

Each node $x_i$ in the graph has an initial feature vector $\mathbf{h}_i^0 \in \mathbb{R}^d$, where
$d$ is the feature dimension. The representation of each node is iteratively updated by the graph attention operation. At the $l$-th step, each node $x_i$ aggregates context information by attending over its neighbors and itself. The updated representation $\mathbf{h}^l_i$ is calculated by the weighted average of the connected nodes,
\begin{equation}
\begin{split}
    \mathbf{h}_i^l

    = \sigma\left(\sum_{x_j \in \mathcal{N}^{+}(x_i)} \alpha_{ij}^l \cdot \left( \mathbf{W}^l \mathbf{h}_j^{l-1}\right)\right),
\end{split}
\label{eq:selfatt}
\end{equation}
where $\mathbf{W}^l \in \mathbb{R}^{d \times d}$ is a learnable parameter, and
$\sigma(\cdot)$ is a nonlinear activation function, e.g. ReLU. The attention coefficient $\alpha_{ij}^l$ is the normalized similarity of the embedding between the two nodes $x_i$ and $x_j$ in a unified space, i.e.
\begin{equation}
\begin{split}
    \alpha_{ij}^l &= \mbox{softmax}_{j} \ f^l_{sim}\left( \mathbf{h}^{l-1}_i, \  \mathbf{h}^{l-1}_j \right) \\
    & = \mbox{softmax}_{j}\left(\mathbf{W}^l_{q} \mathbf{h}_i^{l-1}\right)^T\left(\mathbf{W}^l_{k} \mathbf{h}_j^{l-1}\right),
    \label{eq:attention}
\end{split}
\end{equation}
where $\mathbf{W}^l_{q}$ and $\mathbf{W}^l_{k} \in \mathbb{R}^{d \times d}$ are learnable parameters for projections.

Note that, in Eq. (\ref{eq:attention}), $\alpha_{ij}^l$ is a scalar, which means that all dimensions in $\mathbf{h}_j^{l-1}$ are treated equally. This may limit the capacity to model complex dependencies.
Following \citeauthor{shen2018disan}~\shortcite{shen2018disan}, we replace the vanilla attention with multi-dimensional attention.
Instead of computing a single scalar score, for each embedding $\mathbf{h}_j^{l-1}$, it first computes a feature-wise score vector, and then normalizes
it with feature-wised multi-dimensional softmax (MD-softmax),
\begin{equation}
\begin{split}
    \bm{\alpha}_{ij}^l

    = &\mbox{MD-softmax}_j\left( \hat{\alpha}^l_{ij} + f^l_{m}\left(\mathbf{h}_j^{l-1}\right)\right),
\end{split}
\end{equation}
where $\hat{\alpha}_{ij}^l$ is a scalar calculated by the similarity function $f^l_{sim} ( \cdot)$ in Eq. (\ref{eq:attention}), and $f^l_{m}(\cdot)$ is a vector. The addition in above equation means the scalar will be added to every element of the vector. $\hat{\alpha}_{ij}^l$ is utilized to model the pair-wised dependency of two nodes, while $f^l_{m}(\cdot)$ is used to estimate the contribution of each feature dimension of $\mathbf{h}_j^{l-1}$,
\begin{equation}
    f^l_{m}(\mathbf{h}_j^{l-1}) =  \mathbf{W}^l_{2}  \sigma \left(\mathbf{W}^l_{1} \mathbf{h}_j^{l-1} + \mathbf{b}^l_{1} \right) + \mathbf{b}^l_{2},
\end{equation}
where $\mathbf{W}_{1}^l$, $\mathbf{W}_{2}^l$, $\mathbf{b}^l_{1}$ and $\mathbf{b}^l_{2}$ are learnable parameters.
With the score vector $\bm{\alpha}_{ij}^l$, Eq. (\ref{eq:selfatt}) will be accordingly revised as
\begin{equation}
\begin{split}
    \mathbf{h}_i^l

    &= \sigma\left(\sum_{x_j \in \mathcal{N}^{+}(x_i)} \bm{\alpha}_{ij}^l \odot \left( \mathbf{W}^l \mathbf{h}_j^{l-1} \right) \right),
\end{split}
\label{eq:mdatt}
\end{equation}
where $\odot$ represents element-wise product of two vectors.
For brevity, we use $\mbox{MD-GAT}(\cdot)$ to denote the updating process using multi-dimensional attention mechanism, and rewrite Eq. (\ref{eq:mdatt}) as follows,
\begin{equation}
\begin{split}
    \mathbf{h}_i^l = \mbox{MD-GAT}\left(\mathbf{h}_i^{l-1}, \ \left\{ \mathbf{h}^{l-1}_j | x_j \in \mathcal{N}^{+}(x_i) \right\} \right). \\
\end{split}
\label{eq:mdatt_r}
\end{equation}

After $L$ steps of updating, each node will finally have a context-aware representation $\mathbf{h}_i^L$. In order to achieve a stable training process, we also employ a residual connection followed by a layer normalization between two graph attention layers.

\subsection{HowNet}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\columnwidth]{sagt.pdf}
    \caption{An example of the HowNet structure.}
    \label{fig:sense}
\end{figure}

HowNet~\cite{dong2003hownet} is an external knowledge base that manually annotates each Chinese word sense with one or more relevant sememes. The philosophy of HowNet regards sememe as an atomic semantic unit. Different from WordNet~\cite{miller1995wordnet}, it emphasizes that the parts and attributes of a concept can be well represented by sememes. HowNet has been widely utilized in many NLP tasks such as word similarity computation~\cite{liu2002word}, sentiment analysis~\cite{xianghua2013multi}, word representation learning~\cite{niu2017improved} and language modeling~\cite{gu2018language}.

An example is illustrated in Fig. \ref{fig:sense}.
The word ``Apple'' has two senses including \emph{Apple Brand} and \emph{Apple}. The sense \emph{Apple Brand} has five sememes including {\tt computer}, {\tt PatternValue}, {\tt able}, {\tt bring} and {\tt SpecificBrand}, which describe the exact meaning of sense.

\section{Proposed Approach}
\label{sec:let}

\begin{figure}[h]
    \centering
    \includegraphics[width=1\columnwidth]{net.pdf}
    \caption{The framework of our proposed LET model.}
    \label{fig:net}
\end{figure}

First, we define the Chinese short text matching task in a formal way.
Given two Chinese sentences $\mathcal{C}^{a}=\{c_1^a, c_2^a, \cdots, c_{T_a}^a\}$ and $\mathcal{C}^{b}=\{c_1^b, c_2^b, \cdots, c_{T_b}^b\}$, the goal of a text matching model $f(\mathcal{C}^{a},\mathcal{C}^{b})$ is to predict whether the semantic meaning of $\mathcal{C}^{a}$ and $\mathcal{C}^{b}$ is equal. Here, $c_t^a$ and $c_{t'}^b$ represent the $t$-th and $t'$-th Chinese character in two sentences respectively, and $T_a$ and $T_b$ denote the number of characters in the sentences.

In this paper, we propose a linguistic knowledge enhanced matching model. Instead of segmenting each sentence into a word sequence, we use three segmentation tools and keep these segmentation paths to form a word lattice graph $G=(\mathcal{V},\mathcal{E})$ (see Fig. \ref{fig:update} (a)). $\mathcal{V}$ is the set of nodes and $\mathcal{E}$ is the set of edges.
Each node $x_i \in \mathcal{V}$ corresponds to a word $w_i$ which is a character subsequence starting from the $t_1$-th character to the $t_2$-th character in the sentence. As introduced in Sec. 1, we can obtain all senses of a word $w_i$ by retrieving the HowNet.

For two nodes $x_i \in \mathcal{V}$ and $x_j \in \mathcal{V}$,
if $x_i$ is adjacent to $x_j$ in the original sentence, then there is an edge between them.
$\mathcal{N}_{fw}^{+}(x_i)$ is the set including $x_i$ itself and all its reachable nodes in its forward direction, while $\mathcal{N}_{bw}^{+}(x_i)$ is the set including $x_i$ itself and all its reachable nodes in its backward direction.

Thus for each sample, we have two graphs $G^a=(\mathcal{V}^a,\mathcal{E}^a)$ and $G^b=(\mathcal{V}^b,\mathcal{E}^b)$, and our graph matching model is to predict their similarity.
As shown in Fig. \ref{fig:net}, LET consists of four components: input module, semantic-aware graph transformer (SaGT), sentence matching layer and relation classifier. The input module outputs the initial contextual representation for each word $w_i$ and the initial semantic representation for each sense. The semantic-aware graph transformer iteratively updates the word representation and sense representation, and fuses useful information from each other. The sentence matching layer first incorporates word representation into character level, and then matches two character sequences with the bilateral multi-perspective matching mechanism. The relation classifier takes the sentence vectors as input and predicts the relation of two sentences.

\subsection{Input Module}

\subsubsection{Contextual Word Embedding}
\label{sec:cwe}

For each node $x_i$ in graphs, the initial representation of word $w_i$ is the attentive pooling of contextual character representations.
Concretely, we first concat the original character-level sentences to form a new sequence $\mathcal{C} = \{[\text{CLS}], c_1^a, \cdots, c_{T_a}^a, [\text{SEP}], c_1^b, \cdots, c_{T_b}^b, [\text{SEP}]\}$, and then feed them to the BERT model to obtain the contextual representations for each character $\{\mathbf{c}^{\text{CLS}}, \mathbf{c}_1^a$, $\cdots$, $\mathbf{c}_{T_a}^a$, $\mathbf{c}^{\text{SEP}}$, $\mathbf{c}_1^b$, $\cdots$, $\mathbf{c}_{T_b}^b$, $\mathbf{c}^{\text{SEP}}\}$. Assuming that the word $w_i$ consists of some consecutive character tokens $\{c_{t_1}, c_{{t_1}+1}, \cdots, c_{t_2}\}$\footnote{For brevity, the superscript of $c_{k}\ (t_1 \leq k \leq t_2)$ is omitted.}, a feature-wised score vector is calculated with a feed forward network (FFN) with two layers for each character $c_{k} \ (t_1 \leq k \leq t_2)$, and then normalized with a feature-wised  multi-dimensional softmax (MD-softmax),
\begin{equation}
   \mathbf{u}_{k}=\mbox{MD-softmax}_{k}\left( \text{FFN}(\mathbf{c}_{k}) \right),
 \label{eq:attpool-1}
\end{equation}
The corresponding character embedding $\mathbf{c}_{k}$ is weighted with the normalized scores $\mathbf{u}_{k}$ to obtain the contextual word embedding,
\begin{equation}
   \mathbf{v}_i = \sum_{k=t_1}^{t_2} \mathbf{u}_{k} \odot  \mathbf{c}_{k},
\label{eq:attpool-2}
\end{equation}
For brevity, we use $\mbox{Att-Pooling}(\cdot)$ to rewrite Eq. (\ref{eq:attpool-1}) and Eq. (\ref{eq:attpool-2}) for short, i.e.
\begin{equation}
\mathbf{v}_i=\mbox{Att-Pooling}\left(\{\mathbf{c}_k | t_1 \leq k \leq t_2\}\right).
\end{equation}

\subsubsection{Sense Embedding}

The word embedding $\mathbf{v}_i$ described in Sec. \ref{sec:cwe} contains only contextual character information, which may suffer from the issue of polysemy in Chinese. In this paper, we incorporate HowNet as an external knowledge base to express the semantic information of words.

For each word $w_i$, we denote the set of senses as $\mathcal{S}^{(w_i)}=\{s_{i,1}, s_{i,2}, \cdots, s_{i,K}\}$. $s_{i,k}$ is the $k$-th sense of $w_i$ and we denote its corresponding sememes as $\mathcal{O}^{(s_{i,k})}=\{o_{i,k}^1, o_{i,k}^2, \cdots, o_{i,k}^M\}$.
In order to get the embedding $\mathbf{s}_{i,k}$ for each sense $s_{i,k}$, we first obtain the representation $\mathbf{o}_{i,k}^m$ for each sememe $o_{i,k}^m$ with  multi-dimensional attention function,
\begin{equation}
\begin{split}
    \mathbf{o}^{m}_{i,k} &= \mbox{MD-GAT}\left(\mathbf{e}_{i,k}^{m}, \ \left\{\mathbf{e}_{i,k}^{m'} | o_{i,k}^{m'} \in \mathcal{O}^{(s_{i,k})} \right\} \right), \\
\end{split}
\end{equation}
where $\mathbf{e}_{i,k}^{m}$ is the embedding vector for sememe $o_{i,k}^m$ produced through
the Sememe Attention over Target model (SAT) \cite{niu2017improved}. Then, for each sense $s_{i,k}$, its embedding $\mathbf{s}_{i,k}$ is obtained with attentive pooling of all sememe representations,
\begin{equation}
    \mathbf{s}_{i,k} = \mbox{Att-Pooling}\left(\left\{ \mathbf{o}^{m}_{i,k}  |  o_{i,k}^{m} \in \mathcal{O}^{(s_{i,k})} \right\} \right).
\end{equation}

\subsection{Semantic-aware Graph Transformer}

\begin{figure}[t]
    \centering
    \includegraphics[width=1\columnwidth]{update.pdf}
    \caption{(a) is an example of lattice graph. (b) shows the process of sense updating. $\text{fw}_2$ and $\text{bw}_2$ refer to the words in forward and backward directions of $\text{w}_2$ respectively. $\text{uw}_2$ is the words that $\text{w}_2$ cannot reach. (c) is word updating; we will not update the corresponding word representation if the word is not in HowNet.}
    \label{fig:update}
\end{figure}

For each node $x_i$ in the graph, the word embedding $\mathbf{v}_i$ only contains the contextual information while the sense embedding $\mathbf{s}_{i,k}$ only contains linguistic knowledge. In order to harvest useful information from each other, we propose a semantic-aware graph transformer (SaGT). It first takes $\mathbf{v}_i$ and $\mathbf{s}_{i,k}$ as initial word representation $\mathbf{h}^0_i$ for word $w_i$ and initial sense representation $\mathbf{g}^0_{i,k}$ for sense $s_{i,k}$ respectively, and then iteratively updates them with two sub-steps.

\subsubsection{Updating Sense Representation} At $l$-th iteration, the first sub-step is to update sense representation from $\mathbf{g}^{l-1}_{i,k}$ to $\mathbf{g}^l_{i,k}$. For a word with multiple senses, which sense should be used is usually determined by the context in the sentence. Therefore, when updating the representation, each sense will first aggregate useful information from words in forward and backward directions of $x_i$,
\begin{equation}
\begin{split}
    \mathbf{m}^{l,fw}_{i,k} &= \mbox{MD-GAT}\left(\mathbf{g}_{i,k}^{l-1}, \ \left\{\mathbf{h}_j^{l-1} | x_j \in \mathcal{N}_{fw}^{+}(x_i) \right\} \right), \\
    \mathbf{m}^{l,bw}_{i,k} &= \mbox{MD-GAT}\left(\mathbf{g}_{i,k}^{l-1}, \ \left\{\mathbf{h}_j^{l-1} | x_j \in \mathcal{N}_{bw}^{+}(x_i) \right\} \right), \\
\end{split}
\end{equation}
where two multi-dimensional attention functions $\mbox{MD-GAT}(\cdot)$ have different parameters. Based on $\mathbf{m}^l_{i,k}=[\mathbf{m}^{l,fw}_{i,k}, \mathbf{m}^{l,bw}_{i,k}]$~\footnote{$[\cdot,\cdot]$ denotes the concatenation of vectors.}, each sense updates its representation with a gate recurrent unit (GRU) \cite{cho2014learning},
\begin{equation}
    \mathbf{g}^l_{i,k} = \mbox{GRU}\left( \mathbf{g}_{i,k}^{l-1}, \ \mathbf{m}^l_{i,k}\right).
\label{eq:gru_sense}
\end{equation}

It is notable that we don't directly use $\mathbf{m}_{i,k}^l$ as the new representation $\mathbf{g}_{i,k}^{l}$ of sense $s_{i,k}$.
The reason is that $\mathbf{m}_{i,k}^l$ only contains contextual information, and we need to utilize a gate, e.g. GRU, to control the fusion of contextual information and semantic information.

\subsubsection{Updating Word Representation} The second sub-step is to update the word representation from $\mathbf{h}^{l-1}_{i}$ to $\mathbf{h}^{l}_{i}$ based on the updated sense representations $\mathbf{g}_{i,k}^l \ (1\leq k \leq K)$. The word $w_i$ first obtains semantic information from its sense representations with the multi-dimensional attention,
\begin{equation}
\begin{split}
    \mathbf{q}^{l}_{i} = \mbox{MD-GAT}\left(\mathbf{h}_{i}^{l-1}, \ \left\{\mathbf{g}^{l}_{i,k} | s_{i,k} \in \mathcal{S}^{(w_i)} \right\} \right),
\end{split}
\end{equation}
and then updates its representation with a GRU:
\begin{equation}
    \mathbf{h}^l_{i} = \mbox{GRU}\left( \mathbf{h}_{i}^{l-1}, \ \mathbf{q}^l_{i}\right).
\end{equation}
The above GRU function and the GRU function in Eq. (\ref{eq:gru_sense}) have different parameters.

After multiple iterations, the final word representation $\mathbf{h}_{i}^L$ contains not only contextual word information but also semantic knowledge. For each sentence, we use $\mathbf{h}_{i}^a$ and $\mathbf{h}_{i}^b$ to denote the final word representation respectively.

\subsection{Sentence Matching Layer}

After obtaining the semantic knowledge enhanced word representation $\mathbf{h}_{i}^{a}$ and $\mathbf{h}_{i}^{b}$ for each sentence, we incorporate this word information into characters. Without loss of generality, we will use characters in sentence $\mathcal{C}^a$ to introduce the process. For each character $c_t^a$, we obtain $\mathbf{\hat{c}}^a_t$ by pooling the useful word information,

\begin{equation}
    \mathbf{\hat{c}}^a_t = \mbox{Att-Pooling}\left(\left\{ \mathbf{h}^{a}_{i}  |  w_i^a \in \mathcal{W}^{(c_t^a)} \right\} \right),
\end{equation}
where $\mathcal{W}^{(c_t^a)}$ is a set including words which contain the character $c_t^a$. The semantic knowledge enhanced character representation $\mathbf{y}_t$ is therefore obtained by
\begin{equation}
    \mathbf{y}_t^a = \mbox{LayerNorm}\left( \mathbf{c}_t^a + \mathbf{\hat{c}}^a_t \right),
\end{equation}
where $\mbox{LayerNorm}(\cdot)$ denotes layer normalization, and $\mathbf{c}_t^a$ is the contextual character representation obtained using BERT described in Sec. \ref{sec:cwe}.

For each character $c_t^a$, it aggregates information from sentence $\mathcal{C}^a$ and $\mathcal{C}^b$ respectively using multi-dimensional attention,
\begin{equation}
\begin{split}
    \mathbf{m}^{self}_{t} &= \mbox{MD-GAT} \left(\mathbf{y}_{t}^a, \ \{\mathbf{y}_{t'}^{a} | c_{t'}^a \in \mathcal{C}^a \} \right), \\
    \mathbf{m}^{cross}_{t} &= \mbox{MD-GAT} \ (\mathbf{y}_{t}^a, \ \{\mathbf{y}_{t'}^{b} | c_{t'}^b \in \mathcal{C}^b \} ) \ . \\
\end{split}
\end{equation}
The above multi-dimensional attention functions $\mbox{MD-GAT}(\cdot)$ share same parameters. With this sharing mechanism, the model has a nice property that, when the two sentences are perfectly matched, we have $\mathbf{m}_t^{self} \approx \mathbf{m}_t^{cross}$.

 We utilize the multi-perspective cosine distance \cite{wang2017bilateral} to compare $\mathbf{m}_t^{self}$ and $\mathbf{m}_t^{cross}$,
\begin{equation}
    d_k = \text{cosine}\left( \mathbf{w}_k^{cos} \odot \mathbf{m}_t^{self}, \mathbf{w}_k^{cos} \odot \mathbf{m}_t^{cross} \right),
\end{equation}
where $k \in \{1,2,\cdots,P\}$ ($P$ is number of perspectives).  $\mathbf{w}_k^{cos}$ is a parameter vector, which assigns different weights to different dimensions of messages. With $P$ distances $d_1, d_2, \cdots, d_P$,
we can obtain the final character representation,
\begin{equation}
   \mathbf{\hat{y}}_t^a = \text{FFN} \left(\left[\mathbf{m}_t^{self}, \mathbf{d}_t \right] \right),
\label{eq:ue}
\end{equation}
where $\mathbf{d}_t \triangleq [d_1, d_2, \cdots, d_P]$, and $\mbox{FFN}(\cdot) $ is a feed forward network with two layers.

Similarly, we can obtain the final character representation $\mathbf{\hat{y}}_t^b$ for each character $c_t^b$ in sentence $\mathcal{C}^b$.
Note that the final character representation contains three kinds of information: contextual information, word and sense knowledge, and character-level similarity. For each sentence $\mathcal{C}^a$ or $\mathcal{C}^b$,
the sentence representation vector $\mathbf{r}^a$ or $\mathbf{r}^b$ is obtained using the attentive pooling of all final character representations for the sentence.

\subsection{Relation Classifier}

With two sentence vectors $\mathbf{r}^a$, $\mathbf{r}^b$, and the vector $\mathbf{c}^{\text{CLS}}$ obtained with BERT,  our model will predict the similarity of two sentences,
\begin{equation}
    p = \text{FFN} \left(\left[\mathbf{c}^{\text{CLS}}, \mathbf{r}^a, \mathbf{r}^b, \mathbf{r}^a \odot \mathbf{r}^b  , |\mathbf{r}^a - \mathbf{r}^b| \right]  \right),
\end{equation}
where $\mbox{FFN}(\cdot)$ is a feed forward network with two hidden layers and a sigmoid activation after output layer.

With $N$ training samples $\{\mathcal{C}^{a}_i, \mathcal{C}^{b}_i, y_i \}_{i=1}^N$, the training object is to minimize the binary cross-entropy loss,
\begin{equation}
    \mathcal{L} = - \sum_{i=1}^N \left( y_i \text{log}\left(p_i\right) + \left( 1 - y_i\right) \text{log}\left(1- p_i\right)  \right),
\end{equation}
where $y_i \in \{0,1\}$ is the label of the $i$-th training sample and $p_i \in[0,1]$ is the prediction of our model taking the sentence pair $\{\mathcal{C}^{a}_i, \mathcal{C}_i^{b}\}$ as input.

\section{Experiments}

\subsection{Experimental Setup}
\begin{table*
\renewcommand\arraystretch{1.1}
\centering{
\begin{tabular}{lcccccc}
\specialrule{0.1em}{1pt}{1pt}
\multirow{2}{*}{\textbf{Models}} & \multirow{2}{*}{\textbf{Pre-Training}}  & \multirow{2}{*}{\textbf{Interaction}} & \multicolumn{2}{c}{\textbf{BQ}} &  \multicolumn{2}{c}{\textbf{LCQMC}} \\

\cline{4-7}

 & & & \textbf{ACC.} & \textbf{F1} & \textbf{ACC.} & \textbf{F1} \\
\specialrule{0.1em}{1pt}{1pt}
Text-CNN\cite{he2016text} & $\times$ & $\times$ & 68.52  & 69.17 &  72.80 & 75.70 \\
BiLSTM\cite{mueller2016siamese} & $\times$ & $\times$ & 73.51 & 72.68 & 76.10 & 78.90 \\
Lattice-CNN \cite{lai2019lattice} & $\times$ & $\times$ & 78.20 & 78.30 & 82.14   & 82.41 \\

BiMPM \cite{wang2017bilateral} & $\times$ & $\surd$ & 81.85 & 81.73 & 83.30 & 84.90 \\
ESIM \cite{chen2017enhanced}  & $\times$ & $\surd$ & 81.93 & 81.87 & 82.58 & 84.49 \\
\textbf{LET} (Ours) & $\times$ & $\surd$ & \textbf{83.22} & \textbf{83.03} & \textbf{84.81} & \textbf{86.08} \\
\specialrule{0.1em}{1pt}{1pt}
BERT-wwm \cite{cui2019pre} & $\surd$ & $\surd$ & 84.89 & 84.29 & 86.80 & 87.78 \\
BERT-wwm-ext \cite{cui2019pre} & $\surd$ & $\surd$ & 84.71 & 83.94 & 86.68 & 87.71 \\
ERNIE \cite{sun2019ernie} & $\surd$ & $\surd$ & 84.67 & 84.20 & 87.04 & 88.06 \\\specialrule{0.0em}{0pt}{0pt}
BERT\cite{devlin2019bert} & $\surd$ & $\surd$ & 84.50 & 84.00 & 85.73 & 86.86 \\
\textbf{LET-BERT} (Ours) & $\surd$ & $\surd$ & \textbf{85.30}  & \textbf{84.98} & \textbf{88.38} & \textbf{88.85} \\
\specialrule{0.1em}{1pt}{1pt}
\end{tabular}
\caption{Performance of various models on LCQMC and BQ test datasets. The results are average scores using 5 different seeds. All the improvements over baselines are statistically significant ($p < 0.05$).}
\label{tab:main-res}
}
\end{table*}

\subsubsection{Dataset} We conduct experiments on two Chinese short text matching datasets: LCQMC \cite{liu2018lcqmc} and BQ \cite{chen2018bq}.

LCQMC is a question matching corpus with large-scale open domain. It consists of 260068 Chinese sentence pairs including 238766 training samples, 8802 development samples and 12500 test samples. Each pair is associated with a binary label indicating whether two sentences have the same meaning or share the same intention. Positive samples are 30\% more than negative samples.

BQ is a domain-specific large-scale corpus for bank question matching. It consists of 120000 Chinese sentence pairs including 100000 training samples, 10000 development samples and 10000 test samples. Each pair is also associated with a binary label indicating whether two sentences have the same meaning. The number of positive and negative samples are the same.

\subsubsection{Evaluation metrics}
For each dataset, the accuracy (ACC.) and F1 score are used as the evaluation metrics. ACC. is the percentage of correctly classified examples. F1 score of matching is the harmonic mean of the precision and recall.

\subsubsection{Hyper-parameters}
The input word lattice graphs are produced by the combination of three segmentation tools: jieba~\cite{sun2012jieba}, pkuseg~\cite{pkuseg} and thulac~\cite{li2009punctuation}. We use the pre-trained sememe embedding provided by OpenHowNet~\cite{qi2019openhownet} with 200 dimensions. The number of graph updating steps/layers $L$ is 2 on both datasets, and the number of perspectives $P$ is 20. The dimensions of both word and sense representation are 128. The hidden size is also 128. The dropout rate for all hidden layers is 0.2.
The model is trained by RMSProp with an initial learning rate of 0.0005 and a warmup rate of 0.1. The learning rate of BERT layer is multiplied by an additional factor of 0.1. As for batch size, we use 32 for LCQMC and 64 for BQ. \footnote{Our code is available at \url{https://github.com/lbe0613/LET}.}

\subsection{Main Results}

We compare our models with three types of baselines: representation-based models, interaction-based models and BERT-based models. The results are summarized in Table \ref{tab:main-res}. All the experiments in Table \ref{tab:main-res} and Table \ref{tab:ablation} are running five times using different seeds and we report the \textbf{average} scores to ensure the reliability of results. For the baselines, we run them ourselves using the parameters mentioned in \citet{cui2019pre}.

\textbf{Representation-based models} include three baselines Text-CNN, BiLSTM and Lattice-CNN. Text-CNN~\cite{he2016text} is one type of Siamese architecture with Convolutional Neural Networks (CNNs) used for encoding each sentence. BiLSTM~\cite{mueller2016siamese} is another type of Siamese architecture with Bi-directional Long Short Term Memory (BiLSTM) used for encoding each sentence. Lattice-CNN~\cite{lai2019lattice} is also proposed to deal with the potential issue of Chinese word segmentation. It takes word lattice as input and pooling mechanisms are utilized to merge the feature vectors produced by multiple CNN kernels over different $n$-gram contexts of each node in the lattice graph.

\textbf{Interaction-based models} include two baselines: BiMPM and ESIM. BiMPM~\cite{wang2017bilateral} is a bilateral multi-perspective matching model. It encodes each sentence with BiLSTM, and matches two sentences from multi-perspectives. BiMPM performs very well on some natural language inference (NLI) tasks. There are two BiLSTMs in ESIM~\cite{chen2017enhanced}. The first one is to encode sentences, and the other is to fuse the word alignment information between two sentences. ESIM achieves state-of-the-art results on various matching tasks.
In order to be comparable with the above models, we also employ a model where BERT  in Fig. \ref{fig:net} is replaced by a traditional character-level transformer encoder, which is denoted as LET.

The results of the above models are shown in the first part of Table \ref{tab:main-res}. We can find that our model LET outperforms all baselines on both datasets. More specifically, the performance of LET is better than that of Lattice-CNN. Although they both utilize word lattices, Lattice-CNN only focuses on local information while our model can utilize global information. Besides, our model incorporates semantic messages between sentences, which significantly improves model performance.
As for interaction-based models, although they both use the multi-perspective matching mechanism, LET outperforms BiMPM and ESIM. It shows the utilization of word lattice with our graph neural networks is powerful.

\textbf{BERT-based models} include four baselines: BERT, BERT-wwm, BERT-wwm-ext and ERNIE. We compare them with our presented model LET-BERT. BERT is the official Chinese BERT model released by Google. BERT-wwm is a Chinese BERT with whole word masking mechanism used during pre-training. BERT-wwm-ext is a variant of BERT-wwm with more training data and training steps. ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which include entity-level masking and phrase-level masking. LET-BERT is our proposed LET model where BERT is used as a character level encoder.

The results are shown in the second part of Table~\ref{tab:main-res}. We can find that the three variants of BERT (BERT-wwm, BERT-wwn-ext, ERNIE) all surpass the original BERT, which suggests using word level information during pre-training is important for Chinese matching tasks. Our model LET-BERT performs better than all these BERT-based models. Compared with the baseline BERT which has the same initialization parameters, the ACC. of LET-BERT on BQ and LCQMC is increased by 0.8\% and 2.65\%, respectively.  It shows that utilizing sense information during fine-tuning phrases with LET is an effective way to boost the performance of BERT for Chinese semantic matching.

We also compare results with K-BERT~\cite{liu2020k}, which regards information in HowNet as triples \{word, contain, sememes\} to enhance BERT, introducing soft position and visible matrix during the fine-tuning and inferring phases. The reported ACC. for the LCQMC test set of K-BERT is 86.9\%. Our LET-BERT is 1.48\% better than that. Different from K-BERT, we focus on fusing useful information between word and sense.

\subsection{Analysis}

\begin{table}[t]

\centering
\setlength{\tabcolsep}{5mm}{\begin{tabular}{lccc}
\specialrule{0.1em}{1pt}{1pt}
\textbf{Seg.}    & \textbf{Sense}    & \textbf{ACC.} & \textbf{F1} \\
\specialrule{0.1em}{1pt}{1pt}
jieba   & $\surd$  &  87.84    &  88.47  \\
pkuseg  & $\surd$  &  87.72    &  88.40  \\
thulac  & $\surd$  &  87.50    &  88.27  \\
lattice & $\surd$  & \textbf{88.38}    &  \textbf{88.85} \\
lattice & $\times$ &  87.68    &  88.40  \\
\specialrule{0.1em}{1pt}{1pt}
\end{tabular}
\caption{Performance of using different segmentation on LCQMC test dataset.}
\label{tab:ablation}}
\end{table}

In our view, both multi-granularity information and semantic information are important for LET. If the segmentation does not contain the correct word, our semantic information will not exert the most significant advantage.

Firstly, to explore the impact of using different segmentation inputs, we carry out experiments using LET-BERT on LCQMC test set. As shown in Table \ref{tab:ablation}, when incorporating sense information, improvement can be observed between lattice-based model (the fourth row) and word-based models: jieba, pkuseg and thulac. The improvements of lattice with sense over other models in Table \ref{tab:ablation} are all statistically significant ($p < 0.05$). The possible reason is that lattice-based models can reduce word segmentation errors to make predictions more accurate.

Secondly, we design an experiment to demonstrate the effectiveness of incorporating HowNet to express the semantic information of words. In the comparative model without HowNet knowledge, the sense updating module in SaGT is removed, and we update word representation only by a multi-dimensional self-attention. The last two rows in Table \ref{tab:ablation} list the results of combined segmentation (lattice) with and without sense information. The performance of integrating sense information is better than using only word representation. More specifically, the average absolute improvement in ACC. and F1 scores are 0.7\% and 0.45\%, respectively, which indicates that LET has the ability to obtain semantic information from HowNet to improve the model's performance.
Besides, compared with using a single word segmentation tool, semantic information performs better on lattice-based model. The probable reason is lattice-based model incorporates more possible words so that it can perceive more meanings.

We also study the role of GRU in SaGT. The ACC. of removing GRU in lattice-based model is 87.82\% on average, demonstrating that GRU can control historical messages and combine them with current information. Through experiments, we find that the model with 2 layers of SaGT achieves the best. It indicates multiple information fusion will refine the message and make the model more robust.

\subsubsection{Influences of text length on performance}

As listed in Table \ref{tab:length}, we can observe that text length also has great impacts on text matching prediction. The experimental results show that the shorter the text length, the more obvious the improvement effect of utilizing sense information. The reason is, on the one hand, concise texts usually have rare contextual information, which is difficult for model to understand. However, HowNet brings a lot of useful external information to these weak-context short texts. Therefore, it is easier to perceive the similarity between texts and gain great improvement.
 On the other hand, longer texts may contain more wrong words caused by insufficient segmentation, leading to incorrect sense information. Too much incorrect sense information may confuse the model and make it unable to obtain the original semantics.

\begin{table}[t]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{ccccc}
\specialrule{0.1em}{1pt}{1pt}
\multirow{2}{*}{\textbf{text length}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}number of \\ samples\end{tabular}}} & \multicolumn{2}{c}{\textbf{ACC.}}&\multirow{2}{*}{\textbf{RER(\%)}} \\ \cline{3-4}
                           &                                           & w/o sense     & sense  &   \\ \specialrule{0.1em}{1pt}{1pt}
     $<16$                        &        2793                          &       88.99                        &    90.05     &\textbf{9.63} \\\specialrule{0em}{1pt}{1pt}
          $16-18$                    &        3035                        &    88.49                       &     89.25    &6.60 \\\specialrule{0em}{1pt}{1pt}
           $19-22$                   &      3667                              &       88.58                 &    89.04    &4.03 \\\specialrule{0em}{1pt}{1pt}

        $>22$                   &    3005  &     84.53        &       85.13 &3.88\\\specialrule{0.1em}{1pt}{1pt}
\end{tabular}
}
\caption{Influences of text length on LCQMC test dataset. Relative error reduction (RER) is calculated by $\frac{\text{sense} - \text{w/o sense}}{100 - \text{w/o sense}} \times 100\%$.}
\label{tab:length}
\end{table}

\subsubsection{Case study}
We compare LET-BERT between the model with and without sense information (see Fig.~\ref{fig:case}). The model without sense fails to judge the relationship between sentences which actually have the same intention, but LET-BERT performs well. We observe that both sentences contain the word  ``yuba'', which has only one sense described by sememe {\tt food}. Also, the sense of ``cook'' has a similar sememe {\tt edible} narrowing the distance between texts. Moreover, the third sense of ``fry'' shares the same sememe {\tt cook} with the word ``cook''. It provides a powerful message that makes ``fry'' attend more to the third sense.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\columnwidth]{case.pdf}
    \caption{An example of using sense information to get the correct answer.}
    \label{fig:case}

\end{figure}

\section{Conclusion}
In this work, we proposed a novel linguistic knowledge enhanced graph transformer for Chinese short text matching. Our model takes two word lattice graphs as input and integrates sense information from HowNet to moderate word ambiguity. The proposed method is evaluated on two Chinese benchmark datasets and obtains the best performance. The ablation studies also demonstrate that both semantic information and multi-granularity information are important for text matching modeling.

\section{Acknowledgments}
We thank the anonymous reviewers for their thoughtful comments.  This work has been supported by No. SKLMCPTS2020003 Project.


\section{Introduction}

Bright squeezed vacuum (BSV) is a state of light emerging from the output of a high-gain unseeded parametric amplifier (OPA). Due to its nonclassical properties such as photon-number entanglement and quadrature squeezing, this state  is useful for various quantum-information applications, among them quantum metrology~\cite{metrology}, quantum imaging~\cite{q-imaging}, and quantum lithography~\cite{lithography}. Besides containing a high number of photons in each mode, the state is essentially multimode, both in the frequency and in the angle. These features provide its large information capacity, as quantum information can be encoded in the number of photons in different modes. At the same time, the presence of a large number of modes can be a disadvantage in certain experiments, for instance achieving phase super-sensitivity~\cite{super} or (related) gravitational-wave detection~\cite{gravit}. A possible way to reduce the number of modes without losing nonclassical correlations is to use a nonlinear interferometer, in which only part of the spectrum is amplified. This has been already demonstrated for the angular spectrum in Ref.~\cite{separation}. The goal of this work is to show similar behavior in the frequency domain.

The paper is organized as follows. In the next two subsections, we briefly describe the mode structure of BSV (subsection~\ref{S1.1}) and the idea and operation of a nonlinear interferometer (subsection~\ref{S1.2}). Section~\ref{S2} explains the idea of reducing the number of modes in BSV in space/angle and time/frequency and demonstrates the narrowing of the BSV angular spectrum in a nonlinear interferometer with spatially separated crystals. The experiment on the narrowing of the BSV frequency spectrum is described in Section~\ref{S3}. Section~\ref{S4} contains the conclusions.

\subsection{BSV and its eigenmodes}\label{S1.1}

The most convenient way of generating BSV is high-gain parametric down-conversion in a nonlinear crystal, which can be considered as an unseeded traveling-wave OPA. The frequency-angular spectrum and photon-number correlations are well described by the Bloch-Messiah formalism~\cite{Wasilewski,Silberhorn,Bloch}, in which the Hamiltonian is diagonalized by passing to the eigenmodes of the OPA. For instance, in the case of spatially multimode PDC, the Hamiltonian can be written as~\cite{Bloch}
\begin{equation}
H=i\hbar\Gamma\iint d \mathbf{q}_s  d \mathbf{q}_i F(\mathbf{q}_s,\mathbf{q}_i)a^\dagger_{\mathbf{q}_s}a^\dagger_{\mathbf{q}_i}+h.c.,
\label{eq:Ham}
\end{equation}
where $\Gamma$ characterizes the coupling strength, $\mathbf{q}_{s,i}$ are the transverse wavevectors of the signal and idler radiation, and $a^\dagger_{\mathbf{q}_{s,i}}$ are the photon creation operators in the corresponding plane-wave modes. The central part of the Hamiltonian is the two-photon amplitude (TPA), $F(\mathbf{q}_s,\mathbf{q}_i)$, whose meaning is the probability amplitude of a photon pair created with the wavevectors $\mathbf{q}_s,\mathbf{q}_i$. The Hamiltonian (\ref{eq:Ham}) is diagonalized by representing the TPA as a Schmidt decomposition,
\begin{equation}
F(\mathbf{q}_s,\mathbf{q}_i)=\sum_k \sqrt{\lambda_k}u_k(\mathbf{q}_s)v_k(\mathbf{q}_i),
\label{eq:TPA Schmidt}
\end{equation}
where $\lambda_k$ are the Schmidt eigenvalues, $u_k(\mathbf{q}_s),\,v_k(\mathbf{q}_i)$ the Schmidt modes, and $k$ is a two-dimensional index. By definition, the modes are ordered so that $\lambda_{k+1}\le\lambda_k$. The Hamiltonian (\ref{eq:Ham}) can be now written as a sum of two-mode Hamiltonians,
\begin{equation}
H=\sum_k \sqrt{\lambda_k}H_k,\,\,H_k=i\hbar\Gamma A^\dagger_kB^\dagger_k+h.c.,
\label{eq:BM_dec}
\end{equation}
with the photon creation operators $A^\dagger_k,\,B^\dagger_k$ relating to the Schmidt modes. Moreover, if the signal and idler beams are indistinguishable, their Schmidt modes are the same, and
\begin{equation}
H_k=i\hbar\Gamma (A^\dagger_k)^2+h.c.
\label{eq:Ham_deg}
\end{equation}
It is worth mentioning that the Schmidt decomposition can be alternatively performed in the space coordinates, which is equivalent to the wavevector decomposition~(\ref{eq:TPA Schmidt}). A similar decomposition is valid for the frequency/temporal domain. In different works, the Schmidt modes are also called nonmonochromatic modes~\cite{Opatrny}, squeezing (eigen)modes~\cite{Boyd}, broadband modes~\cite{Silberhorn}, or supermodes~\cite{Fabre}.

Clearly, in terms of these new modes, photon-number correlations are only pairwise. The total mean photon number can be represented as a sum of incoherent contributions from all Schmidt modes,
\begin{equation}
\langle N\rangle=\sum_k \langle N_k\rangle,\,\,\langle N_k\rangle=\sinh^2[\sqrt{\lambda_k}G],
\label{eq:ph_N0}
\end{equation}
where $G=\int\Gamma dt$ is the parametric gain. This means that while at low gain ($G<<1$), the Schmidt modes are populated with the weights given by the Schmidt eigenvalues $\lambda_k$, at high gain these weights are changed to become~\cite{Bloch}
\begin{equation}
 \tilde{\lambda}_k=\frac{\sinh^2[\sqrt{\lambda_k}G]}{\sum_k\sinh^2[\sqrt{\lambda_k}G]}.
\label{eq:renorm}
\end{equation}
According to this, at high gain the lower-order Schmidt modes, initially having higher eigenvalues, become more pronounced.

\subsection{SU(1,1) interferometers}\label{S1.2}

At the very start of nonlinear optics, an idea emerged to realize two nonlinear effects at spatially separated points and to see interference between them. Such \textit{nonlinear interference} would enable the observation of the relative phase between two effects. It was first realized by Chang et al.~\cite{Bloembergen} who measured in this way the complex values of surface quadratic susceptibility for several semiconductors. Later, it became a common way to measure the phases of nonlinear susceptibilities.

After the discovery of parametric amplification via PDC and four-wave mixing (FWM), it was soon suggested to realize nonlinear interference based on these effects.  Yurke et al.~\cite{SU11} proposed an interferometer in which the signal and idler beams emitted in the first parametric amplifier were directed into the second one and got amplified or deamplified depending on the phases introduced in the pump, signal, or idler beams, $\phi_{p,s,i}$ (Fig.~\ref{nonlinear_int}). Because the transformations performed by the interferometer on the fields at its two output modes relate to the SU(1,1) group, this type of interferometer is usually referred to as SU(1,1).
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.5\textwidth]{nonlinear_int.eps}
\caption{(Color online) An SU(1,1) nonlinear interferometer based on two high-gain parametric amplifiers.}\label{nonlinear_int}
\end{center}
\end{figure}
Initially the SU(1,1) interferometer was proposed as a method to perform phase measurement below the shot-noise level, which is possible due to the strong dependence of its output intensity on the phases $\phi_{s,i}$ at high parametric gain~\cite{Ou}. Nevertheless, the first implementations of SU(1,1) interferometers were based on low-gain (spontaneous) PDC~\cite{Herzog,big}. In particular, in a clever modification of such interferometer the effect of `induced coherence without induced emission' was observed~\cite{induced}, which later was successfully implemented for the measurement of absorption~\cite{abs} and dispersion~\cite{disp}, as well as imaging~\cite{imaging} in the infrared or terahertz~\cite{THz} spectral ranges without the detection of infrared or terahertz radiation.

Only recently, the SU(1,1) interferometer using FWM in rubidium vapor has been implemented for overcoming the shot-noise level of phase measurement~\cite{Zhang}. More than $4$ dB improvement has been obtained compared to a conventional (SU(2)) interferometer populated with the same mean number of photons. The operation was at high gain, which provided $7.4$ dB amplification of the radiation from the first FWM source in the second one.

The same mechanism can be used for shaping the spectrum of high-gain PDC or FWM, both in the angle and in the frequency. Because of the nonlinear amplification of the incident radiation, the modes that are not amplified in the second nonlinear crystal will be much weaker at the output than those amplified. Moreover, one can take advantage of the de-amplification of certain modes, which, however, is not accompanied by a noise increase. Such selective amplification of different modes can enable tailoring the structure of the spectrum.

\section{Diffractive and dispersive spreading, and reduction of the mode number}\label{S2}

\subsection{Angular spectrum tailoring}\label{S2.1}
In a nonlinear interferometer formed by two spatially separated traveling-wave high-gain parametric amplifiers (Fig.~\ref{separated})~\cite{separation}, broadband radiation emitted by the first crystal is amplified in the second one. If the distance between the crystals is considerable, only part of the radiation is amplified in the second one - namely, the part that passes through the pump beam in the second crystal. In accordance with this, it was shown~\cite{separation} that at a certain distance between the crystals, the angular spectrum becomes nearly single-mode.
At a sufficiently large distance between the crystals, the angular width of the spectrum amplified in the second crystal should be roughly given by the ratio between the pump diameter $a$ and the distance $L$ between the crystals,
\begin{equation}
\Delta\theta\approx\frac{a}{L}.
\label{eq:angular width}
\end{equation}
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.5\textwidth]{separated2.eps}
\caption{(Color online) A nonlinear interferometer formed by two spatially separated crystals.}\label{separated}
\end{center}
\end{figure}

This effect of spatial spectrum narrowing has a simple explanation in terms of the Schmidt modes of high-gain PDC. Indeed, each of the spatial Schmidt modes of BSV emitted by the first crystal is diffracting (spreading) in the course of propagation to the second crystal. To a good approximation, the spatial Schmidt modes are given by the Hermite-Gauss or Laguerre-Gauss set, the lowest-order mode being simply a Gaussian beam. Lower-order modes spread slowly and in the second crystal they overlap with the pump beam. Therefore, they get amplified provided that the phase acquired in the course of propagation is appropriate. However, higher-order modes (no matter if they are given by Laguerre-Gauss or Hermite-Gauss beams) spread faster in the space between the crystals and do not get amplified. As a result, the spatial spectrum at the output of the second crystal gets narrower. This continues until only the first Schmidt mode gets amplified, after which the angular width remains constant; increasing the distance $L$ only reduces the total intensity.

The dependence of the angular width $\Delta\theta$ on the distance $L$ between the crystals can be derived from this picture as follows. The zeroth-order Schmidt mode is a Gaussian beam of waist radius $w_{0}$. As it propagates from the crystal, the waist radius at a distance $z$ is~\cite{Kogelnik}
\begin{equation}
w_0(z)=\sqrt{w_{0}^{2}+\left(\frac{\lambda z}{\pi w_{0}}\right)^2},\,\, w_0(0)=w_0,
\label{eq:waist_1st}
\end{equation}
with $\lambda$ being the wavelength. The parameter $\theta_0\equiv\lambda/(\pi w_0)$ is the half-angle divergence of the Gaussian beam. Higher-order modes have larger spatial sizes, $w_{m}=Mw_0$; for instance, for Hermite-Gauss beams, $M=\sqrt{2m+1}$. At the same time, they have larger divergences $\theta_m=M\theta_0$, so that as the distance $z$ increases, they spread as
\begin{equation}
w_m(z)=\sqrt{w_{m}^{2}+\left(M^2\frac{\lambda z}{\pi w_{m}}\right)^2}.
\label{eq:waist_m}
\end{equation}
Assuming that for $z=L$, only modes of orders from $0$ to $m$ are amplified in the second crystal, we find the corresponding $M$ from the condition $w_m(L)=a/2$. We obtain
\begin{equation}
M=\frac{a}{2}\left[w_0^{2}+\left(\frac{\lambda L}{\pi w_0}\right)^2\right]^{-1/2}.
\label{eq:M}
\end{equation}
Then, the divergence of the beam will be equal to twice the half-angle divergence of mode $m$, $\Delta\theta=2\theta_m=M\frac{\lambda}{\pi w_{0}}$:
\begin{equation}
\Delta\theta=\left[\frac{1}{\Delta\theta_0^{2}}+\left(\frac{L}{a}\right)^2\right]^{-1/2},
\label{eq:angular width_full}
\end{equation}
where $\Delta\theta_0=\frac{a\lambda}{\pi w_{0}^2}$ is the initial angular width.

We have measured the dependence of the angular width on the distance between the crystals under the same experimental conditions as in Ref.~\cite{separation}: two $3$ mm crystals were placed into a single pump beam of full width at half maximum (FWHM) waist $200$ $\mu$m, and the distance between them was changed from $10$ to $130$ mm. The results are shown in Fig.~\ref{angular}.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.4\textwidth]{angular.eps}
\caption{(Color online) The angular width of the spectrum measured at the output of the second crystal, versus the distance between the crystals. The solid line is calculated with Eq.~(\ref{eq:angular width_full}), without fitting parameters. The dashed line is a guide to an eye. The arrow shows the position at which $m=1.1$ spatial modes were reported in Ref.~\cite{separation}.}\label{angular}
\end{center}
\end{figure}
Equation (\ref{eq:angular width_full}) was used for fitting the dependence at $L\le60$ mm. At larger distances, the angular spectrum shows no dependence on $L$; the dashed line is just a guide to the eye. The position at which nearly single-mode was observed (the number of modes was measured to be $m=1.1$)~\cite{separation} is shown by an arrow.

\subsection{Frequency spectrum tailoring}\label{S2.2}
This effect has an analogue in the frequency/time domain. In this case, the role of the diffractive spreading of beams is played by the dispersive spreading of pulses. Indeed, let a dispersive material of length $d$ be placed between the two crystals. Each frequency Schmidt mode of the BSV from the first crystal will spread in time in the course of propagation through the material, and the spreading will be determined by the group-velocity dispersion (GVD) $k''=d^2k/d\omega^2$. Lower-order modes are narrower in time than higher-order ones, but they will still spread in the GVD material slower than higher-order ones. This is illustrated by Fig.~\ref{modes} where several temporal Schmidt modes are plotted for BSV emitted from a 3 mm crystal pumped by $6$ ps pulses at wavelength $355$ nm (a). The emission is at the degenerate wavelength $710$ nm. The modes are assumed to be the same as for spontaneous parametric down-conversion~\cite{Law}. To a good approximation, they are given by Hermite functions~\cite{Wasilewski}.

After propagation through a dispersive material with the GVD $k''$ and length $d$, the temporal modes will spread in time but maintain their shapes. The latter follows from the fact that, similar to diffractive spreading of beams, dispersive spreading of pulses acts as the Fourier transformation, so that after a sufficiently long GVD material the shape of a pulse becomes similar to its spectral amplitude. At the same time,  Hermite functions are eigenfunctions of the Fourier transformation. Therefore, the whole set of Schmidt modes will be simply rescaled after the propagation through the dispersive material. For instance, the zeroth-order Schmidt mode (dotted line in Fig.~\ref{modes}a), initially a Gaussian pulse of duration $\tau_{0}$ will remain a Gaussian pulse with the duration depending on $d$~\cite{Yariv},
\begin{equation}
\tau_0(d)=\sqrt{\tau_{0}^{2}+\left(\frac{k'' d}{\tau_{0}}\right)^2}.
\label{eq:time_1st}
\end{equation}
Higher-order modes will also maintain their shapes but, similar to the case of the angular modes, will spread faster.

In the right panel of Fig.~\ref{modes}, the Schmidt modes are plotted after the propagation through various lengths $d$ of SF6 glass, whose GVD at the wavelength $710$ nm is $k''=238$ fs$^2$/mm~\cite{refractive index}. In the calculation, Eq.~(\ref{eq:time_1st}) was used for the Gaussian mode, and the higher-order modes were simply rescaled accordingly.) For the length $d=10$ cm (b), only higher-order modes (mode $50$ in the figure) get sufficiently spread so that they do not fully overlap with the pump pulse in the second crystal. Therefore, high-order modes will not be amplified.

 However, at $d=20$ cm (c), even the tenth-order mode becomes considerably spread and will not get fully amplified in the second crystal. This should lead to the narrowing of the spectrum. As the length of the dispersive glass increases, the spectral width should reduce. After $d=60$ cm of glass (d), the zeroth-order Schmidt mode will overlap with the pump pulse. At high gain, this situation should lead to single-mode output emission.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.7\textwidth]{modes.eps}
\caption{(Color online) Temporal Schmidt modes of orders $0$ (red dotted line), $10$ (blue dashed line) and $50$ (green solid line) for a $3$ mm crystal pumped by $6$ ps pulses before (a) and after propagating through $10$ cm (b), $20$ cm (c) and $60$ cm (d) of SF6 glass. For comparison, the amplitude of the pump pulse is shown by magenta dash-dotted line.}\label{modes}
\end{center}
\end{figure}

By analogy with Eq.~\ref{eq:angular width_full}, one can estimate the frequency spectrum of BSV generated in the system of two crystals separated by a GVD material as
\begin{equation}
\Delta\omega=\left[\frac{1}{\Delta\omega_0^{2}}+\left(\frac{k''d}{T_p}\right)^2\right]^{-1/2},
\label{eq:frequency width_full}
\end{equation}
where $T_p$ is the pump pulse duration and $\Delta\omega_0$ the initial spectral width.
In the next section, we describe the experimental results confirming this behavior.

\section{Experiment}\label{S3}
The scheme of the experimental setup is shown in Fig.~\ref{setup}. Collinear high-gain PDC with the central wavelength at 709.3 nm was created in a type-I 3-mm-long BBO crystal by pumping it with a third harmonic of the pulsed Nd:YAG laser at 354.7 nm, with a pulse duration of 18 ps (the coherence time being $6$ ps) and a repetition rate of 1 kHz. The laser power was varied by a half wave plate  $\lambda_p/2$ followed by a polarization beamsplitter $PBS_p$. A telescope, made of plano convex lenses $L_{p1}$ and $L_{p2}$ with the focal distances of 50 cm and 5 cm, respectively, compressed the beam size down to half-power beam width of 225 $\mu m$. A dichroic mirror $DM_1$ separated the pump beam and the PDC. The PDC pulses were passing through the group velocity dispersion (GVD) medium. We had three options of GVD media: SF-6 glass rods of length $9$ cm and $18.3$ cm, and SF-57 glass rod of length $19.4$ cm. The pump pulses were timed, by means of a delay line, to overlap with the time-stretched PDC pulses on the dichroic mirror $DM_2$ and amplify them in the second type-I 3-mm-long BBO crystal. After the crystal the pump was blocked by a pair of dichroic mirrors $DM_3$ and a long-pass filter OG580. The iris $A_1$ placed in the focal plane of the lens L with the focal distance of 10 cm was used to align the crystals for collinear geometry. The lens $L_i$ focused the PDC radiation onto the input slit of the spectrometer with a resolution of 0.15 nm.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.8\textwidth]{setup.eps}
\caption{(Color online) Schematic of the experimental setup.}\label{setup}
\end{center}
\end{figure}

Figure~\ref{results} (a) shows the measured PDC spectra with and without the GVD medium placed between the crystals. The PDC spectrum for the crystals separated by an air gap of $24.2$ cm is shown by a blue line. The measurement was performed at an average pump power of $73.4$ mW in the setup, published before in~\cite{separation}. The interference fringes due to different refractive indexes of the pump and the signal and idler beams in the air were avoided by averaging the spectra taken over different positions of the first crystal in the range from 22.7 cm to 25.7 cm with the step of 2 mm. The FWHM of the spectrum was found to be $45.6$ nm.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.7\textwidth]{results5.eps}
\caption{(Color online) (a) Measured PDC spectra with different GVD media inserted. (b) FWHM of the PDC spectra experimentally measured with and without GVD media placed between the crystals (blue triangles) and the theoretically calculated dependence according to Eq.~\ref{eq:frequency width_full} (red line) plotted versus
$k''d$. }
\label{results}
\end{center}
\end{figure}

Green and red lines in Fig. ~\ref{results} (a) show how the spectrum of the PDC evolves as the value of $k''d$ for the inserted GVD medium is increased. All the spectra are affected by the interference arising from the frequency-dependent phase of the broadband PDC generated in the first crystal after passing through the GVD media. As a result, constructive or destructive interference is observed for the different frequencies at the output of the second crystal. Despite the interference, one can clearly see that the spectra measured with the GVD media are narrower than the one obtained with the air gap between the crystals. The  FWHM of the envelope for each measurement was considered as the width of the spectrum.
In Fig.~\ref{results} (b) we compare the measurement results for the PDC spectral width (blue triangles) and the calculation according to Eq.~\ref{eq:frequency width_full} (red line) using the GVD values for the Schott glass SF6 and SF57~\cite{refractive index}. Instead of the pulse duration $T_p$, the coherence time of the laser $T_c=6 $ ps was used since it is the coherence of the pump that matters for parametric amplification. One can see that the experimental FWHM values agree well with the calculations.

\section{Conclusion}\label{S4}
In conclusion, we have considered a nonlinear interferometer formed by two unseeded traveling-wave parametric amplifiers (based on parametric down-conversion in nonlinear crystals) and showed that its angular and frequency spectrum of emission can be modified by spatially separating the two crystals and/or placing a dispersive material between them. The effect has a simple interpretation in terms of Schmidt modes: higher-order modes spread in space and time faster than low-order ones and hence do not get amplified in the second crystal. Our experimental results show the narrowing of the spatial spectrum, leading ultimately to a single spatial mode. For frequency spectrum narrowing, preliminary results show 30\% narrowing for a glass rod with large group velocity dispersion inserted into the interferometer.

The research leading to these results has received funding from the EU FP7 under grant agreement No. 308803 (project BRISQ2). We thank Xin Jiang and Patricia Schrehardt for providing the samples of SF6 and SF57.