\section{Introduction}
	\label{Introduction}
	Pruning~\cite{lecun1990optimal,hassibi1993second, han2015deep,heyang} a trained neural network is commonly seen in network compression. In particular, for CNNs, channel pruning refers to the pruning of the filters in the convolutional layers. There are several critical factors for channel pruning.
	\textbf{Procedures}. One-shot method~\cite{li2016pruning}: Train a network from scratch; Use a certain criterion to calculate filters’ \textit{Importance Score}, and prune the filters which have small \textit{Importance Score}; After additional training, the pruned network can recover its accuracy to some extent. Iterative method~\cite{lecun1990optimal,he2018soft,frankle2018the}: Unlike One-shot methods, they prune and fine-tune a network alternately.
	\textbf{Criteria}. The filters' \textit{Importance Score} can be definded by a given criterion. From different ideas, many types of pruning criteria have been proposed, such as Norm-based~\cite{li2016pruning}, Activation-based~\cite{hu2016network,luo2017entropy}, Importance-based~\cite{molchanov2016pruning, molchanov2019importance}, BN-based~\cite{liu2017learning} and so on. \textbf{Strategy}.
	Layer-wise pruning: In each layer, we can sort and prune the filters, which have small \textit{Importance Score} measured by a given criterion. Global pruning: Different from layer-wise pruning, global pruning~\cite{liu2017learning,hecap} sort the filters from all the layers through their \textit{Importance Score} and prune them.

	\begin{table}[h]
	\centering

	\caption{
	An example to illustrate the phenomenon that different criteria may select the similar sequence of filters for pruning. Taking VGG16~(3$^{\rm rd}$ Conv) and ResNet18~(12$^{\rm th}$ Conv) on Norm-based criteria as examples. The pruned filters' index ~(the ranks of filters’  \textit{Importance Score}) are almost the same, which lead to the similar pruned structures.}
	\resizebox{\columnwidth}{!}{%
	\begin{tabular}{lllll}

		\hline
		Criteria & \multicolumn{1}{l}{Model} & \multicolumn{1}{l}{Pruned Filters' Index~(Top 8)}& \multicolumn{1}{l}{Model} & \multicolumn{1}{l}{Pruned Filters' Index~(Top 8)}\\
		\hline
		$\ell_1$    &ResNet18       &[111, 212, 33, 61, 68, 152, 171, 45] &VGG16       &[102, 28, 9, 88, 66, 109, 86, 45]\\
		$\ell_2$    &ResNet18       &[111, 33, 212, 61, 171, 42, 243, 129] &VGG16       &[102, 28, 88, 9, 109, 66, 86, 45]\\
		$\mathbf{GM}$    &ResNet18       &[111, 212, 33, 61, 68, 45, 171, 42] &VGG16       &[102, 28, 9, 88, 109, 66, 45, 86]\\
		$\mathbf{Fermat}$ &ResNet18       &[111, 212, 33, 61, 45, 171, 42, 68] &VGG16 &[102, 28, 88, 9, 109, 66, 45, 86]\\
		\hline

	\end{tabular}%
}
\label{filtersort}%
	\vspace{-0.4cm}
\end{table}%

	\begin{figure*}[t]
		\centering
		\includegraphics[width=1\textwidth]{app.jpg}
				\vspace{-0.7cm}
		\caption{Visualization of Applicability problem, \textit{i.e.,} the histograms of the \textit{Importance Score} measured by different types of pruning criteria~(like BN\_$\gamma$, Taylor $\ell_2$ and $\ell_2$ norm). The \textit{Importance Score} in each layer are close enough, which implies that it is hard for these criteria to distinguish redundant filters well in layer-wise pruing.}
\vspace{-0.2cm}
		\label{fig:app}
	\end{figure*}

	%

	In this work, we conduct our investigation on a variety of pruning criteria.
	%
	As one of the simplest and most effective channel pruning criteria, $\ell_1$ pruning~\cite{li2016pruning} is widely used in practice. The core idea of this criterion is to sort the $\ell_1$ norm of filters in one layer and then prune the filters with a small $\ell_1$ norm. Similarly, there is $\ell_2$ pruning which instead leverages the $\ell_2$ norm~\cite{frankle2018the,he2018soft}. $\ell_1$ and $\ell_2$ can be seen as the criteria which use absolute \textit{Importance Score} of filters.
	Through the study of the distribution of norm, \cite{heyang} demonstrates that these criteria should satisfy two conditions: (1)~the variance of the norm of the filters cannot be too small;  (2)~the minimum norm of the filters should be small enough. Since these two conditions do not always hold, a new criterion considering the relative \textit{Importance Score} of the filters is proposed~\cite{heyang}. Since this criterion uses the Fermat point~(\textit{i.e.}, geometric median~\cite{cohen2016geometric}), we call this method $\mathbf{Fermat}$. Due to the high calculation cost of Fermat point, \cite{heyang} further relaxed the $\mathbf{Fermat}$ and then introduced another criterion denotes as $\mathbf{GM}$. To illustrate each of the pruning criteria, let $F_{ij}\in \mathbb{R}^{N_i\times k\times k}$ represent the $j^{\rm th}$ filter of the $i^{\rm th}$ convolutional layer, where $N_i$ is the number of input channels for $i^{\rm th}$ layer and $k$ denotes the kernel size of the convolutional filter. In $i^{\rm th}$ layer, there are $N_{i+1}$ filters. For each criteria, details are shown in Table~\ref{tab:criteria}, where $\mathbf{F}$ denotes the Fermat point of $F_{ij}$ in Euclidean space. These four pruning criteria are called Norm-based pruning in this paper as they utilize norm in their design.

	Previous works~\cite{luo2017thinet,han2015deep,ding2019global,dong2017learning,renda2020comparing}, including the criteria mentioned above, the main concerns commonly consist of (a)~How much the model was compressed; (b)~How much performance was restored; (c)~The inference efficiency of the pruned network and (d)~The cost of finding the pruned network. However, few works discussed the following two blind spots about the pruning criteria:

		\begin{wraptable}{r}{6cm}
	\vspace{-0.7cm}
	\caption{Norm-based pruning criteria. }
	\vspace{-0.2cm}

	\begin{center}
		\begin{small}

			\begin{tabular}{ll}
				\hline
				Criterion & Details of \textit{Importance Score}\\
				\hline
				$\ell_1$~\cite{li2016pruning}    & $||F_{ij}||_1$\\
				$\ell_2$~\cite{frankle2018the}    & $||F_{ij}||_2$\\
				$\mathbf{Fermat}$~\cite{heyang}    & $||\mathbf{F} - F_{ij}||_2$\\
				$\mathbf{GM}$~\cite{heyang}    & $\sum_{k=1}^{N_{i+1}}||F_{ik}-F_{ij}||_2$\\
				\hline
			\end{tabular}

		\end{small}
	\end{center}
	\label{tab:criteria}
	\vspace{-0.4cm}
\end{wraptable}

	\textbf{(1)~Similarity: What are the actual differences among these pruning criteria?} Taking the VGG16 and ResNet18 on ImageNet as an example, we show the ranks of filters’ \textit{Importance Score} under different criteria in Table~\ref{filtersort}. It is obvious that they have almost the same sequence, leading to similar pruned structures. In this situation, the criteria used absolute \textit{Importance Score} of filters~($\ell_1$,$\ell_2$) and the criteria used relative \textit{Importance Score} of filters~($\mathbf{Fermat}$, $\mathbf{GM}$) may not be significantly different.

	\textbf{(2)~Applicability: What is the applicability of these pruning criteria to prune the CNNs?} There is a toy example w.r.t. $\ell_2$ criterion. If the $\ell_2$ norm of the filters in one layer are 0.9, 0.8, 0.4 and 0.01, according to \textit{smaller-norm-less-informative assumption}~\cite{ye2018rethinking}, it’s apparent that we should prune the last filter. However, if the norm are close, such as 0.91, 0.92, 0.93, 0.92, it is hard to determine which filter should be pruned even though the first one is the smallest. In Fig.~\ref{fig:app}, we demonstrate some real examples, \textit{i.e.,} the visualization of Applicability problem under different networks and criteria.

 In this paper, we provide comprehensive observations and in-depth analysis of these two blind spots. Before that, in Section~\ref{distribution}, we propose an assumption about the parameters distribution of CNNs, called \textit{Convolution Weight Distribution Assumption}~(CWDA), and use it as a theoretical tool to analyze the two blind spots. We explore the Similarity and Applicability problem of pruning criteria in the following order: (1)~Norm-based criteria~(layer-wise pruning) in Section~\ref{sec:norm}; (2)~Other types of criteria~(layer-wise pruning) in Section~\ref{sec:others}; (3) and different types of criteria~(global pruning) in Section~\ref{sec:global}. Last but not least, we provide further discussion on: (i) the conditions for CWDA to be satisfied, (ii) how our findings help the community in Section~\ref{Discussion}. In order to focus on the pruning criteria, all the pruning experiments are based on the relatively simple pruning procedure, \textit{i.e.,} one-shot method.

	The main \textbf{contributions} of this work are two-fold:

	\textbf{(1)}~We analyze the Applicability problem and the Similarity of different types of pruning criteria. These two blind spots can guide and motivate researchers to design more reasonable criteria. We also break some stereotypes, such as that the results of $\ell_1$ and $\ell_2$ pruning are not always similar.

	\textbf{(2)}~We propose and verify an assumption called CWDA, which reveals that the well-trained convolutional filters approximately follow a Gaussian-alike distribution. Using CWDA, we succeeded in explaining the multiple observations about these two blind spots theoretically.

		\section{Weight Distribution Assumption}
	\label{distribution}
	In this section,  we propose and verify an assumption about the parameters distribution of the convolutional filters.

	\textbf{(Convolution Weight Distribution Assumption)}~Let $F_{ij}\in \mathbb{R}^{N_i\times k\times k}$ be the $j^{\rm th}$ well-trained filter of the $i^{\rm th}$ convolutional layer. In general\footnote{In Section~\ref{Discussion}, we make further discussion and analysis on the conditions for CWDA to be satisfied.}, in $i^{\rm th}$ layer, $F_{ij}~( j=1,2,...,N_{i+1})$ are i.i.d and follow such a distribution:
	\begin{equation}
	F_{ij} \sim \mathbf{N}(\mathbf{0}, \mathbf{\Sigma}^i_{\text{diag}} + \epsilon\cdot\mathbf{\Sigma}^i_{\text{block}}),
	\label{cwda_org}
	\end{equation}
	where $\mathbf{\Sigma}^i_{\text{block}} = \mathrm{diag}(K_1,K_2,...,K_{N_i})$ is a block diagonal matrix and the diagonal elements of $\mathbf{\Sigma}^i_{\text{block}}$ are 0. $\epsilon$ is a small constant. The values of the off-block-diagonal elements are 0 and $K_l \in R^{k^2\times k^2}, l=1,2,...,N_i$. $\mathbf{\Sigma}^i_{\text{diag}}= \mathrm{diag}(a_1,a_2,...,a_{N_i \times k \times k})$ is a diagonal matrix and the elements of $\mathbf{\Sigma}^i_{\text{diag}}$ are close enough.

	This assumption is based on the observation shown in the Fig.~\ref{fig:cwda}. To estimate $\mathbf{\Sigma}^i_{\text{diag}} + \epsilon\cdot\mathbf{\Sigma}^i_{\text{block}}$, we use the correlation matrix  $FF^T$ where $F \in \mathbb{R}^{(N_i\times k \times k) \times N_{i+1}}$ denotes all the parameters in $i^{\rm th}$ layer. Taking a convolutional layer of ResNet18 trained on ImageNet as an example, we find that $FF^T$ is a block diagonal matrix. Specifically, each block is a $k^2 \times k^2$ matrix and the off-diagonal elements are close to 0. We visualize the $j^{\rm th}$ filter $F_{ij}\in \mathbb{R}^{N_i\times k \times k}$ in $i^{\rm th}$ layer in Fig.~\ref{fig:cwda}(c), and this phenomenon reveals that the parameters in the same channel of $F_{ij}$ tend to be linearly correlated, and the parameters of any two different channels~(yellow and green channel in Fig.~\ref{fig:cwda}(c)) in $F_{ij}$ only have a low linear correlation.
	\begin{wrapfigure}{r}{7cm}
		\includegraphics[width=0.95\linewidth]{cwda.jpg}
		\vspace{-0.5cm}
		\caption{(a-b)~Visualization of $FF^T$ in ResNet-18 trained on ImageNet dataset. More experiments can be found in Appendix~\ref{app:diag_matrix}. These experiments are based on torchvison model zoo~\cite{pytorch}, which can guarantee the generality and reproducibility. (c)~A convolutional filter. $k$ is the kernel size and $N_i$ denotes the number of input channels.}
		\label{fig:cwda}
		\vspace{-0.6cm}
\end{wrapfigure}

	\subsection{Statistical test for CWDA}
	\label{Statistical test}

	In fact, CWDA is not easy to be verified, \textit{e.g.}, for ResNet164 trained on Cifar100, the number of filters in the first stage is only 16, which is too small to be used to estimate the statistics in CWDA accurately. Thus, We consider verifying four \textbf{necessary conditions} of CWDA:

	(1)~\textbf{Gaussian.}~Whether the weights of $F_{ij}$ approximately follows a Gaussian-alike distribution;
	(2)~\textbf{Variance.}~Whether the variance of the diagonal elements of $\Sigma_{\text{diag}}$ are small enough;
	(3)~\textbf{Mean.}~Whether the mean of weights of $F_{ij}$ is close to 0.
	(4)~\textbf{The magnitude of $\epsilon$.}~Whether $\epsilon$ is small enough.

	The results of the tests are shown in Appendix~\ref{app:Statistical Test}, where we consider a variety of factors for the statistical tests, including different network structure, optimizer, regularization, initialization, dataset, training strategy, and other tasks in computer vision~(\textit{e.g}., semantic segmentation, detection and so on). The test results show that CWDA has a great generality for CNNs.

	\section{About the Norm-based criteria}
	\label{sec:norm}
	We start from the criteria in Table~\ref{tab:criteria}, which are widely cited and compared~\cite{liu2020joint,li2020group,he2020learning,liu2020rethinking,li2020eagleeye}.
	\subsection{Similarity}
	\label{Experiment and theory}
	In this section, we further verify the observation that the Norm-based pruning criteria in Table~\ref{tab:criteria} are highly similar from two perspectives. Empirically, we conducted large amount of experiments on image classification to investigate the similarities. Theoretically, we rigorously prove the similarities of the criteria in Table~\ref{tab:criteria} in layer-wise pruning under CWDA.

\begin{wrapfigure}{r}{7cm}
	\centering
    \includegraphics[width=1\linewidth]{extra_exp.jpg}
	\vspace{-0.35cm}
	\caption{Test accuracy of the ResNet56 on CIFAR10/100 while using different pruning ratios. ``L1 pruned'' and ``L1 tuned'' denote the test accuracy of the ResNet56 after $\ell_1$ pruning and fine-tuning, respectively. If ratio is 0.5, we prune 50\% filters in all layers.}
	\label{fig:exp_pruned_tuned2}
	\vspace{-0.2cm}
\end{wrapfigure}

	\textbf{Empirical Analysis}. (1)~In Fig.~\ref{fig:exp_pruned_tuned2}, we show the test accuracy of the ResNet56 after pruning and fine-tuning under different pruning ratios and datasets. The test accuracy curves of different pruning criteria at different stages are very close under different pruning ratios. This phenomenon implies that those pruned networks using different Norm-based criteria are very similar, and there are strong similarities among these pruning criteria. The experiments about other commonly used configs of pruning ratio can be found in Appendix~\ref{app:cls}.
	(2)~In Fig.~\ref{fig:vggmore}, we show the Spearman's rank correlation coefficient\footnote{Sp is a nonparametric measurement of ranking correlation, and it assesses how well the relationship between two variables can be described using a monotonic function,  \textit{i.e.}, filters ranking sequence in the same layer under two criteria in this paper.}~(Sp) between different pruning criteria. The Sp in most convolutional layers are more than 0.9, which means the network structures are almost the same after pruning. Note that the Sp in transition layer are relatively small, and the transition layer refers to the layer where the dimensions of the filter change, like the layer between stage 1 and stage 2 of a ResNet. The reason for this phenomenon may be that the layers in these areas are sensitive. It is interesting but will not greatly impact the structural similarity of the whole pruned network. The similar observations are shown in Fig.~2 in \cite{ding2019global}, Fig.~6 and Fig.~10 in \cite{li2016pruning}.

	\begin{figure*} [htbp]
		\vspace{-0.1cm}
	\centering
	\includegraphics[width=0.9\linewidth]{sp_network.jpg}

	\centering
	\caption{Spearman's rank correlation coefficient~(Sp) between different pruning criteria on several networks and datasets~(more experiments can be found in Appendix~\ref{app:sp_network}). }
	\vspace{-0.2cm}
	\label{fig:vggmore}
\end{figure*}

	\textbf{Theoretical Analysis}. Besides the experimental verification, the similarities via using layer-wise pruning among the criteria in Table~\ref{tab:criteria} can also be proved theoretically in this section. Let $C_1$ and $C_2$ be two pruning criteria to calculate the \textit{Importance Score} for
	all convolutional filters in one layer. If they can produce the similar ranks of \textit{Importance Score}, we define that $C_1$ and $C_2$ are \textit{approximately monotonic} to each other and use $C_1 \cong C_2$  to represent this relationship. In Section~\ref{Experiment and theory}, we use the Sp to describe this relationship but it's hard to be analyzed theoretically. Therefore, we focus on a stronger condition. Let $\mathbf{X} = (x_1,x_2,...,x_k)$ and $\mathbf{Y} = (y_1,y_2,...,y_k)$ be two given sequences\footnote{Since $\mathbf{X}$ is not random variables here, $\mathbb{E}(\mathbf{X})$ and $\mathbf{Var}(\mathbf{X})$ denote the average value $\sum_{i=1}^kx_i/k$ and the sample variance $\sum_{i=1}^k(x_i-\mathbb{E}(\mathbf{X}))/(k-1)$, respectively.}.
	we first normalize their magnitude, \textit{i.e.}, let $\widehat{\mathbf{X}} = \mathbf{X}/\mathbb{E}(\mathbf{X})$ and $\widehat{\mathbf{Y}} = \mathbf{Y}/\mathbb{E}(\mathbf{Y})$~. This operation does not change the ranking sequence of the elements of $\mathbf{X}$ and $\mathbf{Y}$, because $\mathbb{E}(\mathbf{X})$ and $\mathbb{E}(\mathbf{Y})$ are constants, \textit{i.e.}, $\mathbf{\widehat{X}} \cong \mathbf{\widehat{Y}} \Leftrightarrow \mathbf{X} \cong \mathbf{Y}$.~After that, if both $\mathbf{Var(\widehat{\mathbf{X}}/\widehat{\mathbf{Y}})}$ and $\mathbf{Var(\widehat{\mathbf{Y}}/\widehat{\mathbf{X}})}$ are small enough, then the Sp between $\mathbf{X}$ and $\mathbf{Y}$ is close to 1, where $\widehat{\mathbf{X}}/\widehat{\mathbf{Y}} = (\widehat{x_1}/\widehat{y_1},..,\widehat{x_k}/\widehat{y_k})$. The reason is that in these situations, the ratio $\widehat{\mathbf{X}}/\widehat{\mathbf{Y}}$ and $\widehat{\mathbf{Y}}/\widehat{\mathbf{X}}$ will be close to two constants $a,b$. For any $1 \leq i \leq k$, $\widehat{x_i} \approx a\cdot \widehat{y_i}$ and $\widehat{y_i} \approx b\cdot \widehat{x_i}$. So, $ab \approx 1$ and $a,b \neq 0$. Therefore, there exists an \textit{approximately monotonic} mapping from $\widehat{y_i}$ to $\widehat{x_i}$~(linear function), which makes the Sp between $\mathbf{X}$ and $\mathbf{Y}$ close to 1. With this basic fact, we propose the Theorem \ref{theo:similarity-layer}, which implies that many Norm-based pruning criteria produces almost the same ranks of \textit{Importance Score}.

\begin{theorem}\label{theo:similarity-layer}
Let $n-$dimension random variable $X$ meet CWDA, and the pair of criteria $(C_1,C_2)$ is one of $(\ell_1,\ell_2)$, $(\ell_2,\mathbf{Fermat})$ or $(\mathbf{Fermat},\mathbf{GM})$, we have
	\begin{equation}
	\mathbf{max}\left\{\mathbf{Var}_{X}\left(\frac{\widehat{C}_2(X)}{\widehat{C}_1(X)}\right),\mathbf{Var}_{X}\left(\frac{\widehat{C}_1(X)}{\widehat{C}_2(X)}\right) \right\}\lesssim B(n),
	\end{equation}
	\label{theo:bound}
	where $\widehat{C}_1(X)$ denotes $C_1(X)/\mathbb{E}(C_1(X))$ and $\widehat{C}_2(X)$ denotes $C_2(X)/\mathbb{E}(C_2(X))$. $B(n)$ denotes the upper bound of left-hand side and when $n$ is large enough, $B(n) \to 0$.
\end{theorem}
\begin{proof}
	(See Appendix \ref{proof:theorm1}).\qedhere
\end{proof}

	In specific, for $i^{\rm th}$ convolutional layer of a CNN, since $F_{ij}\in \mathbb{R}^n$, $j= 1,2,...N_{i+1}$, meet CWDA and the dimension $n$ is generally large, we can obtain $\ell_1 \cong \ell_2$, $\ell_2 \cong \mathbf{Fermat}$ and $\mathbf{Fermat} \cong \mathbf{GM}$ according to Theorem~\ref{theo:bound}. Therefore,  we have $\ell_1 \cong \ell_2 \cong \mathbf{Fermat} \cong \mathbf{GM}$, which verifies the strong similarities among the criteria shown in Table~\ref{tab:criteria}.

	\subsection{Applicability}
\label{Applicability}

	In this section, we analyze the Applicability problem of the Norm-based criteria. In Fig.~\ref{fig:app}~(Right), we know that there are some cases where the values of \textit{Importance Score} measured by $\ell_2$ criterion are very close~(e.g., the distribution looks sharp), which make $\ell_2$ criterion cannot distinguish the redundant filters well. It's related to the variance of \textit{Importance Score}. \cite{heyang} argue that a \textit{small norm deviation}~(the values of variance of \textit{Importance Score} are small) makes it difficult to find an appropriate threshold to select filters to prune. However, even if the values of the variance are large, it still cannot guarantee to solve this problem. Since the magnitude of these \textit{Importance Score} may be much greater than the values of the variance, we can use the mean of \textit{Importance Score} to represent their magnitude. Therefore, we consider using a relative variance $\mathbf{Var}_r[C(F_A)]$ to describe the Applicability problem. Let $\mathbb{E}[C(F_A)] > 0$ and
	\begin{equation}
	    \mathbf{Var}_r[C(F_A)] = \mathbf{Var}[C(F_A)]/\mathbb{E}[C(F_A)],
	    \label{eqn:appli}
	\end{equation}
	where $C$ is a given pruning criterion and $F_A$ denotes the filters in layer $A$. The criterion $C$ for layer $A$ has Applicability problem when $\mathbf{Var}_r[C(F_A)]$ is close to 0. Then we introduce the Proposition~\ref{prop:mean_var_cri} to provide the estimation of the mean and variance w.r.t. different criteria when the CWDA is hold:

    \begin{proposition}
    If the convolutional filters $F_A$ in layer $A$ meet CWDA, then we have following estimations:
\begin{table}[H]
	\vspace{-0.4cm}

	\begin{center}
		\begin{small}
			\begin{tabular}{lll}
				\hline
				 Criterion & Mean & Variance\\
				\hline
				$\ell_1(F_A)$    & $\sqrt{2/\pi}\sigma_Ad_A$& $(1-\frac{2}{\pi})\sigma_A^2d_A$\\
				$\ell_2(F_A)$    & $\sqrt{2}\sigma_A\Gamma(\frac{d_A+1}{2})/\Gamma(\frac{d_A}{2})$& $\sigma_A^2/2$\\
				$\mathbf{Fermat}(F_A)$   & $\sqrt{2}\sigma_A\Gamma(\frac{d_A+1}{2})/\Gamma(\frac{d_A}{2})$& $\sigma_A^2/2$\\
				\hline
			\end{tabular}
		\end{small}
	\end{center}
	\label{criteria}
\end{table}
where $d_A$ and $\sigma_A^2$ denote the dimension of $F_A$ and the variance of the weights in layer $A$, respectively.
   \label{prop:mean_var_cri}
   \vspace{-0.4cm}
    \end{proposition}
\begin{proof}
	(See Appendix \ref{app:prop}).\qedhere
	\vspace{-0.3cm}
\end{proof}
Based on the Proposition~\ref{prop:mean_var_cri}, we further provide the theoretical analysis for each criteria:

(i)~For $\ell_2(F_A)$. From Proposition~\ref{prop:mean_var_cri}, we can obtain that
\begin{align}
\vspace{-0.3cm}
\mathbf{Var}_r[\ell_2(F_A)]
&= \frac{\sigma_A^2}{2} / [\sqrt{2}\sigma_A\Gamma(\frac{d_A+1}{2})/\Gamma(\frac{d_A}{2})] = O(\sigma_A/g(d_A)),
\label{eqn:rv_l2}
\end{align}
where $g(d_A) = \Gamma(\frac{d_A+1}{2})/\Gamma(\frac{d_A}{2})$ is a monotonically increasing function w.r.t $d_A$. From Eq.~(\ref{eqn:rv_l2}), $\mathbf{Var}_r[\ell_2(F_A)]$ depend on $\sigma_A$ and $d_A$. When $\sigma_A$ is small or $d_A$ is large enough, $\mathbf{Var}_r[\ell_2(F_A)]$ tends to be 0.

	(ii) For $\mathbf{Fermat}(F_A)$. From the proof in Appendix~\ref{proof:l1vsl2}, we know that the Fermat point $\mathbf{F}$ of $F_A$ and the origin $\mathbf{0}$ approximately coincide. From Table~\ref{criteria}, $||\mathbf{F}-F_A||_2 \approx ||\mathbf{0} - F_A||_2 = ||F_A||_2$. Therefore, the mean and variance of $\mathbf{Fermat}(F_A)$ are the same as $\ell_2(F_A)$'s in Proposition~\ref{prop:mean_var_cri}. Hence, a similar conclusion can be obtained for $\mathbf{Fermat}$ criterion. \textit{i.e.,} the \textit{Importance Score} tends to be identical and it’s hard to distinguish the network redundancy well when $\sigma_A$ is small or $d_A$ is large enough.

	(iii)~For $\ell_1(F_A)$. Intuitively, the $\ell_1$ criterion should have the same conclusion as the $\ell_2$ criterion. However, given the Proposition~\ref{prop:mean_var_cri}, we can obtain that
\begin{align}
\vspace{-0.5cm}
\mathbf{Var}_r[\ell_1(F_A)]
&= (1-\frac{2}{\pi})\sigma_A^2d_A / [\sqrt{2/\pi}\sigma_Ad_A] = \epsilon(\pi)\cdot\sigma_A,
\label{eqn:rv_l1}
\vspace{-0.3cm}
\end{align}
where $\epsilon(\pi)<1$ is a constant w.r.t $\pi$. Note that $\mathbf{Var}_r[\ell_1(F_A)]$ only depend on $\sigma_A$, but not the dimension $n$. Moreover, for the common network structures, like VGG, ResNet shown in Fig.~\ref{fig:magnitude}~(b) and (d), the dimension of the filters are usually large enough. Therefore, compared with $\ell_2$, $\ell_1$ criterion is relatively not prone to have Applicability problems, unless the $\sigma_A$ is very small.

\begin{figure*} [htbp]
	\centering
	\includegraphics[width=0.92\linewidth]{other_simi.jpg}
	\centering
	\caption{The Similarity and Applicability problem for different types of pruning criteria in layer-wise or global pruning. }

	\label{fig:other_simi}
\end{figure*}

\section{About other types of pruning criteria}
\label{sec:others}

In this section, we study the Similarity and Applicability problem in other types of pruning criteria through numerical experiments, such as Activation-based pruning~\cite{hu2016network,luo2017entropy}, Importance-based pruning~\cite{molchanov2016pruning, molchanov2019importance} and BN-based pruning~\cite{liu2017learning}. For each type, we choose two representative criteria and we call them: (1)~Norm-based: $\ell_1$ and $\ell_2$; (2)~Importance-based: Taylor $\ell_1$ and  Taylor $\ell_2$~\cite{molchanov2016pruning, molchanov2019importance, molchanov2019taylor}; (3)~BN-based:  BN\_$\gamma$\footnote{The empirical result for slimming training~\cite{liu2017learning} is shown in Appendix~\ref{slimming}.} and BN\_$\beta$~\cite{liu2017learning}; (4) Activation-based:  Entropy~\cite{luo2017entropy} and APoZ~\cite{hu2016network}. The details of these criteria can be found in Appendix \ref{app:other_criteria}.

\textbf{The Similarity for different types of pruning criteria}. In Fig.~\ref{fig:other_simi}~(a-d), we show the Sp between different types of pruning criteria, and only the Sp greater than 0.7 are shown because if Sp $<$ 0.7, it means that there is no strong similarity between two criteria in the current layer.

According to the Sp shown in Fig.~\ref{fig:other_simi}~(a-d), we obtain the following observations:
(1)~As verified in Section~\ref{Experiment and theory}, $\ell_1$ and $\ell_2$ can maintain a strong similarity in each layer;
(2)~In the layers shown in Fig.~\ref{fig:other_simi}~(a) and Fig.~\ref{fig:other_simi}~(d), the Sp between most different pruning criteria are not large in these layers, which indicates that these criteria have great differences in the redundancy measurement of convolutional filters. This may lead to a phenomenon that one criterion considers a convolutional filter to be important, while another considers it redundant. We find a specific example which is shown in Appendix~\ref{app:case};
(3)~Intuitively, the same type of criteria should be similar. However, Fig.~\ref{fig:other_simi}~(b) and Fig.~\ref{fig:other_simi}~(c) show that the Sp between Taylor $\ell_1$ and Taylor $\ell_2$ is not large, but Taylor $\ell_2$ has strong similarity with both two Norm-based criteria. Moreover, the Sp between BN\_$\gamma$ and each Norm-based criteria exceeds 0.9, but it is not large in other layers~(Fig.~\ref{fig:other_simi}~(a) and Fig.~\ref{fig:other_simi}~(d)). These phenomena are worthy of further study.

	\begin{figure*} [htbp]
		\vspace{-0.2cm}
	\centering
	\includegraphics[width=1.0\linewidth]{estimation.jpg}
	\centering
	\vspace{-0.4cm}
	\caption{The magnitude of the \textit{Importance Score} measured by $\ell_1$ and $\ell_2$ criteria. }

	\label{fig:magnitude}
\end{figure*}

\textbf{The Applicability for different types of pruning criteria}. According to the analysis in Section~\ref{Applicability}, the Applicability problem depends on the mean and variance of the \textit{Importance Score}. Fig.~\ref{fig:other_simi}~(g-i) shows the result of the \textit{Importance Score} measured by different pruning criteria on each layer of VGG16. Due to the difference in the magnitude of \textit{Importance Score} for different criteria, for the convenience of visualization, the value greater than 1 is represented by 1.

  First, we analyze the Norm-based criteria. In most layers, the relative variance $\mathbf{Var}_r[\ell_2]$ is much smaller than that of $\mathbf{Var}_r[\ell_1]$, which means that the $\ell_2$ pruning has Applicability problem in VGG16, while the $\ell_1$ does not. This is consistent with our conclusion in Section~\ref{Applicability}.
  Next, for the Activation-based criteria, the relative variance $\mathbf{Var}_r$ is large in each layer, which means that these two Activation-based criteria can distinguish the network redundancy well from their measured filters' \textit{Importance Score}. However, for the Importance-based and BN-based criteria, their relative variance $\mathbf{Var}_r$ are close to 0. According to Section~\ref{Applicability}, these criteria have Applicability problem, especially in the deeper layers (e.g., from 6$^{\rm th}$ layer to the last layer).

	\begin{wrapfigure}{r}{7cm}
		\vspace{-0.3cm}
    \includegraphics[width=1\linewidth]{global_situ.jpg}
	\vspace{-0.5cm}
	\caption{The global pruning simulation for the unpruned network with only two layers.}
	\label{fig:exp_pruned_tuned}
	\vspace{-0.2cm}
\end{wrapfigure}

\section{About global pruning}\label{sec:global}

	Compared with layer-wise pruning, global pruning is more widely~\cite{liu2018rethinking, molchanov2016pruning,liu2017learning} used in the current research of channel pruning. Therefore, in this section we may also analyze the Similarity and Applicability problem of global pruning.

	\textbf{Applicability while using global pruning}. In fact, for global pruning, both $\ell_1$ and $\ell_2$ criteria are not prone to Applicability problems. From Proposition~\ref{prop:mean_var_cri}, we show that the estimations for the mean of \textit{Importance Score} in layer $A$ for $\ell_1$ and $\ell_2$ are $\sigma_A\cdot d_A\sqrt{\frac{2}{\pi}}$ and $\sqrt{2}\sigma_A\cdot\Gamma(\frac{d_A+1}{2})/\Gamma(\frac{d_A}{2})$, respectively. Since $\sigma_A$ and $d_A$ are quite different, shown in Fig.~\ref{fig:magnitude}~(b) and (d), hence the variance of the \textit{Importance Score} may be large in this situation.
	Fig.~\ref{fig:magnitude}~(a) and (c) show such kind of difference of the magnitude on different convolutional layers. In addition, from our estimations in Fig.~\ref{fig:magnitude}~(c), this inconsistent magnitude can be explained for another common problem in practical applications of global pruning: the ResNet is easily pruned off. As shown in Fig.~\ref{fig:magnitude}~(c), we take ResNet56 as an example. Since the \textit{Importance Score} in first stage is much smaller than the \textit{Importance Score} in the deeper layer, global pruning will give priority to prune the convolutional filters of the first stage. For problem, we suggest that some normalization tricks should be implemented or a protection mechanism should be established, \textit{e.g.}, a mechanism which can ensure that each layer has at least a certain number of convolutional filters that will not be pruned. Unlike some previous works~\cite{hecap,chin2020towards,wang2019cop}, which make suggestions from qualitative observation, we provide a quantitative view to illustrate that these tricks are necessary.

	\textbf{Similarity while using global pruning}. In Fig.~\ref{fig:other_simi}~(e-f), we show the similarity of different types of pruning criteria using global pruning on VGG16 and ResNet56. Comparing to the results from the layer-wise pruning shown in Fig.~\ref{fig:other_simi}~(a-d), we can find that the similarities of most pruning criteria are quite different in global pruning. In addition, the same criteria may have different results for different network structures in global pruning, \textit{e.g.,} in Fig.~\ref{fig:other_simi}~(e), we can find $\ell_2 \cong$ Taylor $\ell_2$ and BN$_\gamma \cong \ell_2$, but this observation does not hold in Fig.~\ref{fig:other_simi}~(f). In particular, different from the result about ResNet56 in Fig.~\ref{fig:other_simi}~(f), the similarity between $\ell_1$ and $\ell_2$ is not as strong as the one in the layer-wise case. This phenomenon is counter intuitive.

	To understand this phenomenon, we first consider about a simple case, \textit{i.e.,} the unpruned network has only two convolutional layers~(layer $A$ and layer $B$). The filters in these two layers are $F_A = (F_A^1,F_A^2,...,F_A^n)$ and $F_B = (F_B^1,F_B^2,...,F_B^m)$. According to CWDA, for $1\leq i \leq n$ and $1 \leq j \leq m$, $F_A^i$ and $F_B^j$ can follow $N(\mathbf{0},\sigma_A^2\mathbf{I}_{d_A})$ and $N(\mathbf{0},\sigma_B^2\mathbf{I}_{d_B})$, respectively. Next, we show Sp between \textit{Importance Score} measured by $\ell_1$ and $\ell_2$ pruning in different dimension ratio $d_A/d_B$, $\sigma_A$ and $\sigma_B$ in Fig.~\ref{fig:exp_pruned_tuned}~(e-h). Moreover, to analyze this phenomenon concisely, we draw some scatter plots as shown in Fig.~\ref{fig:exp_pruned_tuned}~(a-d), where the coordinates of each point are given by (value of $\ell_1$, value of $\ell_2$). The set of the points consisting of the filters in layer $A$ is called group-$A$. Then we introduce the Proposition 2.
    \begin{proposition}
    If the convolutional filters $F_A$ in layer $A$ meet CWDA, then $\mathbb{E}[\ell_1(F_A)/\ell_2(F_A)]$ and $\mathbb{E}[\ell_2(F_A)/\ell_1(F_A)]$ only depend on their dimension $d_A$.
    \label{prop:slope}
    \vspace{-0.3cm}
    \end{proposition}

\begin{proof}
	(See Appendix \ref{app:prop}).\qedhere
	\vspace{-0.4cm}
\end{proof}

    Now we analyze the simple case under different situations

	(1)~For $d_A/d_B = 1$. If $\sigma_A^2 = \sigma_B^2$, in fact, it's the same situation as layer-wise pruning. From Theorem~\ref{theo:similarity-layer}, we know that group-$A$ and group-$B$ coincide and approximately lie on the same line, resulting $\ell_1 \cong \ell_2$ . If $\sigma_A^2 \not= \sigma_B^2$, group-$A$ and group-$B$ lie on two lines, respectively. However, these two lines have the same slope based on Proposition~\ref{prop:slope}, as shown in Fig.~\ref{fig:exp_pruned_tuned}~(a). For these reasons, we have $\ell_1 \cong \ell_2$ when $d_A/d_B = 1$.

	(2)~For $d_A/d_B \not= 1$. In Fig.~\ref{fig:exp_pruned_tuned}~(b-d), there are three main situations about the position relationship between group-$A$ and group-$B$. In Fig.~\ref{fig:exp_pruned_tuned}~(b), according to Theorem~\ref{theo:similarity-layer}, the points in group-$A$ and group-$B$ are monotonic respectively. Moreover, their \textit{Importance Score} measured by $\ell_1$ and $\ell_2$ do not overlap, which make $\ell_1$ and $\ell_2$ are \textit{approximately monotonic} overall. Thus, $\ell_1 \cong \ell_2$. However, for Fig.~\ref{fig:exp_pruned_tuned}~(c-d), the Sp is small since the points in these two group are not monotonic (the \textit{Importance Score} measured by $\ell_1$ or $\ell_2$ has a large overlap). From Proposition~\ref{prop:mean_var_cri} and the approximation $\Gamma(\frac{d_A+1}{2})/\Gamma(\frac{d_A}{2}) \approx \sqrt{d_A/2}$~(Appendix~\ref{proof:l1vsl2}), these two situations can be described as:
\begin{equation}
\sigma_{A} d_{A} \approx \sigma_{B} d_{B}\quad or\quad \sigma_{A} \sqrt{d_{A}} \approx \sigma_{B} \sqrt{d_{B}},
\label{eqn:overlap}
\end{equation}
where $d_A \not= d_B$. Through Eq.~(\ref{eqn:overlap}) we can obtain the two red lines shown in Fig.~\ref{fig:exp_pruned_tuned}~(f-h). It can be seen that the area surrounded by these two red lines is consistent with the area where the Sp is relatively small, which means our analysis is reasonable. Based on the above analysis, we can summarize the conditions about $\ell_1 \cong \ell_2$ in global pruning for two convolutional layers as shown in Table~\ref{tab:twolayer_global}.

\begin{wraptable}{r}{7cm}
\vspace{-0.6cm}
  \centering
  \small
  \caption{The conditions about $\ell_1 \cong \ell_2$ in global pruning for two layers~(layer $A$ and layer $B$)}
  	\resizebox{0.5\columnwidth}{!}{
    \begin{tabular}{rccc|c}
    \hline
          & \multicolumn{1}{l}{$d_A = d_B$?} & \multicolumn{1}{l}{$\frac{\sigma_{A}}{\sigma_{B}}  \approx \frac{d_{B}}{d_{A}} $?} & \multicolumn{1}{l|}{$ \frac{\sigma_{A}}{\sigma_{B}}   \approx \frac{\sqrt{d_{B}}}{\sqrt{d_{A}}} $?} & \multicolumn{1}{l}{$\ell_1 \cong \ell_2$?} \\
    \hline
    (1)     &{\color{black}{\Checkmark}}       & \textbf{--}      & \textbf{--}      &{\color{green}{\Checkmark}}  \\
    (2)     &{\color{black}{\XSolidBrush}}       &{\color{black}{\XSolidBrush}}       &{\color{black}{\XSolidBrush}}       &{\color{green}{\Checkmark}}   \\
    (3)     &{\color{black}{\XSolidBrush}}     &{\color{black}{\Checkmark}}       & \textbf{--}      &{\color{red}{\XSolidBrush}}  \\
    (4)     &{\color{black}{\XSolidBrush}}      & \textbf{--}      &{\color{black}{\Checkmark}}       &{\color{red}{\XSolidBrush}}  \\

    \hline
    \end{tabular}%
    }
  \label{tab:twolayer_global}%
  \vspace{-0.2cm}
\end{wraptable}

Next, we go back to the the situation about real neural networks in Fig.~\ref{fig:other_simi}~(e-f).
 (1)~For ResNet56. As shown in Fig.\ref{fig:magnitude}~(d), the dimensions of the filters in each stage are almost the same. From Table~\ref{tab:twolayer_global}~(1), the pruning results after $\ell_1$ and $\ell_2$ pruning in each stage are similar. And, the magnitudes of the \textit{Importance Score} in each stage are very different, since Table~\ref{tab:twolayer_global}~(2), we can obtain that $\ell_1 \cong \ell_2$ for ResNet56.

(2) For VGG16. As shown in Fig.\ref{fig:magnitude}~(a-b), compared with ResNet56, VGG16 has some layers with different dimensions but similar \textit{Importance Score} measured by $\ell_1$ or $\ell_2$, such as ``layer 2'' and ``layer 8'' for $\ell_2$ criterion in Fig.\ref{fig:magnitude}~(a). From Table~\ref{tab:twolayer_global}~(3-4), these pairs of layers make the Sp small, which explain why the result of $\ell_1$ and $\ell_2$ pruning is not similar in Fig.~\ref{fig:other_simi}~(e) for VGG16. In Appendix~\ref{app:support}, more experiments show that we can increase the Sp in global pruning by ignoring part of these pairs of layers, which support our analysis.

	\section{Discussion}
	\label{Discussion}

	\subsection{Why CWDA sometimes does not hold?}
	\label{why}
	CWDA may not always hold. As shown in Appendix~\ref{app:Statistical Test}, a small number of convolutional filters may not pass all statistical tests. In this section, we try to analyze this phenomenon.

	(1)~\textbf{The network is not trained well enough}. The distribution of parameters should be discussed \textbf{only when} the network is trained well. If the network does not converge, it is easy to construct a scenario which does not satisfy CWDA, \textit{e.g.}, for a network with uniform initialization, when it is only be trained for a few epochs, the distribution of parameters may be still close to a uniform distribution. At this time, the distribution obviously does not satisfy CWDA. A specific example is in Appendix~\ref{app:other_result}.

	(2)~\textbf{The number of filters is insufficient.} In Appendix~\ref{app:Statistical Test}, the layers that can not pass the statistical tests are almost those whose position is in the front of the network. A common characteristic of these layers is that they have a few filters, which may not estimate statistics well. Taking the second convolutional layer~(64 filters) in VGG16 on CIFAR10 as an example, first, the filters in this layer can not pass all the statistical tests. And then the Sp in this transition layer is relatively small, as shown in Fig.~\ref{fig:vggmore}. However, in Fig.~\ref{fig:vgg_change}, we change the number of filters in this layer from 64 to 128 or 256. After that, the Sp increases significantly, and the filters can pass all the statistical tests when the number of filters is 256. These observations suggest that the number of filters is a major factor for CWDA to be hold.

 		\begin{figure*} [h]
	\centering
	\vspace{-0.2cm}
	\includegraphics[width=0.92\linewidth]{sp_vgg_2nd.jpg}

	\centering
	\vspace{-0.2cm}
	\caption{The Sp between different pruning criteria on VGG16~(CIFAR10). The number of filters in the second convolutional layers is changed from 64 to 256. The filters in this layer can pass all the statistical tests when the number of filters is 256.}
	\vspace{-0.2cm}
	\label{fig:vgg_change}
\end{figure*}

	\subsection{How our findings help the community?}
	\label{how}

	(1)~We propose an assumption about the parameters distribution of the CNNs called CWDA, which is an effective theoretical tool for analyzing convolutional filter. In this paper, CWDA is successfully used to explain many phenomena in the Similarity and Applicability of pruning criteria. In addition, it also explains why the ResNet is easily pruned off in global pruning. In Section~\ref{Statistical test}, since CWDA can pass statistical tests in various situations, it can be expected that it can also be used as an effective and concise analysis tool for other CNNs-related areas, \textbf{not just} pruning area.

	(2)~In this paper, we study the Similarity and Applicability problem about pruning criteria, which can guide and motivate the researchers to design more reasonable criteria. For Applicability problem, we suggest that, intuitively, it is reasonable that the \textit{Importance Score} should be distinguishable for the proposed novel criteria. For Similarity, as more and more criteria are proposed, these criteria can be used for ensemble
learning to enhance their pruning performance~\cite{he2020learning}. In this case,
the similarity analysis between criteria in this paper is important, because highly similar criteria cannot bring gains to ensemble learning.

	(3)~In pruning area, $\ell_1$ and $\ell_2$ are usually regarded as the same pruning criteria, which is intuitive. In layer-wise pruning, we do prove that the $\ell_1$ and $\ell_2$ pruning are almost the same. However, in global pruning, the pruning results by these two criteria are sometimes very different. In addition, compared with $\ell_1$ criterion, $\ell_2$ criterion is prone to Applicability problems. These counter-intuitive phenomena enlighten us that we can't just rely on intuition when analyzing problems.

\begin{table}[htbp]
  \centering
  \caption{The random pruning results of VGGNet with different criteria which have the Applicability problem. The VGG16 and VGG19 are trained on CIFAR100. The unpruned baseline accuracy of VGG16 and VGG19 are 72.99 and 73.42, respectively.}
  \resizebox{\columnwidth}{!}{%
    \begin{tabular}{cl|cccc|cccc}
    \hline
    \textbf{Model} & \textbf{criterion} & \textbf{min (r=10\%)} & \textbf{max (r=10\%)} & \textbf{mean (r=10\%)} & \textbf{$\Delta$} & \textbf{min (r=20\%)} & \textbf{max (r=20\%)}& \textbf{mean  (r=20\%)} & \textbf{$\Delta$} \\
    \hline
    \multirow{5}[1]{*}{\begin{sideways}VGG16\end{sideways}} & $\ell_2$    & 71.41 & 72.65 & 71.75 & 1.24  & 71.01 & 72.47 & 71.32 & 1.46 \\
          & Taylor $\ell_1$  & 71.67 & 72.34 & 71.89 & 0.67  & 71.32 & 72.32 & 71.45 & 1.01 \\
          & Taylor $\ell_2$  & 71.87 & 72.37 & 71.91 & 0.5   & 71.66 & 72.27 & 71.65 & 0.61 \\
          & BN$_\gamma$ & 71.09 & 71.66 & 71.36 & 0.57  & 71.02 & 71.57 & 71.12 & 0.55 \\
          & BN$_\beta$ & 71.15 & 72.58 & 71.43 & 1.43  & 71.06 & 72.11 & 71.87 & 1.05 \\
    \hline
    \multirow{5}[2]{*}{\begin{sideways}VGG19\end{sideways}} & $\ell_2$     & 71.99 & 73.15 & 72.26 & 1.16  & 71.11 & 73.02 & 72.15 & 1.91 \\
          & Taylor $\ell_1$  & 71.67 & 73.04 & 72.23 & 1.37  & 71.6  & 72.98 & 72.24 & 1.38 \\
          & Taylor $\ell_2$  & 72.12 & 72.99 & 72.28 & 0.87  & 72.04 & 72.83 & 72.54 & 0.79 \\
          & BN$_\gamma$ & 72.01 & 73.23 & 72.25 & 1.22  & 71.98 & 72.32 & 72.12 & 0.34 \\
          & BN$_\beta$ & 72.25 & 73.23 & 72.41 & 0.98  & 72.04 & 72.65 & 72.33 & 0.61 \\
    \hline
    \end{tabular}%
    }
  \label{tab:addlabel}%
\end{table}%

(4)~Similar to the setting in Fig.~\ref{fig:other_simi}, we can explore the effect of pruning filters with similar \textit{Importance Score} on the performance. First, we find that the criteria ($\ell_2$,Taylor $\ell_1$, Taylor $\ell_2$, BN$_{\gamma}$  and BN$_{\beta}$) for VGGNet can cause the Applicability problem in most layers (Fig.~\ref{fig:other_simi}). As such, we randomly select 10\% or 20\% filters to be pruned by the uniform distribution $U[0,1]$ in each layer, and the selective filters will be in similar \textit{Importance Score}. Finally, we finetune the pruned model (there are 20 random repeated experiments). $\Delta$ denotes the difference between max acc. and min acc. (\textit{i.e.} max acc. - min acc.) . Since their \textit{Importance Score} are very similar, when the network is pruned and finetuned, it can be expected that the performance should be similar in these repeated experiments. However, from the results in the above table, although the \textit{Importance Score} of the pruned filters is very close, we can still get pruning results with very different results (\textit{e.g.} the $\Delta$ of VGG16 on $\ell_2$ are more than 1). It means that these criteria may not really represent the importance of convolutional filters. Therefore, it is necessary to re-evaluate the correctness of the existing pruning criteria.

\textbf{Acknowledgments}. Z. Huang gratefully acknowledges the technical and writing support from Mingfu Liang~(Northwestern University), Senwei Liang~(Purdue University) and Wei He~(Nanyang Technological University). Moreover, he sincerely thanks Mingfu Liang for offering his self-purchasing GPUs and Qinyi Cai (NetEase, Inc.) for checking part of the proof in this paper. This work was supported in part by the General Research Fund of Hong Kong No.27208720, the National Key R\&D Program of China under Grant No. 2020AAA0109700, the National Science Foundation of China under Grant  No.61836012 and 61876224, the National High Level Talents Special
Support Plan (Ten Thousand Talents Program), and GD-NSF (no.2017A030312006).