# General Scaled Support Vector Machines

Xin Liu and Ying Ding ECHO Labs Nanjing University of Posts and Telecommunications Nanjing, China 210003 Email: liuxin.nupt@gmail.com Forrest Sheng Bao Department of Computer Science/Electrical Engineering Texas Tech University Lubbock, Texas 79401 Email: forrest.bao@gmail.com Abstract—Support Vector Machines (SVMs) are popular tools for data mining tasks such as classification, regression, and density estimation. However, original SVM (C-SVM) only considers local information of data points on or over the margin. Therefore, C-SVM loses robustness. To solve this problem, one approach is to translate (i.e., to move without rotation or change of shape) the hyperplane according to the distribution of the entire data. But existing work can only be applied for 1-D case. In this paper, we propose a simple and efficient method called General Scaled SVM (GS-SVM) to extend the existing approach to multi-dimensional case. Our method translates the hyperplane according to the distribution of data projected on the normal vector of the hyperplane.

Compared with C-SVM, GS-SVM has better performance on several data sets.

## I. Introduction

In past several decades, large margin machines have been widely studied and used. Support vector machines (SVMs)
(also known as C-SVM) [1], the most important and effcient one proposed by Vapnik *et al.* [2], have been proven of good performance in text mining, bioinformatics, computer vision, and so forth [3]–[5]. Unlike many other classifiers minimizing the *empirical risk*, C-SVM is based on statistical learning theory [2], which emphasizes on minimizing the *structural risk*. C-SVM constructs a maximal margin between two classes. A hyperplane falls in the middle of this margin.

While the margin is solely determined by a few data points known as support vectors, remaining data points have no influence on building the classifier. Obviously, C-SVM loses some robustness because it cannot use the global information in the entire data set.

Inspired by this observation, we believe it is necessary to embed the global information into C-SVM. For a binary classification task, the distribution of two classes are usually not the same. It is reasonable to translate (i.e., to move without rotation or change of shape) the hyperplane closer to the class of the smaller variance. In [6], Feng proposed Scaled SVM (S-SVM) and gave a theoretical distance of the by extreme theory in 1-D case.

In this paper, we propose a simple method called General Scaled Support Vector Machine (GS-SVM) to generalize Feng's method to multi-dimensional case. Our method has three steps. First, it uses C-SVM algorithms to obtain the hyperplane. Then it projects all data points onto the normal vector of the hyperplane and estimates the distribution of each class on this direction. Finally, it translates the hyperplane according to Feng's conclusion. With kernel tricks, we can easily extend our method to feature space. In this framework, GS-SVM considers both local information of the data (SVs) and the global information.

The rest of the paper is organized as follows. In the next section, we give a brief background of C-SVM and Feng's conclusion (S-SVM) that our method bases on. We extend 1-
D S-SVM, to multi-dimensional case, GS-SVM, in Sect. III. Following that, we evaluate GS-SVM on toy data sets and several benchmarks. This paper is concluded in Sect. V.

## Ii. Background A. Support Vector Machines

Support Vector Machines are the implementations of Statistical Learning Theory [2] which emphasizes on minimizing structural risk. For a binary classfication problem, the two classes are labeled as +1 and−1 respectively. The C-SVM problem can be written as:

$$\min\|{\bf w}\cdot{\bf w}\|+C\sum_{i=1}^{l}\xi_{i}^{2}\tag{1}$$ $$s.t.\left\{\begin{array}{l}y_{i}({\bf w}\cdot{\bf x}_{i}+b)\geq1-\xi_{i},\qquad i=1,2\ldots,l\\ \xi_{i}\geq0,\end{array}\right.$$

whereξi∈R,xi∈R
n andyi ∈ {+1,−1} are the slack variable, feature vector and the label of thei-th data point respectively,n ∈ {1,2**, . . .**} is the dimension of feature vectors,C is the penalty coefficient, andl is the number of data points. To be solved,w∈R
n (weighing vector) and b (bias) determine the direction and offset of the hyperplane, respectively.

kw·wk 2is known as the margin width. Laying in the middle of the margin, the hyperplane bears the equation w · xi + b = 0, ∀i ∈ 1..l.

By the method of Lagrange multipliers, Eq. (1) is equivalent to:

$$\frac{\partial L}{\partial\mathbf{w}}=\mathbf{w}-\sum_{i=1}^{l}y_{i}\mathbf{x}_{i}\alpha_{i}=\mathbf{0}$$ $$\frac{\partial L}{\partial\boldsymbol{\xi}}=C\boldsymbol{\xi}-\boldsymbol{\alpha}=\mathbf{0}\tag{2}$$ $$\frac{\partial L}{\partial b}=\sum_{i=1}^{l}\alpha_{i}y_{i}=0$$
