<jupyter_start><jupyter_text>Picking a Link FunctionGeneralized linear models usually tranform a linear model of the predictors by using a [link function](https://en.wikipedia.org/wiki/Generalized_linear_modelLink_function). In logistic regression, the link function is the [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function). We can implement this really easily.<jupyter_code>def sigmoid(scores):
    return 1 / (1 + np.exp(-scores))<jupyter_output><empty_output><jupyter_text>Maximizing the Likelihood To maximize the likelihood, I need a way to compute the likelihood and the gradient of the likelihood. Fortunately, the likelihood (for binary classification) can be reduced to a fairly intuitive form by switching to the log-likelihood. We're able to do this without affecting the weights parameter estimation because log transformation are [monotonic](https://en.wikipedia.org/wiki/Monotonic_function).For anyone interested in the derivations of the functions I'm using, check out Section 4.4.1 of Hastie, Tibsharani, and Friedman's [Elements of Statistical Learning](http://statweb.stanford.edu/~tibs/ElemStatLearn/). For those less mathematically inclined, <NAME> (Univesity of Washington) details one possible derivation of the log-likelihood in a series of short lectures on [Coursera](https://www.coursera.org/learn/ml-classification/lecture/1ZeTC/very-optional-expressing-the-log-likelihood) using indicator functions. Calculating the Log-LikelihoodThe log-likelihood can be viewed as as sum over all the training data. Mathematically,损失函数$$\begin{equation}ll = \sum_{i=1}^{N}y_{i}\beta ^{T}x_{i} - log(1+e^{\beta^{T}x_{i}})\end{equation}$$ 越小越好where $y$ is the target class, $x_{i}$ represents an individual data point, and $\beta$ is the weights vector.I can easily turn that into a function and take advantage of matrix algebra.<jupyter_code>#分类问题的损失函数
def log_likelihood(features, target, weights):
    scores = np.dot(features, weights)
    ll = np.sum( target*scores - np.log(1 + np.exp(scores)) )
    return ll<jupyter_output><empty_output><jupyter_text>Calculating the Gradient![image.png](attachment:image.png)Now I need an equation for the gradient of the log-likelihood. By taking the derivative of the equation above and reformulating in matrix form, the gradient becomes: $$\begin{equation}\bigtriangledown ll = X^{T}(Y - Predictions)\end{equation}$$Again, this is really easy to implement. It's so simple I don't even need to wrap it into a function. The gradient here looks very similar to the output layer gradient in a neural network (see my [post](https://beckernick.github.io/neural-network-scratch/) on neural networks if you're curious).This shouldn't be too surprising, since a neural network is basically just a series of non-linear link functions applied after linear manipulations of the input data.  Building the Logistic Regression FunctionFinally, I'm ready to build the model function. I'll add in the option to calculate the model with an intercept, since it's a good option to have.权重的迭代公式：![image.png](attachment:image.png)<jupyter_code>#梯度下降最优化：取误差函数的极小值
def logistic_regression(features, target, num_steps, learning_rate, add_intercept = False):
    if add_intercept: #intercept：有截距b
        intercept = np.ones((features.shape[0], 1)) #features.shape返回(60,3)[0]返回60。生成60行，1列的1
        features = np.hstack((intercept, features)) #在feature前加上一列1  np.vstack():在竖直方向上堆叠；np.hstack():在水平方向上平铺
        
    weights = np.zeros(features.shape[1])  #features.shape[1]返回3，3个权重
    
    for step in range(num_steps):  #迭代多少步
        scores = np.dot(features, weights)
        predictions = sigmoid(scores)

        # Update weights with log likelihood gradient
        output_error_signal = target - predictions
        
        gradient = np.dot(features.T, output_error_signal)
        weights += learning_rate * gradient

        # Print log-likelihood every so often
        if step % 10000 == 0:
            print(log_likelihood(features, target, weights))
        
    return weights<jupyter_output><empty_output><jupyter_text>Time to do the regression.<jupyter_code>weights = logistic_regression(features, label,
                     num_steps = 50000, learning_rate = 5e-5, add_intercept=True)  #收敛了
print(weights)
def predict(features, weights):
    global mean
    global std
    features = (features - mean)/std
    intercept = np.ones((features.shape[0], 1))
    features = np.hstack((intercept, features))
    scores = np.dot(features, weights)
    predictions = sigmoid(scores)
    
    return predictions
student1 = np.array([[188, 85, 2]])
print(predict(student1, weights))
student2 = np.array([[165, 50, 25]])
print(predict(student2, weights))<jupyter_output>[0.76002054]