Logistic regression
Fit

The usual measure of goodness of fit for a logistic regression uses logistic loss (or log loss), the negative log-likelihood. For a given x and y, write p k = p (x k) {\displaystyle p_{k}=p(x_{k})}. The p k {\displaystyle p_{k}} are the probabilities that the corresponding y k {\displaystyle y_{k}} will equal one and 1 − p k {\displaystyle 1-p_{k}} are the probabilities that they will be zero (see Bernoulli distribution). We wish to find the values of β 0 {\displaystyle \beta _{0}} and β 1 {\displaystyle \beta _{1}} which give the "best fit" to the data. In the case of linear regression, the sum of the squared deviations of the fit from the data points (y), the squared error loss, is taken as a measure of the goodness of fit, and the best fit is obtained when that function is minimized.

The log loss for the k -th point ℓ k {\displaystyle \ell _{k}} is:

The log loss can be interpreted as the " surprisal " of the actual outcome y k {\displaystyle y_{k}} relative to the prediction p k {\displaystyle p_{k}}, and is a measure of information content. Log loss is always greater than or equal to 0, equals 0 only in case of a perfect prediction (i.e., when p k = 1 {\displaystyle p_{k}=1} and y k = 1 {\displaystyle y_{k}=1}, or p k = 0 {\displaystyle p_{k}=0} and y k = 0 {\displaystyle y_{k}=0}), and approaches infinity as the prediction gets worse (i.e., when y k = 1 {\displaystyle y_{k}=1} and p k → 0 {\displaystyle p_{k}\to 0} or y k = 0 {\displaystyle y_{k}=0} and p k → 1 {\displaystyle p_{k}\to 1}), meaning the actual outcome is "more surprising". Since the value of the logistic function is always strictly between zero and one, the log loss is always greater than zero and less than infinity. Unlike in a linear regression, where the model can have zero loss at a point by passing through a data point (and zero loss overall if all points are on a line), in a logistic regression it is not possible to have zero loss at any points, since y k {\displaystyle y_{k}} is either 0 or 1, but 0 < p k < 1 {\displaystyle 0<p_{k}<1}.

These can be combined into a single expression:

This expression is more formally known as the cross-entropy of the predicted distribution (p k, (1 − p k)) {\displaystyle {\big (}p_{k},(1-p_{k}){\big)}} from the actual distribution (y k, (1 − y k)) {\displaystyle {\big (}y_{k},(1-y_{k}){\big)}}, as probability distributions on the two-element space of (pass, fail).

The sum of these, the total loss, is the overall negative log-likelihood − ℓ {\displaystyle -\ell }, and the best fit is obtained for those choices of β 0 {\displaystyle \beta _{0}} and β 1 {\displaystyle \beta _{1}} for which − ℓ {\displaystyle -\ell } is minimized.

Alternatively, instead of minimizing the loss, one can maximize its inverse, the (positive) log-likelihood:

or equivalently maximize the likelihood function itself, which is the probability that the given data set is produced by a particular logistic function:

This method is known as maximum likelihood estimation.
oulipo:avoid_letter[t], combination:repeat_prompt, keywords:frequency