Here is a small survey which I did with professionals with 1–3 years of experience in analytics industry (my sample size is ~200).
I was amazed to see such low percent of analyst who actually knows what goes behind the scene. We have now moved towards a generation where we are comfortable to see logistic regression also as a black box. In this article, I aim to kill this problem for once and all. The objective of the article is to bring out how logistic regression can be made without using inbuilt functions and not to give an introduction on Logistic regression.
Refreshers of mathematics terminology
Logistic regression is an estimation of Logit function. Logit function is simply a log of odds in favor of the event. This function creates a s-shaped curve with the probability estimate, which is very similar to the required step wise function. Here goes the first definition :
Logit Function:
Logistic regression is an estimate of a logit function. Here is how the logit function looks like:
Now that you know what we are trying to estimate, next is the definition of the function we are trying to optimize to get the estimates of coefficient. This function is analogous to the square of error in linear regression and is known as the likelihood function. Here goes our next definition.
Likelihood Function
Given the complicated derivative of the likelihood function, we consider a monotonic function which can replicate the likelihood function and simplify derivative. This is the log of likelihood function. Here goes the next definition.
Log Likelihood
Finally we have the derivatives of log likelihood function. Following are the first and second derivative of log likelihood function.
Derivative of Likelihood Function
Hessian Matrix (second derivative)
Finally, we are looking to solve the following equation.
As we now have all the derivative, we will finally apply the Newton Raphson method to converge to optimal solution. Here is a recap of Newton Raphson method.
