AUC stands for the area under the curve, and ROC stands for Reciever operating characteristic.
This curve was actually designed by electronics engineers during the second world war to predict how well their missiles are working.
This is only used for binary classification.
Imagine I have data like this:
X Y Y^ X1 1 0.95 X2 1 0.92 X3 0 0.80 X4 1 0.76 X5 1 0.71
Let’s assume our binary classification model not only gives a class label, but it also gives us a score, like probability score. The more the score, the more is the chance it belongs to class 1.
First things, we have sorted the data in decreasing order of Y^, this is the first step to plot ROC Curve.
The second step is thresholding, take any value, let’s say 0.95 as T1.
a) T1 – 0.95
if Y^ >= T1 then it is from class 1, else 0
X Y Y^ Y^T1=0.95 X1 1 0.95 1 X2 1 0.92 0 X3 0 0.80 0 X4 1 0.76 0 X5 1 0.71 0
We have actual Y and predicted Y^T1
so we will have class labels for all points, then we can calculate TPR and FPR.
Now we will take another threshold, now T2=0.92, so our scores are as follows:
X Y Y^ Y^T1=0.95 Y^T2=0.92 X1 1 0.95 1 1 X2 1 0.92 0 1 X3 0 0.80 0 0 X4 1 0.76 0 0 X5 1 0.71 0 0
Similarly, we can keep changing thresholds up to n points so we will we T1, T2, T3, . . . ., Tn.
For every Threshold, we have TPR and FPR, as for T1 we have TPR1, FPR1, . . . ., TPRn, FPRn.
Image Source: https://en.wikipedia.org/wiki/Receiver_operating_characteristic#:~:text=A%20receiver%20operating%20characteristic%20curve,its%20discrimination%20threshold%20is%20varied.
If the Model is sensible we will our curve like a green line.
Total Area will be 1, and the area under the diagonal will be 0.5.
The area under the ROC Curve is the total area under the green curve.
Higher the area, better it is where 0 is terrible and 1 is best.
Properties of AUC,
- If you have imbalanced data, AUC can be high even for a “dumb” model, AUC is impacted by imbalanced data.
- AUC is not dependent on the Y^ scores themselves, it depends on the ordering.
- If the model is random, the AUC will be a straight diagonal line, AUC(random model) = 0.5.
- Suppose for a Model (Mk) you got AUC = 0.2, this is worse than random.
if 0.5< AUC<1, then AUC is fine.
if AUC = 0.5, then AUC is random.
if 0< AUC<0.5, then AUC is worse.
If AUC is worse then interchange the class labels, then you will get the AUC which is (1-0.2) = 0.8.