Evaluation

Metrics

Classification#

Confusion Matrix#

source

Metric	Def	Formula
Type I Error	reject
Type II Error	accept
Accuracy	% of correct predictions
Precision	% of actual Ps among predicted Ps
Recall/Sensitivity	% of predicted Ps among actual Ps
Specificity	% of predicted Ns among actual Ns
F1-score	Balance tradeoff between precision & recall
**F

ROC & AUC#

source

• TPR (True Positive Rate): = Recall,
• FPR (False Positive Rate): = 1 – Specificity,
• Classification threshold: anything above this defined value is classified as 1, (default 0.5)
• ROC (Receiver Operating Characteristic): performance at different classification thresholds
• AUC (Area Under the Curve): sum up performance across all classification thresholds
  • : random guess (worst)
  • : all correct (best)
  • : all wrong (reverse)

Regression#

Errors#

Basically loss functions.

Metric	Def	Formula
MAE	Mean of Residuals
MSE	Variance of Residuals
RMSE	Standard deviation of Residuals

Notes:

• MAE is robust against outliers, but MAE doesn’t penalize large errors heavily and is not differentiable.
• MSE penalizes large errors heavily and is differentiable, but MSE is sensitive to outliers.
• RMSE has the same units as the dependent variables, so it is generally more preferred than both.

Explained Variance#

Metric	Def	Formula
Explained Variance	% of
Adjusted

Notes:

• Explained variance measures how well the independent variables explain the variance in dependent variables.
• is also called Coefficient of Determination, or Goodness of Fit, because it can capture how well unseen samples are likely to be predicted by the model.
• Adjusted s increases, which is undesired due to redundancy.
• Adjusted .