This is module eight, and two parts to it. The first part may be a little dry, but like we looked at r-squared, we looked at error, there are other ways of measuring the performance of classification models, and so we will go through them and you will see sudden nuanced evaluation of classifiers. The second part is more fun, and we will see how we can recommend which music to listen to, which movie to watch based on, or what books to buy or what objects to buy based on past purchases. So I think that's the fun part. So these are the two parts performance evaluation and recommendation. I broadly classify the second part and recommendation, the purists will argue against me, but I have been keeping things simple in classification as in labels that I'm attaching and I'm sharing with you the ones I use, it's not uniform. So let's start with model performance. So technically, there are many ways of looking at performance. One is bias versus variance. Bias means under fitting, that is, not explaining the training set too well. Variance means overfitting, and not being able to predict well out of the sample. The why does this come about? You do want, they do want to use all the data which is available with the best model out there. The problem is, if I use too much data, it may lead to poor prediction outside the dataset, too much information not too much data, too much data cannot be bad. But if I use too many variables there may be noise which can completely throw my prediction. So to make sure we have fit our model correctly to the training data, I think the main point we have been talking about it is it should be parsimonious. So we would like to test our model, keep aside some information and test how it does, not just on the training set but also on the test, on the validation set. So, in fitting the model we don't use the validation dataset. We will use it to choose the final model. We got a training set, we got a validation set, we can try different models and see how it does in the validation set. But still you may complain that it's doing well only on the validation set, you probably biasing it by that. So we could keep up third piece separate, we call it the test set. We can test the final module on the test set. That's why data is divided into three parts: train, validate and test. Now remember, our model is always checked on the validation set until we refine it and then finally it goes to the test set. That'll actually wants you say, "You are looking at the trading data don't do that." Another trick that people have learned is called cross-validation. So what it does is this idea is to avoid over fitting the data through cross-validation. We're also really seen this in the random forest. You partition the data into 10 subsets. What you do is, you fit the model on nine of them, and then test on the 10th one and keep doing it. So this can be done in times by deciding which 10th part we set aside, and then you can compute the error across these 10 repetitions, and you can average to get the expected model performance. So that's the idea of cross-validation. Now comes all the things which we may use to refine our model, okay? Here are some metrics, error rates and then we have seen error rates already. We have seen what is a misclassification in our and we have seen a confusion matrix, but we'll see it again. In addition now, we're going to talk about true positives, false positives, true negatives, false negatives, precision, recall, sensitivity, blob, excellent and lift. We're also going to talk about scoring these models. So these are the extra refinement, and I'll try to communicate the spirit of them. No need to memorize any of this, you can always go back and verify them. So essentially, the error rate says, "How you're doing on all your data." I think the issues really come up when you care for one class more than the other. For example, there may be a cancer patient and you don't detect the cancer. It's maybe a false negative, you have yes a cancer, but you said no you don't have a cancer, that's dangerous. But let's go to spam. Spam goes the other way around. A email is not spam and you classify it as a spam that's more dangerous than classifying a spam email as a proper email, because if you classify a good email is spam you are throwing away a email which is important to you, and you remember often you have to say, "Let me go see my spam," and that's after two days, and you say, "Oh I'm sorry it went to spam." So that is a delay in communication. So there are situations where one category becomes more important than the other, then all these other ideas I'm going to share with you become more important.
