Performance of the Trained Models on Unseen Data
Explore how trained models perform on unseen data.
In reality, we don’t have the labels available for the unseen data. So, we have a separate unseen dataset that we can use to evaluate our trained models. Let’s read the unseen data and see the performance of our trained models.
unseen=pd.read_csv("BioAssay_AID362red_test.csv")print(unseen.head())
We can use our custom function to check the missing data columns.
# Utility function to check the missing columnsdef missing_col_check(df):# Checking if there is any missing dataemp_col_list=[]for col_ in df.columns:if ((df[col_].isnull().sum()/df.shape[0]*100)>0.0):emp_col_list.append(col_)#print(col_)print("The dataset scanned!")if len(emp_col_list)==0:print("No missing data column found!")else:print("Missing data found in: {}".format(emp_col_list))return Nonemissing_col_check(unseen)
Let’s separate our features and targets and observe the class balance in unseen data.
X_unseen=unseen.drop("Outcome",axis=1)y_unseen=unseen.Outcomeprint("The class balance in unseen data is:")print(y_unseen.value_counts(normalize=True))
We have trained the following three models from the previous lesson:
logRon the original data.logR_oson the data after oversampling (creating copies), the minority class.logR_smoteusing SMOTE to develop synthetic data.
Let's check their performance on the unseen data.
Accuracy score
Let’s get the accuracy scores of the individual models first and proceed with logR, which was trained on the original data.
print("************************************************************")print("Trained on imbalanced data.")print("************************************************************")print("Accuracy Score :",logR.score(X_unseen,y_unseen))print("The confusion matrix")print(metrics.confusion_matrix(y_unseen,logR.predict(X_unseen)))print("The classification report.")print(metrics.classification_report(y_unseen,logR.predict(X_unseen)))
With logR_os, which is trained on the data after oversampling (creating copies), the scores for the minority class will be fetched as follows:
print("************************************************************")print("Trained on oversampled with replacement.")print("************************************************************")print("Accuracy Score :",logR_os.score(X_unseen,y_unseen))print("The confusion matrix")print(metrics.confusion_matrix(y_unseen,logR_os.predict(X_unseen)))print("The classification report.")print(metrics.classification_report(y_unseen,logR_os.predict(X_unseen)))
Finally, we’ll use logR_smote, which was trained using SMOTE to create synthetic data.
print("************************************************************")print("Trained on oversampled using SMOTE")print("************************************************************")print("Accuracy Score :",logR_smote.score(X_unseen,y_unseen))print("The confusion matrix")print(metrics.confusion_matrix(y_unseen,logR_smote.predict(X_unseen)))print("The classification report.")print(metrics.classification_report(y_unseen,logR_smote.predict(X_unseen)))
Changing the performance matrix is helpful, and for general purposes, AUC-ROC is useful.
Area under ROC
Let's start with predicting class probabilities of the logR ...