Training NABirds Classification Model Using VGG16 on Google Colab¶
Table of Contents¶
- Introduction
- Workflow Context
- Objectives
- Transition to Google Colab
- Setup and Imports
- Mount Google Drive
- Import Necessary Libraries
- Dataset Preparation
- Load Preprocessed Images
- Setup Training and Testing Generators
- Model Configuration
- Initialize VGG16 Model
- Customization and Fine-Tuning
- Training and Saving the Model
- Define Training Parameters
- Start Training
- Model Evaluation and Plotting
- Predict Classes for Unseen Images
- Visualize Predictions
Introduction¶
This notebook continues the workflow for classifying North American bird species from the NABirds dataset by implementing the training pipeline using Google Colab. It builds on the outputs generated in the preceding notebook, which was executed locally to preprocess the dataset.
Workflow Context¶
Previous Notebook (Local Execution):
- Focused on image preprocessing for the NABirds dataset.
- Tasks included combining metadata, cropping and resizing images to uniform dimensions (
224x224), and saving preprocessed images to a structured directory.
This Notebook (Google Colab Execution):
- Transitions the workflow to Google Colab to leverage its computational resources for deep learning model training.
- Uses the saved preprocessed images from the local workflow as input for training.
Objectives¶
Setup Image Generators:
- Load training and testing images using TensorFlow/Keras image generators.
- Ensure alignment between training and testing class structures.
Model Configuration:
- Utilize the VGG16 architecture, pre-trained on ImageNet, for fine-tuning.
- Apply data augmentation and hyperparameter optimization for robust training.
Training and Evaluation:
- Train the VGG16 model on the preprocessed dataset.
- Evaluate the model’s performance and address discrepancies, such as class mismatches between training and testing datasets.
Save Model and Results:
- Save the trained model for future inference.
- Record observations, performance metrics, and areas for improvement.
Transition to Google Colab¶
By shifting from local processing to cloud-based training in this notebook, the workflow achieves scalability and efficiency, enabling high-performance bird species classification.
In [ ]:
In [ ]:
Setup and Imports¶
In [18]:
import os
import shutil
import time
import pandas as pd
import random
import matplotlib.pyplot as plt
import numpy as np
import math
import logging
from tensorflow.keras.applications import EfficientNetB0
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array
from tensorflow.keras.regularizers import l2
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, BatchNormalization, Dropout, Input, Resizing
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, Callback
import tensorflow as tf
import seaborn as sns
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report, roc_curve, roc_auc_score, auc
from sklearn.preprocessing import label_binarize
In [2]:
# Setup logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')
logger = logging.getLogger(__name__)
In [3]:
from google.colab import drive
drive.mount('/content/drive')
data_path = "/content/drive/My Drive/NaBirds"
print(tf.config.list_physical_devices('GPU'))
Mounted at /content/drive [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
In [4]:
base_data_dir = r'/content/drive/My Drive/NaBirds'
my_birds_image_dir = r'/content/drive/My Drive/NaBirds/processed_images'
# Define image shape for the model
# batch_size_data_load_val = 30 # Define your batch size
transfer_model = tf.keras.applications.VGG16
transfer_model_shp = (224, 224) # Typical for VGG16
transfer_model_input_shape = (224, 224, 3)
MyEpochs = 300
batch_size_val = 5
batch_size_data_load_val = 10
learning_rate_val = 0.001
dropout_rate = 0.25
In [5]:
start_time = time.time()
# Step-by-step process using the example logic
# Load bounding box information
file_path = os.path.join(data_path, "bounding_boxes.txt")
with open(file_path, 'r') as file:
lines = [line.strip() for line in file.readlines() if line.strip()]
bounding_boxes = pd.DataFrame(
[{'UUID': line.split()[0], 'bb_x': int(line.split()[1]), 'bb_y': int(line.split()[2]),
'bb_width': int(line.split()[3]), 'bb_height': int(line.split()[4])} for line in lines]
)
# Load class labels
file_path = os.path.join(data_path, "image_class_labels.txt")
with open(file_path, 'r') as file:
lines = [line.strip() for line in file.readlines() if line.strip()]
class_labels = pd.DataFrame(
[{'UUID': line.split()[0], 'class': int(line.split()[1])} for line in lines]
)
# Load train/test split
file_path = os.path.join(data_path, "train_test_split.txt")
with open(file_path, 'r') as file:
lines = [line.strip() for line in file.readlines() if line.strip()]
train_test_split = pd.DataFrame(
[{'UUID': line.split()[0], 'is_training_image': int(line.split()[1])} for line in lines]
)
# Load class descriptions
file_path = os.path.join(data_path, "classes.txt")
with open(file_path, 'r') as file:
lines = [line.strip() for line in file.readlines() if line.strip()]
class_descriptions = pd.DataFrame(
[{'class': int(line.split(" ", 1)[0]), 'description': line.split(" ", 1)[1]} for line in lines]
)
# Load image sizes
file_path = os.path.join(data_path, "sizes.txt")
with open(file_path, 'r') as file:
lines = [line.strip() for line in file.readlines() if line.strip()]
image_sizes = pd.DataFrame(
[{'UUID': line.split()[0], 'im_width': int(line.split()[1]), 'im_height': int(line.split()[2])} for line in lines]
)
# Load image paths
file_path = os.path.join(data_path, "images.txt")
with open(file_path, 'r') as file:
lines = [line.strip() for line in file.readlines() if line.strip()]
image_paths = pd.DataFrame(
[{'UUID': line.split()[0], 'path': line.split()[1]} for line in lines]
)
# Merge dataframes
merged_data = (bounding_boxes
.merge(class_labels, on="UUID")
.merge(train_test_split, on="UUID")
.merge(class_descriptions, on="class")
.merge(image_sizes, on="UUID")
.merge(image_paths, on="UUID"))
merged_data['class'] = merged_data['class'].astype(str)
# Display the merged dataset and number of rows
print("Merged DataFrame Head:")
print(merged_data.head(2))
# Print the number of rows
print(f"Number of rows in the dataset: {len(merged_data)}")
end_time = time.time()
time_elapsed = round((end_time - start_time),2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
Merged DataFrame Head:
UUID bb_x bb_y bb_width bb_height \
0 0000139e-21dc-4d0c-bfe1-4cae3c85c829 83 59 128 228
1 0000d9fc-4e02-4c06-a0af-a55cfb16b12b 328 88 163 298
class is_training_image description im_width im_height \
0 817 0 Oak Titmouse 296 341
1 860 0 Ovenbird 640 427
path
0 0817/0000139e21dc4d0cbfe14cae3c85c829.jpg
1 0860/0000d9fc4e024c06a0afa55cfb16b12b.jpg
Number of rows in the dataset: 48562
Time to run this chunk of code: 4.23 seconds
In [6]:
start_time = time.time()
processed_data_dir = os.path.join(base_data_dir, "processed_images") # Matches Section 2
bounding_boxes_file = os.path.join(base_data_dir, "bounding_boxes.txt")
# Ensure largest_box_size is consistent with Section 2
bounding_boxes = pd.read_csv(
bounding_boxes_file, sep='\s+', header=None, # Updated for FutureWarning
names=["UUID", "bb_x", "bb_y", "bb_width", "bb_height"]
)
largest_box_size = int(max(bounding_boxes["bb_width"].max(), bounding_boxes["bb_height"].max()))
print(f"Recalculated largest box size: {largest_box_size}")
# Filter only the classes that will be used in the model training
important_classes = {'295', '296', '297', '298', '299', '313', '314', '315', '316', '317'}
filtered_data = merged_data[merged_data['class'].isin(important_classes)].reset_index(drop=True)
end_time = time.time()
time_elapsed = round((end_time - start_time),2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
Recalculated largest box size: 1024 Time to run this chunk of code: 0.13 seconds
In [ ]:
Dataset Preparation¶
In [7]:
filtered_data['path'] = filtered_data['path'].apply(lambda x: os.path.join(my_birds_image_dir, x))
# Assuming `filtered_data` is your DataFrame
# Splitting into training and testing based on `is_training_image`
train_data = filtered_data[filtered_data['is_training_image'] == 1]
test_data = filtered_data[filtered_data['is_training_image'] == 0]
# Define directories for organized data
train_dir = os.path.join(my_birds_image_dir, 'train')
test_dir = os.path.join(my_birds_image_dir, 'test')
In [8]:
start_time = time.time()
# Create directories if they don't exist
os.makedirs(train_dir, exist_ok=True)
os.makedirs(test_dir, exist_ok=True)
# Function to copy images to target directory
def copy_images(df, target_dir):
for _, row in df.iterrows():
class_label = row['class']
src_path = os.path.join(data_path, row['path']) # Ensure the full path is resolved
class_dir = os.path.join(target_dir, str(class_label))
os.makedirs(class_dir, exist_ok=True)
dest_path = os.path.join(class_dir, os.path.basename(src_path))
if not os.path.exists(src_path):
print(f"Warning: Source file not found: {src_path}")
continue # Skip this file if it doesn't exist
shutil.copy(src_path, dest_path)
# Copy training and testing images
copy_images(train_data, train_dir)
copy_images(test_data, test_dir)
end_time = time.time()
time_elapsed = round((end_time - start_time),2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
Time to run this chunk of code: 608.95 seconds
In [ ]:
In [9]:
start_time = time.time()
# --- Define Directories ---
# Creating ImageDataGenerators for training and testing
data_gen = ImageDataGenerator(
rescale=1. / 255, # Normalize pixel values to [0, 1]
rotation_range=30, # Randomly rotate images by up to 30 degrees
width_shift_range=0.2, # Randomly shift images horizontally
height_shift_range=0.2, # Randomly shift images vertically
shear_range=0.2, # Apply shearing transformations
zoom_range=0.2, # Randomly zoom in/out on images
horizontal_flip=True, # Randomly flip images horizontally
vertical_flip=False, # Optionally flip images vertically (unlikely for birds)
brightness_range=[0.8, 1.2], # Randomly adjust brightness
fill_mode='nearest' # Handle pixels outside the boundary of the image
)
# Training generators
train_data_shuffled_generator = data_gen.flow_from_directory(
train_dir,
target_size=transfer_model_shp,
batch_size=batch_size_data_load_val,
class_mode='categorical',
shuffle=True
)
train_data_no_shuffle_generator = data_gen.flow_from_directory(
train_dir,
target_size=transfer_model_shp,
batch_size=batch_size_data_load_val,
class_mode='categorical',
shuffle=False
)
# Test generator (using test_dir as a standalone validation directory)
test_data_generator = data_gen.flow_from_directory(
test_dir,
target_size=transfer_model_shp,
batch_size=batch_size_data_load_val,
shuffle=False,
class_mode='categorical'
)
# Check the number of classes
numClasses = len(train_data_shuffled_generator.class_indices) # Use class indices from train generator
print(f"Number of classes: {numClasses}")
end_time = time.time()
time_elapsed = round((end_time - start_time), 2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
Found 456 images belonging to 10 classes. Found 456 images belonging to 10 classes. Found 434 images belonging to 10 classes. Number of classes: 10 Time to run this chunk of code: 0.11 seconds
In [ ]:
In case of a msismatch between training and testing classes¶
There is a mismatch between the number of classes in your training data (11 classes) and the number of classes in your test data (10 classes). This discrepancy causes the shapes of the target (labels) and output (predictions) to differ, leading to a ValueError, down along the code.
Filter Out the Class from train_dir: Remove the directory for the missing class from train_dir to exclude it from the generator.
Re-run the Image Generator Scripts.
In [ ]:
# train_classes = set(train_data_shuffled_generator.class_indices.keys())
# test_classes = set(test_data_generator.class_indices.keys())
# print("Train generator classes: \n", train_classes)
# print("Test generator classes: \n", test_classes)
# if len(train_classes) != len(test_classes):
# missing_classes = train_classes - test_classes
# print(f"Classes in train but not in test: {missing_classes}")
# # Filter Out the Class from train_dir: Remove the directory for the missing class from train_dir to exclude it from the generator.
# for missing_class in missing_classes:
# missing_class_dir = os.path.join(train_dir, missing_class)
# if os.path.exists(missing_class_dir):
# print(f"Removing class: {missing_class}")
# shutil.rmtree(missing_class_dir) # Deletes the directory and its contents
# start_time = time.time()
# # --- Define Directories ---
# # Creating ImageDataGenerators for training and testing
# data_gen = ImageDataGenerator(
# rescale=1. / 255, # Normalize pixel values to [0, 1]
# rotation_range=30, # Randomly rotate images by up to 30 degrees
# width_shift_range=0.2, # Randomly shift images horizontally
# height_shift_range=0.2, # Randomly shift images vertically
# shear_range=0.2, # Apply shearing transformations
# zoom_range=0.2, # Randomly zoom in/out on images
# horizontal_flip=True, # Randomly flip images horizontally
# vertical_flip=False, # Optionally flip images vertically (unlikely for birds)
# brightness_range=[0.8, 1.2], # Randomly adjust brightness
# fill_mode='nearest' # Handle pixels outside the boundary of the image
# )
# # Training generators
# train_data_shuffled_generator = data_gen.flow_from_directory(
# train_dir,
# target_size=transfer_model_shp,
# batch_size=batch_size_data_load_val,
# class_mode='categorical',
# shuffle=True
# )
# train_data_no_shuffle_generator = data_gen.flow_from_directory(
# train_dir,
# target_size=transfer_model_shp,
# batch_size=batch_size_data_load_val,
# class_mode='categorical',
# shuffle=False
# )
# # Test generator (using test_dir as a standalone validation directory)
# test_data_generator = data_gen.flow_from_directory(
# test_dir,
# target_size=transfer_model_shp,
# batch_size=batch_size_data_load_val,
# shuffle=False,
# class_mode='categorical'
# )
# # Check the number of classes
# numClasses = len(train_data_shuffled_generator.class_indices) # Use class indices from train generator
# print(f"Number of classes: {numClasses}")
# end_time = time.time()
# time_elapsed = round((end_time - start_time), 2)
# print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
In [ ]:
In [ ]:
Model Configuration¶
In [10]:
start_time = time.time()
# --- Define Directories ---
cnn = transfer_model(
include_top=False,
weights="imagenet",
input_shape=transfer_model_input_shape,
)
cnn.trainable = False ## using FREEZing loop
cnn.summary()
end_time = time.time()
time_elapsed = round((end_time - start_time), 2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5 58889256/58889256 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Model: "vgg16"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ input_layer (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block1_conv1 (Conv2D) │ (None, 224, 224, 64) │ 1,792 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block1_conv2 (Conv2D) │ (None, 224, 224, 64) │ 36,928 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block1_pool (MaxPooling2D) │ (None, 112, 112, 64) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block2_conv1 (Conv2D) │ (None, 112, 112, 128) │ 73,856 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block2_conv2 (Conv2D) │ (None, 112, 112, 128) │ 147,584 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block2_pool (MaxPooling2D) │ (None, 56, 56, 128) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block3_conv1 (Conv2D) │ (None, 56, 56, 256) │ 295,168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block3_conv2 (Conv2D) │ (None, 56, 56, 256) │ 590,080 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block3_conv3 (Conv2D) │ (None, 56, 56, 256) │ 590,080 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block3_pool (MaxPooling2D) │ (None, 28, 28, 256) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block4_conv1 (Conv2D) │ (None, 28, 28, 512) │ 1,180,160 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block4_conv2 (Conv2D) │ (None, 28, 28, 512) │ 2,359,808 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block4_conv3 (Conv2D) │ (None, 28, 28, 512) │ 2,359,808 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block4_pool (MaxPooling2D) │ (None, 14, 14, 512) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block5_conv1 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block5_conv2 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block5_conv3 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ block5_pool (MaxPooling2D) │ (None, 7, 7, 512) │ 0 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 14,714,688 (56.13 MB)
Trainable params: 0 (0.00 B)
Non-trainable params: 14,714,688 (56.13 MB)
Time to run this chunk of code: 3.37 seconds
In [ ]:
In [11]:
start_time = time.time()
# global_avg_pooling_layer = GlobalAveragePooling2D()
flatten_layer = tf.keras.layers.Flatten()
dense_layer_1 = Dense(500, activation='relu', kernel_regularizer=l2(0.01))
batch_norm_1 = BatchNormalization() # Batch normalization after the first dense layer
dropout_layer_1 = Dropout(dropout_rate) # Add dropout after the first dense layer
dense_layer_2 = Dense(250, activation='relu', kernel_regularizer=l2(0.01))
batch_norm_2 = BatchNormalization() # Batch normalization after the second dense layer
dropout_layer_2 = Dropout(dropout_rate) # Add dropout after the second dense layer
dense_layer_3 = Dense(30, activation='relu', kernel_regularizer=l2(0.01))
dropout_layer_3 = Dropout(dropout_rate) # Add dropout after the third dense layer
prediction_layer = Dense(numClasses, activation='softmax') # Final output layer
Classifier = Sequential([
cnn, # Convolutional base (e.g., VGG16, pre-trained or custom)
flatten_layer,
dense_layer_1, # First dense layer
batch_norm_1, # Batch normalization for stability
dropout_layer_1, # Dropout after first dense layer
dense_layer_2, # Second dense layer
batch_norm_2, # Batch normalization for stability
dropout_layer_2, # Dropout after second dense layer
dense_layer_3, # Third dense layer
dropout_layer_3, # Dropout after third dense layer
prediction_layer # Output layer
])
Classifier.summary()
end_time = time.time()
time_elapsed = round((end_time - start_time), 2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ vgg16 (Functional) │ (None, 7, 7, 512) │ 14,714,688 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ flatten (Flatten) │ (None, 25088) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense (Dense) │ (None, 500) │ 12,544,500 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ batch_normalization │ (None, 500) │ 2,000 │ │ (BatchNormalization) │ │ │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 500) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 250) │ 125,250 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ batch_normalization_1 │ (None, 250) │ 1,000 │ │ (BatchNormalization) │ │ │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout_1 (Dropout) │ (None, 250) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 30) │ 7,530 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout_2 (Dropout) │ (None, 30) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_3 (Dense) │ (None, 10) │ 310 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 27,395,278 (104.50 MB)
Trainable params: 12,679,090 (48.37 MB)
Non-trainable params: 14,716,188 (56.14 MB)
Time to run this chunk of code: 0.09 seconds
In [ ]:
In [12]:
class EpochMetricsLogger(Callback):
def on_epoch_end(self, epoch, logs=None):
if logs is None:
logs = {}
# Extract and log metrics
metrics = {key: logs.get(key, 'nan') for key in ['accuracy', 'loss', 'val_accuracy', 'val_loss']}
# Safeguard for NaN detection in loss
val_loss = metrics['val_loss']
if isinstance(val_loss, float) and math.isnan(val_loss):
logger.warning(f"NaN detected in validation loss at epoch {epoch + 1}. Stopping training.")
self.model.stop_training = True
return
# Log metrics
logger.info(
f"Epoch {epoch + 1}: "
f"Train Accuracy: {metrics['accuracy']:.4f}, "
f"Train Loss: {metrics['loss']:.4f}, "
f"Validation Accuracy: {metrics['val_accuracy']:.4f}, "
f"Validation Loss: {metrics['val_loss']:.4f}"
)
# Define callbacks
callbacks = [
ModelCheckpoint(
filepath='best_vgg16_model.weights.keras',
monitor='val_accuracy',
save_best_only=True,
verbose=1
),
EarlyStopping(
monitor='val_accuracy',
patience=50,
restore_best_weights=True,
verbose=1
),
ReduceLROnPlateau(
monitor='val_loss',
factor=0.5,
patience=5,
min_lr=1e-4,
verbose=1
),
EpochMetricsLogger()
]
In [ ]:
Training and Saving the Model¶
In [ ]:
# More options in Compiling and Training CNN
start_time = time.time()
opt = tf.keras.optimizers.SGD(learning_rate=learning_rate_val)
Classifier.compile(
loss=tf.keras.losses.categorical_crossentropy,
optimizer=opt,
metrics=['accuracy']
)
history = Classifier.fit(
train_data_shuffled_generator, # Training generator
epochs=MyEpochs, # Number of epochs
validation_data=test_data_generator, # Validation generator
shuffle=True, # Shuffling handled by the training generator
verbose=0, # Verbose mode for training updates
batch_size = batch_size_val,
callbacks=callbacks,
)
end_time = time.time()
time_elapsed = round((end_time - start_time), 2)
print(f"\n\n\n Time to run this chunk of code: {time_elapsed} seconds")
In [ ]:
# Create a figure with 2 subplots side by side
fig, axs = plt.subplots(1, 2, figsize=(12, 5))
# Plot training and validation accuracy on the first subplot
axs[0].plot(history.history['accuracy'], label='Train Accuracy')
axs[0].plot(history.history['val_accuracy'], label='Validation Accuracy')
axs[0].set_xlabel('Epochs')
axs[0].set_ylabel('Accuracy')
axs[0].legend()
axs[0].set_title('Training and Validation Accuracy')
# Plot training and validation loss on the second subplot
axs[1].plot(history.history['loss'], label='Train Loss')
axs[1].plot(history.history['val_loss'], label='Validation Loss')
axs[1].set_xlabel('Epochs')
axs[1].set_ylabel('Loss')
axs[1].legend()
axs[1].set_title('Training and Validation Loss')
# Adjust layout and display the plots
plt.tight_layout()
plt.show()
In [ ]:
In [ ]:
Model Evaluation and Plotting¶
In [13]:
best_vgg16_model_location = base_data_dir + '/best_vgg16_model.weights.keras'
print(best_vgg16_model_location)
/content/drive/My Drive/NaBirds/best_vgg16_model.weights.keras
In [14]:
Classifier.load_weights(best_vgg16_model_location)
print("Weights loaded successfully!")
Weights loaded successfully!
In [ ]:
In [15]:
class_labels = list(train_data_no_shuffle_generator.class_indices.keys())
print(class_labels)
['295', '296', '297', '298', '299', '313', '314', '315', '316', '317']
In [ ]:
In [19]:
train_data_shuffled_generator.reset()
test_data_generator.reset()
predicted_scores = Classifier.predict(train_data_no_shuffle_generator, verbose=1)
predicted_labels = predicted_scores.argmax(axis=1)
train_labels = train_data_no_shuffle_generator.labels
print(train_labels)
print(predicted_labels)
acc_score = accuracy_score(train_labels, predicted_labels)
CFM = confusion_matrix(train_labels, predicted_labels)
print("\n", "Accuracy: " + str(format(acc_score,'.3f')))
print("\n", "CFM: \n", confusion_matrix(train_labels, predicted_labels))
print("\n", "Classification report: \n", classification_report(train_labels, predicted_labels, target_names=class_labels))
46/46 ━━━━━━━━━━━━━━━━━━━━ 8s 164ms/step [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9] [0 1 0 0 2 0 0 7 0 5 0 0 0 0 1 0 0 0 0 2 0 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 8 1 1 1 8 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 7 2 2 2 2 2 2 2 2 2 2 2 2 8 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 2 2 2 2 2 2 2 2 1 2 2 2 2 2 3 4 3 3 4 4 4 3 3 3 3 3 9 3 3 3 3 3 8 3 3 3 4 3 3 3 3 3 4 3 3 3 3 3 3 3 3 4 4 4 4 4 4 3 4 4 4 4 4 4 3 4 4 4 4 4 4 3 4 4 4 4 4 4 4 4 4 3 4 4 4 4 7 4 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 1 7 5 9 5 5 5 5 5 4 7 5 0 5 5 5 5 4 5 0 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 2 6 6 6 6 7 6 6 6 6 6 6 6 6 6 2 6 6 6 6 6 6 6 6 6 6 2 6 3 6 6 6 6 6 7 7 7 7 7 7 7 6 7 7 7 7 7 7 8 7 7 7 7 7 0 7 7 8 9 7 7 8 7 4 7 7 7 7 7 7 7 7 7 7 8 2 7 7 7 7 7 7 1 7 7 7 7 7 7 7 7 4 8 7 8 7 8 8 8 8 5 4 8 8 8 8 8 8 8 8 8 8 7 8 3 8 8 8 7 8 8 8 8 9 7 8 8 8 8 8 8 7 8 0 8 8 8 8 8 8 8 0 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 8 9 9 9 9 9 9 9 9 5 9 9 9 8 9 9 9 9 9 9 9 9] Accuracy: 0.851 CFM: [[17 3 2 0 0 1 0 1 0 0] [ 0 32 0 0 0 0 0 0 2 0] [ 1 1 45 0 0 0 0 1 1 0] [ 0 0 0 29 6 0 0 0 1 1] [ 0 0 0 5 54 0 0 1 0 0] [ 3 1 0 0 2 12 0 2 0 1] [ 0 0 3 1 0 0 55 1 0 0] [ 1 1 1 0 2 0 1 48 5 1] [ 2 0 0 1 1 1 0 5 40 1] [ 0 0 0 1 0 1 0 0 2 56]] Classification report: precision recall f1-score support 295 0.71 0.71 0.71 24 296 0.84 0.94 0.89 34 297 0.88 0.92 0.90 49 298 0.78 0.78 0.78 37 299 0.83 0.90 0.86 60 313 0.80 0.57 0.67 21 314 0.98 0.92 0.95 60 315 0.81 0.80 0.81 60 316 0.78 0.78 0.78 51 317 0.93 0.93 0.93 60 accuracy 0.85 456 macro avg 0.84 0.83 0.83 456 weighted avg 0.85 0.85 0.85 456
In [20]:
train_classification_report = classification_report(train_labels, predicted_labels, target_names=class_labels)
# print(train_classification_report)
# Parse the classification report string
train_lines = train_classification_report.strip().split("\n")
train_data = []
for line in train_lines[2:]: # Skip headers
parts = line.split()
if len(parts) < 5: # Skip lines that don't follow the expected format
continue
class_name = " ".join(parts[:-4]) # Handle multi-word class names
support = int(parts[-1])
train_data.append((class_name, support))
# Convert to DataFrame for easier handling
train_support_df = pd.DataFrame(train_data, columns=["Class", "Support"])
# Convert to horizontal format
train_support_horizontal = train_support_df[:10].set_index("Class").transpose()
print(train_support_horizontal)
Class 295 296 297 298 299 313 314 315 316 317 Support 24 34 49 37 60 21 60 60 51 60
In [ ]:
In [ ]:
In [21]:
test_predicted_scores = Classifier.predict(test_data_generator, verbose=1)
test_predicted_labels = test_predicted_scores.argmax(axis=1)
test_labels = test_data_generator.labels
print(test_labels)
#print(predicted_scores)
print(test_predicted_labels)
test_acc_score = accuracy_score(test_labels, test_predicted_labels)
test_CFM = confusion_matrix(test_labels, test_predicted_labels)
print("\n", "Accuracy: " + str(format(test_acc_score,'.3f')))
print("\n", "CFM: \n", confusion_matrix(test_labels, test_predicted_labels))
print("\n", "Classification report: \n", classification_report(test_labels, test_predicted_labels, target_names=class_labels))
1/44 ━━━━━━━━━━━━━━━━━━━━ 7s 169ms/step
/usr/local/lib/python3.10/dist-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:122: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored. self._warn_if_super_not_called()
44/44 ━━━━━━━━━━━━━━━━━━━━ 10s 237ms/step [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9] [1 0 0 0 0 0 0 0 0 1 0 0 0 0 4 0 0 0 0 1 0 0 1 0 0 1 1 0 0 1 1 1 1 1 1 6 1 1 3 1 1 1 1 1 1 7 1 6 6 1 1 1 1 1 1 1 6 1 8 2 2 2 1 2 9 0 2 2 1 2 2 2 2 7 2 2 2 2 2 2 2 2 2 8 2 2 2 9 9 2 2 2 8 2 2 2 9 4 4 4 4 3 3 4 3 4 4 4 3 6 3 3 4 3 4 4 3 4 4 3 3 4 3 4 4 7 4 3 4 4 4 4 3 3 7 4 6 4 4 4 4 4 4 4 3 4 3 3 4 9 4 4 4 4 4 4 3 4 4 4 4 4 4 4 4 3 4 4 4 4 4 4 4 4 3 4 4 3 4 4 4 4 4 4 4 3 4 5 7 1 5 4 5 7 7 5 4 5 2 3 5 5 5 1 5 7 7 5 7 8 5 5 5 0 6 6 6 6 6 6 6 6 6 6 9 6 6 6 6 6 6 6 6 6 6 6 6 6 6 8 6 6 6 7 6 6 6 4 8 6 6 9 6 6 7 6 6 6 6 6 6 6 6 6 6 6 6 6 2 6 6 6 6 6 8 9 7 7 7 7 1 1 7 7 7 5 7 7 8 7 7 7 5 7 7 7 7 7 7 7 7 7 8 7 7 7 0 7 7 7 9 7 5 7 7 2 7 0 7 0 1 1 7 7 4 1 1 7 7 1 9 7 8 2 8 6 8 4 8 8 8 8 5 7 3 8 8 8 2 7 0 8 1 0 7 2 7 8 8 8 7 8 8 7 8 8 8 8 2 8 8 0 8 8 9 0 6 9 9 9 6 0 9 4 9 9 0 9 9 9 9 9 9 0 9 8 9 9 9 8 9 7 9 9 9 9 9 3 9 9 9 0 8 5 6 9 9 9 9 9 7 8 9 9 5 9 9 8 6 9 9 9 9 2] Accuracy: 0.675 CFM: [[22 6 0 0 1 0 0 0 0 0] [ 0 23 0 1 0 0 4 1 1 0] [ 1 2 28 0 0 0 0 1 2 4] [ 0 0 0 11 17 0 1 1 0 0] [ 0 0 0 11 46 0 1 1 0 1] [ 0 2 1 1 2 13 0 6 1 0] [ 1 0 1 0 1 0 51 2 2 2] [ 3 6 1 0 1 3 1 36 3 2] [ 3 2 4 1 1 1 1 7 24 1] [ 5 0 1 1 1 2 4 2 5 39]] Classification report: precision recall f1-score support 295 0.63 0.76 0.69 29 296 0.56 0.77 0.65 30 297 0.78 0.74 0.76 38 298 0.42 0.37 0.39 30 299 0.66 0.77 0.71 60 313 0.68 0.50 0.58 26 314 0.81 0.85 0.83 60 315 0.63 0.64 0.64 56 316 0.63 0.53 0.58 45 317 0.80 0.65 0.72 60 accuracy 0.68 434 macro avg 0.66 0.66 0.65 434 weighted avg 0.68 0.68 0.67 434
In [22]:
test_classification_report = classification_report(test_labels, test_predicted_labels, target_names=class_labels)
# print(test_classification_report)
# Parse the classification report string
test_lines = test_classification_report.strip().split("\n")
test_data = []
for line in test_lines[2:]: # Skip headers
parts = line.split()
if len(parts) < 5: # Skip lines that don't follow the expected format
continue
class_name = " ".join(parts[:-4]) # Handle multi-word class names
support = int(parts[-1])
test_data.append((class_name, support))
# Convert to DataFrame for easier handling
test_support_df = pd.DataFrame(test_data, columns=["Class", "Support"])
# Convert to horizontal format
test_support_horizontal = test_support_df[:10].set_index("Class").transpose()
print(test_support_horizontal)
Class 295 296 297 298 299 313 314 315 316 317 Support 29 30 38 30 60 26 60 56 45 60
In [ ]:
In [23]:
# Compute confusion matrices
train_cfm = confusion_matrix(train_labels, predicted_labels)
test_cfm = confusion_matrix(test_labels, test_predicted_labels)
# Create a figure with two subplots
fig, axs = plt.subplots(1, 2, figsize=(20, 8), sharey=True)
# Training confusion matrix
sns.heatmap(train_cfm, annot=True, fmt='d', cmap='Blues', xticklabels=class_labels, yticklabels=class_labels, ax=axs[0])
axs[0].set_title("Training Confusion Matrix")
axs[0].set_xlabel("Predicted Labels")
axs[0].set_ylabel("True Labels")
# Testing confusion matrix
sns.heatmap(test_cfm, annot=True, fmt='d', cmap='Oranges', xticklabels=class_labels, yticklabels=class_labels, ax=axs[1])
axs[1].set_title("Testing Confusion Matrix")
axs[1].set_xlabel("Predicted Labels")
# Adjust layout and show the plots
plt.tight_layout()
plt.show()
In [ ]:
In [24]:
# Training metrics
train_report = classification_report(train_labels, predicted_labels, target_names=class_labels, output_dict=True)
train_report_df = pd.DataFrame(train_report).transpose()
# Testing metrics
test_report = classification_report(test_labels, test_predicted_labels, target_names=class_labels, output_dict=True)
test_report_df = pd.DataFrame(test_report).transpose()
# Extract class-wise metrics
train_precision = [train_report[class_name]['precision'] for class_name in class_labels]
test_precision = [test_report[class_name]['precision'] for class_name in class_labels]
train_recall = [train_report[class_name]['recall'] for class_name in class_labels]
test_recall = [test_report[class_name]['recall'] for class_name in class_labels]
train_f1 = [train_report[class_name]['f1-score'] for class_name in class_labels]
test_f1 = [test_report[class_name]['f1-score'] for class_name in class_labels]
# Bar chart comparison
y = np.arange(len(class_labels)) # Position of class labels
width = 0.35 # Width of the bars
# Create subplots side by side
fig, axs = plt.subplots(1, 3, figsize=(18, 8), sharey=True)
# Precision comparison
axs[0].barh(y - width / 2, train_precision, height=width, label="Train")
axs[0].barh(y + width / 2, test_precision, height=width, label="Test")
axs[0].set_yticks(y)
axs[0].set_yticklabels(class_labels)
axs[0].set_xlabel("Precision Value")
axs[0].set_title("Precision")
axs[0].invert_yaxis() # Align class labels from top to bottom
# Recall comparison
axs[1].barh(y - width / 2, train_recall, height=width)
axs[1].barh(y + width / 2, test_recall, height=width)
axs[1].set_xlabel("Recall Value")
axs[1].set_title("Recall")
# F1-score comparison
axs[2].barh(y - width / 2, train_f1, height=width)
axs[2].barh(y + width / 2, test_f1, height=width)
axs[2].set_xlabel("F1-Score Value")
axs[2].set_title("F1-Score")
# Add a single legend for all subplots, positioned slightly below the title
handles, labels = axs[0].get_legend_handles_labels()
fig.legend(handles, labels, loc="upper center", ncol=2, fontsize=12, bbox_to_anchor=(0.5, 0.92))
# Add a main title
fig.suptitle("VGG16 Model Classification Comparison for the 10 Chosen Bird Class Labels", fontsize=16, y=0.96)
# Adjust layout and show the plots
plt.tight_layout(rect=[0, 0, 1, 0.90]) # Adjust to leave space for the title and legend
plt.show()
In [ ]:
In [ ]:
In [25]:
# Binarize test labels
test_labels_binarized = label_binarize(test_labels, classes=range(len(class_labels)))
# Compute ROC curve and AUC for each class
plt.figure(figsize=(10, 8))
for i, class_name in enumerate(class_labels):
fpr, tpr, _ = roc_curve(test_labels_binarized[:, i], test_predicted_scores[:, i])
roc_auc = auc(fpr, tpr)
plt.plot(fpr, tpr, label=f"{class_name} (AUC = {roc_auc:.2f})")
plt.plot([0, 1], [0, 1], 'k--') # Diagonal line for random guessing
plt.title("ROC-AUC Curve for Each Class")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.legend()
plt.show()
In [ ]: