Assignment 2: Training on a new dataset and measuring model performance
Due: 02/15/2024
Credit: 5 points
The goal of this assignment is to train a model on a custom dataset. Very often, we find tutorials online which make it easy to train on standard datasets like CIFAR-10, MNIST or ImageNet. However, for many projects you will need to train on non-standard datasets, which do not necessarily cooperate with PyTorch out of the box and require a bit of engineering. In Tutorial 2 (https://github.com/MorganBDT/pytorch_data_loader_tutorial), we went over an example of how to write a custom Dataset class in pytorch. In this assignment, you will be developing your own custom Dataset class for a new dataset.
Learning outcomes:
(1) Writing a custom data loader
(2) Test your skills on a new dataset
Instructions::
1. Go over the Tutorial 2 codebase to understand how to write your own Dataset class. A good introduction to pytorch Datasets and Dataloaders can also be found here: https://pytorch.org/tutorials/beginner/basics/data_tutorial.html2. Download the Biased-Cars dataset here: https://dataverse.harvard.edu/api/access/datafile/5347218 and the corresponding labels here: https://dataverse.harvard.edu/api/access/datafile/5793795
3. Unzip the data. The data is organized into folders with images of cars in a synthetic 3D city. Each image has 4 labels - scale, rotation, car color, and car model (in this order). These labels are stored in the dictionary att_dict_simplified.p (python pickle file).
4. Write a new Dataset class that does the following: for all images which have a label in att_dict_simplified.p, read the images from the given directories and the labels from the dictionary. For each sub-folder under biased_cars_1/, training data is located under 'train/images/' and validation data is under 'val/images/'. Images under the 'labels' folders can be ignored for this assignment. After loading the val and train datasets using your Dataset class, train a model to classify these according to car model (there are 18 different car model classes in the dictionary). Ignore the first three labels for scale, rotation, and color. Train it for 5 epochs using the ResNet18 architecture, and generate plots showing its performance as described below. Please note: to receive full credit, you must write your own Dataset class and NOT use ImageFolder or similar. Even though “shortcuts” like ImageFolder may be quite useful in many situations in practice, the intention here is for you to fully understand how the Dataset class works.
5. Answer the two questions (see “Grade breakdown and questions” below).
Tips and tricks:
1. We recommend using Google Colab with a GPU runtime enabled (see also, the config.py file in the tutorial 2 codebase), as training may take a long time on a CPU. Since Google limits how much time you get with the GPU, it’s best to use a CPU runtime while you are developing and testing your code initially, and then switch to GPU once everything works and you are ready to train.
2. Optionally, feel free to experiment with hyperparameter tuning (e.g., batch sizes, learning rates, learning rate decay). We are not grading based on the accuracy of your trained model, but we would expect to see above-chance performance and some improvement during training.
Grade breakdown and questions:
Please submit a colab/Jupyter notebook as a PDF which shows:
1. The code of your custom Dataset class (along with any other necessary code for training your model) [2 points]
2. Some sample images, with labels displayed, loaded using your custom Dataset class. [0.5 points]
3. A printout of some basic statistics of the training and validation sets. How many images are there in each class, and how many images are there in total? (for training and validation sets separately). You can list each class by its integer number (i.e., class 0, class 1, etc.) [0.5 points]
4. Plots showing accuracy and loss on the training and validation sets of Biased-Cars, across 5 or more training epochs. There should be four line plots in total - they can be plotted separately or together on one plot (they should be clearly labeled in either case). [1 point]
Additionally, please answer the following questions in 1-2 sentences each (in a “text” cell in the notebook, or appended to the end of the pdf):
Question 1. Examine your accuracy and loss plots.
(a) Do the overall trends you observe for loss and accuracy make sense? (i.e., should they be increasing or decreasing?) [0.25 points]
(b) What would you expect to happen if you trained the model for a huge number of epochs? (e.g., 1 million epochs). Specifically, what do you think would happen to the training loss, the validation loss, the training accuracy, and the validation accuracy? (hint: tutorial 2) [0.25 points]
Question 2. When evaluating a model’s performance, it is important to keep in mind what the model’s accuracy would be “at chance,” i.e. if it learned nothing during training.
(a) Let’s say the model truly learns nothing, and guesses one of the 18 classes completely at random for each image. What would you expect its accuracy to be on the validation set? [0.25 points]
(b) Now let’s say the model learns only a trivial piece of information about the dataset: which class is the most common. What would the model’s accuracy be on the validation set if it always guessed the class that is most common in the training set? [0.25 points]
