Behavioural Cloning

The objective of the project is to effectively teach a vehicle to drive autonomously in a simulated
driving application by using deep learning. We are using TensorFlow and Keras to predict the
steering angle from images of the road.

Goals
Goals/Steps of the project are as follows:

▪ Use the simulator to collect data of good driving behaviour

▪ Build, a convolution neural network in Keras that predicts steering angles from images
▪ Train and validate the model with a training and validation set
▪ Test that the model successfully drives around track one without leaving the road
▪ Summarize the results with a written report

Files
This project includes the following files:

▪ model.py: Contains the script to pre-process the image, generate data, create and train the
model.
▪ drive.py: For driving the car in autonomous mode.
▪ model.h5: Containing a trained convolution neural network.
▪ video.py: Script to generate the video from autonomous driving images.
▪ video.mp4: Video of autonomous driving mode.
▪ writeup_report.pdf: Writeup report of the project.

Dataset
The project uses the dataset provided by Udacity. The dataset contains 24108 images of size
160x320x3

Steps
The code can be divided into the following major steps:

1. Load and Process Data

2. Augment Data
3. Process Image
4. Define Model
5. Train, Save and Test Model

1. Load and Process Data

The dataset provided by Udacity is used for this project. First step is to read the image path and
steering angle from the CSV file. Later all this data is shuffled to better train the model.
Next, we split the data into training and validation set having 80% and 20% images each.

Images are divided into multiple batches of 32 images. After this we augment the images and
steering angle to send it to the model for training/verification. In this step we make the use of
generators to make the project more memory efficient.

2. Augment Data
In this step the image is selected and processed.

First, we randomly select an image from left, right and centre image’s. If the images belong to left or
right camera then 0.25 correction is added or removed from it respectively.

To make the dataset more diverse and less diverse towards left turn, few images are flipped
randomly with the probability of 0.5

3. Process Image
To make the model compatible with various light conditions, brightness of images is randomly
changed.

First the images are converted from RGB to HSV. A random brightness parameter is generated and
multiplied to the image. After this the image is converted back to the RGB format.

The image is then resized to 320x160

4. Define Model
The architecture followed in this project was inspired by NVIDIA. However, some changes were
made to avoid overfitting and adding non-linearity for prediction improvement.

The regression model started with an image of 160x320x3. Lambda layer with lambda function was
added to normalize the image. Also applied a Cropping layer to remove 70 pixels from top and 25
pixels from bottom to remove the non-required areas of image.

In further layers, I applied 3 convolution layers having filters of 32, 16 and 16 with ELU as activation.
Also added 2x2 max pooling to keep the model small enough to quickly run on GPU.

Two dropouts of 0.4 were added after the Convolution layer 2 and 3 and one dropout of 0.3 was
added after fully connected layer. Initially I was very aggressive with the dropouts but this impacted
car’s performance on track so decreased the values to till the parameters worked fine.

The output was flattened in the next step and fully connected with 1024 node layer. Later the layers
were decreased from 1024 to 512 to 1. The final output was a single value as we need to predict the
steering angle.
 Layer 1
•Lambda Layer to normalize the data
•Cropping Layer to crop 70 pixels from top and 25 pixels from bottom of the image

Layer 2
•Convolution Layer using 32 5x5 Filters
•ELU Activation

Layer 3
•Convolution Layer using 16 3x3 Filters
•ELU Activation
•Dropout by 0.4
•Max Pooling 2D of 2x2

Layer 4
•Convolution Layer using 16 3x3 Filters
•ELU Activation
•Dropout by 0.4

Layer 5
•Flatten Layer

Layer 6
•Fully Connected Layer with 1024 neurons
•Dropout by 0.3
•ELU Activation

Layer 7
•Fully Connected Layer with 512 neurons
•ELU Activation

Layer 8
•Fully Connected Layer with 1 neuron

Layer (Type) Output Shape Parameters

lambda_1 (Lambda) (None, 160, 320, 3) 0
cropping2d_1 (Cropping2D) (None, 65, 320, 3) 0
conv2d_1 (Conv2D) (None, 33, 160, 32) 2432
elu_1 (ELU) (None, 33, 160, 32) 0
conv2d_2 (Conv2D) (None, 31, 158, 16) 4624
elu_2 (ELU) (None, 31, 158, 16) 0
dropout_1 (Dropout) (None, 31, 158, 16) 0
 max_pooling2d_1 (MaxPooling2D) (None, 15, 79, 16) 0
conv2d_3 (Conv2D) (None, 13, 77, 16) 2320
elu_3 (ELU) (None, 13, 77, 16) 2320
dropout_2 (Dropout) (None, 13, 77, 16) 0
flatten_1 (Flatten) (None, 16016) 0
dense_1 (Dense) (None, 1024) 16401408
dropout_3 (Dropout) (None, 1024) 0
elu_4 (ELU) (None, 1024) 0
dense_2 (Dense) (None, 512) 524800
elu_5 (ELU) (None, 512) 0
dense_3 (Dense) (None, 1) 513
Total params: 16,936,097
Trainable params: 16,936,097
Non-trainable params: 0

5. Train, Save and Test Model

To train the model Keras fit_generator is used. In this training_generator contains the training
dataset and validation_generator contains validation dataset. Parameters used are:

Epochs: 3
Optimizer: Adam
Learning Rate: 0.001
Loss Function: MSE

We used model.py file to generate a model that predicts the steering angle using the dashboard
camera images. Command to generate the model:

python model.py

The model is saved as model.h5 file

Result:

Epoch 1/3
200/200 [==============================] - 31s - loss: 0.6502 - val_loss: 0.0020
Epoch 2/3
200/200 [==============================] - 25s - loss: 0.0482 - val_loss:
5.8204e-04
Epoch 3/3
200/200 [==============================] - 23s - loss: 0.0451 - val_loss:
2.5030e-04

Driving
Post creation of the model it is used to simulate driving on the test track using autonomous mode.
So, the drive.py script uses model.h5 file for training and the complete command is:

python drive.py model.h5 run1

The forth argument run1 is the directory which saves the images captured during autonomous
mode. The images stored in this folder are named as per the timestamp of when the images were
captured.

Video
Script video.py is used to create the video by combining images present in run1 directory. The video
name will be similar to the directory name i.e. run1.mp4

Also, the FPS (frames per second) of the video can also be mentioned. Complete command is:

python video.py run1 -- fps 50

In this project the video is captured at 50 FPS.