writeup.md

Advanced Lane Finding

Writeup by Hannes Bergler

---

Advanced Lane Finding Project

The goals / steps of this project were the following:

• Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
• Apply a distortion correction to raw images.
• Use color transforms, gradients, etc., to create a thresholded binary image.
• Apply a perspective transform to rectify binary image ("birds-eye view").
• Detect lane pixels and fit to find the lane boundary.
• Determine the curvature of the lane and vehicle position with respect to center.
• Warp the detected lane boundaries back onto the original image.
• Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

Rubric Points

Here I will consider the rubric points individually and describe how I addressed each point in my implementation.

---

I. Submission Files

My project includes the following files:

• writeup.md - summarizing the project results (You're reading it!)
• Advanced_Lane_Finding_P4.ipynb - Jupyter notebook containing the project's python code
• output_images - folder containing all output images
• output_videos - folder containing the output videos

II. Camera Calibration

The code for this step is contained in the second code cell of the Jupyter notebook Advanced_Lane_Finding_P4.ipynb located in the project's base directory.

I start by preparing "object points", which will be the (x, y, z) coordinates of the chessboard corners in the world. Here I am assuming the chessboard is fixed on the (x, y) plane at z=0, such that the object points are the same for each calibration image. Thus, objp is just a replicated array of coordinates, and objpoints will be appended with a copy of it every time I successfully detect all chessboard corners in a test image. imgpoints will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.

I then used the output objpoints and imgpoints to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera() function. I applied this distortion correction to the test image using the cv2.undistort() function and obtained this result:

III. Pipeline

1. Distortion Correction

To demonstrate this step, I will describe how I apply the distortion correction to one of the test images like this one:

Here is the undistorted version of this test image, which was generated by the cv2.undistort() function using the camera calibration matrix and the distortion coefficients as described above:

2. Identifying Lane Line Pixels

I used a combination of color and gradient thresholds to generate a binary image (thresholding steps can be found in the function lane_filter() in code block four of the Jupyter notebook). Here's an example of my output for this step.

I then masked the resulting binary image using a trapezoid and a triangle (defined in the pipeline() function in code block six of the Jupyter notebook) to drop all pixels outside of my region of interest using the region_of_interest() function in code block four. Here's an example of a masked binary image:

3. Perspective Transform

I transformed the image to birds-eye view using the cv2.warpPerspective() function, see code block six in the Jupyter notebook. The required transform matrix M was calculated using the function cv2.getPerspectiveTransform().

The source (src) and destination (dst) points for the perspective transform were defined as follows:

dst_x_offset = 300
src = np.float32([[img_size_x/2 - 56, 456],[img_size_x/2 + 56, 456], \
                  [img_size_x/2 + 446, 719],[img_size_x/2 - 446, 719]])
dst = np.float32([[img_size_x/2 - dst_x_offset, 0],[img_size_x/2 + dst_x_offset, 0], \
                  [img_size_x/2 + dst_x_offset, 719],[img_size_x/2 - dst_x_offset, 719]])

This resulted in the following source and destination points:

Source	Destination
584, 456	340, 0
696, 456	940, 0
1086, 719	940, 719
194, 719	340, 719

Here you can see the binary masked image from above after the perspective transform:

4. Identify Lane Line Pixels And Fit Their Positions With A Polynomial

I then applied a method called 'sliding window search' on the binary warped image to identify the lane line pixels, see function fit_lines() in code block no. four. As starting points for the search I used the maxima of a histogram taken along all the columns in the lower half of the image. The maximum of the left half of the histogram was taken as the starting point for the left line's sliding window search. The same goes for the right half and the right line.

Then I fit my lane lines with a 2nd order polynomial over the line pixels identified by the sliding window method for both the left and the right line. The result can be seen in the following image:

5. Calculate Radius Of Curvature

I calculated the radius of curvature in meters for each line in the method calc_values_for_current_fit() of the Line class in code block five of the Jupyter notebook. I evaluated the line curvature at the lowest point of the image, closest to the vehicle. For the overall lane curvature I took the mean of the left and right line's curvature.

6. Plot The Detected Lane Back Down Onto The Road

I implemented this step in the pipeline() function in code block six of the Jupyter notebook.

As an example, here is my final result for the same image used above:

7. Applying The Pipeline To Videos

For videos, I used the exact same (pipeline()) as for images. I calculated the lane lines frame by frame. Before outputting the detected lane, I carried out some sanity checks on the lines, see validate_current_fit() method of the Line class. If the checks were passed, the individual lines were appended to a list of recent fits (self.recent_fits). The lane lines seen in the video output are the mean of those recent fits for both the left and the right line.

Here's a link to my video result. And here you can find the (imperfect) output of my pipeline for the challenge video.

IV. The Classes Line And Lane

The classes Line and Lane keep track of all the interesting parameters that the pipeline detects from frame to frame. See code cell no. 5 in the Jupyter notebook.

1. The Line Class

Here's the python code of the Line class and it's __init__() function including all member variables of the class, the complete code including all methods can be found in code cell five of the Jupyter notebook:

class Line():
    """
    A class to receive the characteristics of each line detection.
    This class also defines a couple of methods to handel this data.
    """
    def __init__(self, lane, position, hist_size=4, failed_max=5):
        # lane object which holds this line and the other line
        self.lane = lane
        # lane position, e.g. 'left' or 'right'
        self.position = position
        # number of recent fits that will be stored
        self.hist_size = hist_size
        # maximum number of failed detections in a row before the search history
        # will be deleted and the search will be started from scratch
        self.failed_max = failed_max
        
        # history of the last n fits...
        
        # was the line detected in the last iteration?
        self.detected = False
        # number of failed line detections in a row
        self.failed_count = 0
        # radius of curvature of the last n fits of the line in meters
        self.recent_radius = []
        # average radius of curvature of the last n fits of the line in meters
        self.best_radius = None
        # the mean x values of the last n fits of the line
        self.recent_x_mean = []
        # average mean x values of the fitted line over the last n iterations
        self.best_x_mean = None
        # polynomial coefficients of the last n fits of the line
        self.recent_fits = []
        # polynomial coefficients averaged over the last n iterations
        self.best_fit = None
        
        # values of current fit...
        
        # polynomial coefficients for the most recent fit
        self.current_fit = [np.array([False])]
        # x values for detected line pixels
        self.allx = None
        # y values for detected line pixels
        self.ally = None
        # radius of curvature of the line in meters
        self.radius_of_curvature = None
        # distance in pixels of vehicle center from the line
        self.line_base_pos = None

2. The Lane Class

Here's the python code of the Lane class and it's __init__() function including all member variables of the class (most importantly the two Line objects self.left_line and self.right_line). The complete code including all methods can be found in code cell five of the Jupyter notebook.

class Lane():
    """
    The Lane class holds two lane lines as objects of the class Line.
    """
    def __init__(self, img_size_x, img_size_y, lane_width=3.7, hist_size=3, failed_max=6):
        # x size of the lane image
        self.img_size_x = img_size_x
        # y size of the lane image
        self.img_size_y = img_size_y
        # Lane width in meters
        self.lane_width = lane_width
        
        # Create two Line objects
        self.left_line = Line(self, 'left', hist_size, failed_max)
        self.right_line = Line(self, 'right', hist_size, failed_max)
        
        # Vehicle position relative to the lane center in meters
        self.vehicle_position = None

V. Discussion

The pipeline I implemented is dedicated to highway scenarios only. The region of interest, where I search for the lane lines is set up in a way that very curvy smaller roads will cause the pipeline to fail. Also the pipeline has it's difficulties with dark lighting situations with very low contrast (like the area under the bridge in the 'challenge video'). To overcome this issue I would try an adaptive approach, where the color and gradient thresholds would be adapted automatically to different lighting situations, if I had the time to pursue this project further.