In this project, we try to implement and compare the performance of various deep learning architectures such as CNNs, RNNs and Dense Feed-Forward systems in classifying the accent of the speaker given the speech signal. This is limited to English language only for now, and later on can be extended to multiple languages.