NLP for Manglish (Code mixed Malayalam+English)

This repository contains state of the art Language models and Classifier for Code mixed Manglish (Malayalam and English) - spoken in Indian sub-continent.

Dataset

1. Malayalam Wikipedia Articles : Preprocessed, Transliterated and Translated versions of this dataset, used for language modeling in this repo, can be downloaded directly from here

2. Dravidian Codemix HASOC @ FIRE 2020

3. Dravidian Codemix Sentiment Analysis @ FIRE 2020

Results

Language Model Perplexity (on validation set)

Architecture/Dataset	Malayalam Wikipedia Articles	Malayalam Wikipedia Articles
	Latin Script	Mixed Script
ULMFiT	45.84	41.22

Classification Metrics

ULMFiT

Dataset	F1	Precision	Recall	Notebook to Reproduce results
Dravidian Codemix HASOC @ FIRE 2020 (Latin Script)	0.74	0.76	0.72	Link
Dravidian Codemix HASOC @ FIRE 2020 (Mixed Script)	0.91	0.92	0.91	Link
Dravidian Codemix Sentiment Analysis @ FIRE 2020	0.69	0.69	0.7	Link

Visualizations

Word Embeddings

Architecture/Dataset	Malayalam Wikipedia Articles	Malayalam Wikipedia Articles
	Latin Script	Mixed Script
ULMFiT	Embeddings projection	Embeddings projection

Pretrained Models

Language Models

Download Latin script pretrained ULMFiT LM from here

Download Mixed Script pretrained ULMFiT LM from here

Tokenizer

Trained tokenizer using Google's sentencepiece

Download the trained model and vocabulary for both Latin and Mixed scripts from here