New-word-discovery

Chinese new word discovery

This is a simple project to find Chinese new word. Inspired by Matrix67 and ChineseWordSegmentation, wirtten with python.

Usage

Place the corpus in the root directory of the project.
All hyperparameters are configured in the "./config.py"

Involved algorithm

PMI
Left and right entropy
Position-Word Probability (word-initial,suffix)