milliemaoo / New-word-discovery Public

Notifications You must be signed in to change notification settings
Fork 0
Star 5

Chinese new word discovery with computing PMI, Left and right entropy and Word-position Probability. 新词发现

5 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
README.md		README.md
config.py		config.py
position.py		position.py
probability.py		probability.py
sequence.py		sequence.py
wordseg.py		wordseg.py

Repository files navigation

New-word-discovery

Chinese new word discovery

This is a simple project to find Chinese new word. Inspired by Matrix67 and ChineseWordSegmentation, wirtten with python.

Usage

Place the corpus in the root directory of the project.
All hyperparameters are configured in the "./config.py"

Involved algorithm

PMI
Left and right entropy
Position-Word Probability (word-initial,suffix)

About

Chinese new word discovery with computing PMI, Left and right entropy and Word-position Probability. 新词发现

Report repository

Releases

No releases published

Packages

No packages published

Languages

Python 100.0%