Skip to content

Tokenizing strings of text. Regex extracting arrays of words and optionally numbers, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

License

Notifications You must be signed in to change notification settings

eklem/words-n-numbers

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Words'n'numbers

Tokenizing strings of text. Extracting arrays of words and optionally number, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions. Part of document processing for search-index and nowsearch.xyz.

Inspired by extractwords

NPM version NPM downloads Build Status JavaScript Style Guide MIT License

Breaking change

From v8.0.0 - emojis-regular expression now extracts single emojis, so no more "words" formed by several emojis. This because each emoji in a sense are words. You can still make a custom regular expression to grab several emojis in a row as one item with const customEmojis = '\\p{Emoji_Presentation}' and then use it as your custom regex.

Meaning that instead of:

extract('A ticket to 倧ι˜ͺ costs Β₯2000 πŸ‘ŒπŸ˜„ 😒', { regex: emojis})
// ['πŸ‘ŒπŸ˜„', '😒']

...you will get:

extract('A ticket to 倧ι˜ͺ costs Β₯2000 πŸ‘ŒπŸ˜„ 😒', { regex: emojis})
// ['πŸ‘Œ', 'πŸ˜„', '😒']

Initiating

CJS

const { extract, words, numbers, emojis, tags, usernames, email } = require('words-n-numbers')
// extract, words, numbers, emojis, tags, usernames, email available

ESM

import { extract, words, numbers, emojis, tags, usernames, email } from 'words-n-numbers'
// extract, words, numbers, emojis, tags, usernames, email available

Browser

<script src="https://cdn.jsdelivr.net/npm/words-n-numbers/dist/words-n-numbers.umd.min.js"></script>

<script>
  //wnn.extract, wnn.words, wnn.numbers, wnn.emojis, wnn.tags, wnn.usernames, wnn.email available
</script>

Browser demo

A simple browser demo of wnn to show how it works.

Screenshot of the words-n-numbers demo

Use

The default regex should catch every unicode character from for every language. Default regex flags are giu. emojisCustom-regex won't work with the u-flag (unicode).

Only words

const stringOfWords = 'A 1000000 dollars baby!'
extract(stringOfWords)
// returns ['A', 'dollars', 'baby']

Only words, converted to lowercase

const stringOfWords = 'A 1000000 dollars baby!'
extract(stringOfWords, { toLowercase: true })
// returns ['a', 'dollars', 'baby']

Combining predefined regex for words and numbers, converted to lowercase

const stringOfWords = 'A 1000000 dollars baby!'
extract(stringOfWords, { regex: [words, numbers], toLowercase: true })
// returns ['a', '1000000', 'dollars', 'baby']

Combining predefined regex for words and emoticons, converted to lowercase

const stringOfWords = 'A ticket to 倧ι˜ͺ costs Β₯2000 πŸ‘ŒπŸ˜„ 😒'
extract(stringOfWords, { regex: [words, emojis], toLowercase: true })
// returns [ 'A', 'ticket', 'to', '倧ι˜ͺ', 'costs', 'πŸ‘Œ', 'πŸ˜„', '😒' ]

Combining predefined regex for numbers and emoticons

const stringOfWords = 'A ticket to 倧ι˜ͺ costs Β₯2000 πŸ‘ŒπŸ˜„ 😒'
extract(stringOfWords, { regex: [numbers, emojis], toLowercase: true })
// returns [ '2000', 'πŸ‘Œ', 'πŸ˜„', '😒' ]

Combining predefined regex for words, numbers and emoticons, converted to lowercase

cons stringOfWords = 'A ticket to 倧ι˜ͺ costs Β₯2000 πŸ‘ŒπŸ˜„ 😒'
extract(stringOfWords, { regex: [words, numbers, emojis], toLowercase: true })
// returns [ 'a', 'ticket', 'to', '倧ι˜ͺ', 'costs', '2000', 'πŸ‘Œ', 'πŸ˜„', '😒' ]

Predefined regex for #tags

const stringOfWords = 'A #49ticket to #倧ι˜ͺ or two#tickets costs Β₯2000 πŸ‘ŒπŸ˜„πŸ˜„ 😒'
extract(stringOfWords, { regex: tags, toLowercase: true })
// returns [ '#49ticket', '#倧ι˜ͺ' ]

Predefined regex for @usernames

const stringOfWords = 'A #ticket to #倧ι˜ͺ costs [email protected], @alice and @ηΎŽζž— Β₯2000 πŸ‘ŒπŸ˜„πŸ˜„ 😒'
extract(stringOfWords, { regex: usernames, toLowercase: true })
// returns [ '@alice123', '@ηΎŽζž—' ]

Predefined regex for email addresses

const stringOfWords = 'A #ticket to #倧ι˜ͺ costs [email protected], [email protected], [email protected] and @ηΎŽζž— Β₯2000 πŸ‘ŒπŸ˜„πŸ˜„ 😒'
extract(stringOfWords, { regex: email, toLowercase: true })
// returns [ '[email protected]', '[email protected]', '[email protected]' ]

Predefined custom regex for all Unicode emojis

const stringOfWords = 'A #ticket to #倧ι˜ͺ costs [email protected], [email protected], [email protected] and @ηΎŽζž— Β₯2000 πŸ‘ŒπŸ˜„πŸ˜„ πŸ˜’πŸ‘©πŸ½β€πŸ€β€πŸ‘¨πŸ» πŸ‘©πŸ½β€πŸ€β€πŸ‘¨πŸ»'
extract(stringOfWords, { regex: emojisCustom, flags: 'g' })
// returns [ 'πŸ‘Œ', 'πŸ˜„', 'πŸ˜„', '😒', 'πŸ‘©πŸ½β€πŸ€β€πŸ‘¨πŸ»', 'πŸ‘©πŸ½β€πŸ€β€πŸ‘¨πŸ»' ]

Custom regex

Some characters needs to be escaped, like \and '. And you escape it with a backslash - \.

const stringOfWords = 'This happens at 5 o\'clock !!!'
extract(stringOfWords, { regex: '[a-z\'0-9]+' })
// returns ['This', 'happens', 'at', '5', 'o\'clock']

API

Extract function

Returns an array of words and optionally numbers.

extract(stringOfText, \<options-object\>)

Options object

{
  regex: 'custom or predefined regex',  // defaults to words
  toLowercase: [true / false]             // defaults to false
  flags: 'gmixsuUAJD' // regex flags, defaults to giu - /[regexPattern]/[regexFlags]
}

Order of combined regexes

You can add an array of different regexes or just a string. If you add an array, they will be joined with a |-separator, making it an OR-regex. Put the email, usernames and tags before words to get the extraction right.

// email addresses before usernames before words can give another outcome than
extract(oldString, { regex: [email, usernames, words] })

// than words before usernames before email addresses
extract(oldString, { regex: [words, usernames, email] })

Predefined regexes

words              // only words, any language <-- default
numbers            // only numbers
emojis             // only emojis
emojisCustom       // only emojis. Works with the `g`-flag, not `giu`. Based on custom emoji extractor from https://github.com/mathiasbynens/rgi-emoji-regex-pattern
tags               // #tags (any language
usernames          // @usernames (any language)
email              // email addresses. Most valid addresses,
                   //   but not to be used as a validator

Flags for regexes

All but one regex uses the giu-flag. The one that doesn't is the emojisCustom that will need only a g-flag. emojisCustom is added because the standard emojis regex based on \\p{Emoji_Presentation} isn't able to grab all emojis. When browsers support p\{RGI_emoji} under a giu`-flag the library will be changed.

Languages supported

Supports most languages supported by stopword, and others too. Some languages like Japanese and Chinese simplified needs to be tokenized. May add tokenizers at a later stage.

PR's welcome

PR's and issues are more than welcome =)

About

Tokenizing strings of text. Regex extracting arrays of words and optionally numbers, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published