Search results

js-tokens

Tiny JavaScript tokenizer.

• JavaScript
• js
• ECMAScript
• es
• token
• tokens
• tokenize
• tokenizer
• regex
• regexp

lydell

published 9.0.0 • 5 months agopublished 9.0.0 5 months ago

strtok3

A promise based streaming tokenizer

• tokenizer
• reader
• token
• async
• promise
• parser
• decoder
• binary
• endian
• uint
• stream
• streaming

borewit

published 8.0.0 • a day agopublished 8.0.0 a day ago

css-tree

A tool set for CSS: fast detailed parser (CSS → AST), walker (AST traversal), generator (AST → CSS) and lexer (validation and matching) based on specs and browser implementations

• css
• ast
• tokenizer
• parser
• walker
• lexer
• generator
• utils
• syntax
• validation

lahmatiy

published 2.3.1 • 2 years agopublished 2.3.1 2 years ago

@csstools/css-tokenizer

Tokenize CSS

• css
• tokenizer

romainmenke

published 2.4.1 • 13 days agopublished 2.4.1 13 days ago

chevrotain

Chevrotain is a high performance fault tolerant javascript parsing DSL for building recursive decent parsers

• parser
• syntax
• lexical
• analysis
• grammar
• lexer
• tokenizer
• generator
• compiler
• fault
• tolerant

bd82

published 11.0.3 • a year agopublished 11.0.3 a year ago

natural

General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech tagging, sentiment analysis, classification, inflection, phonetics, tfidf, WordNet, jaro-winkler, Levenshtein distance, Dice's Coefficient) facilities for node.

• natural language processing
• artifical intelligence
• statistics
• Porter stemmer
• Lancaster stemmer
• tokenizer
• bigram
• trigram
• quadgram
• ngram
• stemmer
• bayes
• classifier
• phonetic
• View more

hugo-ter-doest

published 7.1.0 • 11 days agopublished 7.1.0 11 days ago

micromark

small commonmark compliant markdown parser with positional info and concrete tokens

• commonmark
• compiler
• gfm
• html
• lexer
• markdown
• markup
• md
• unified
• parse
• parser
• plugin
• process
• remark
• View more

wooorm

published 4.0.0 • a year agopublished 4.0.0 a year ago

react-bootstrap-typeahead

React typeahead with Bootstrap styling

• auto complete
• auto suggest
• auto-complete
• auto-suggest
• autocomplete
• autosuggest
• bootstrap
• bootstrap tokenizer
• bootstrap typeahead
• bootstrap-tokenizer
• bootstrap-typeahead
• react
• react autocomplete
• react autosuggest
• View more

ericgio

published 6.3.2 • 9 months agopublished 6.3.2 9 months ago

gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

• BPE
• encoder
• decoder
• tokenizer
• GPT
• GPT-2
• GPT-3
• GPT-3.5
• GPT-4
• GPT-4o
• NLP
• Natural Language Processing
• Text Generation
• OpenAI
• View more

niieani

published 2.2.0 • 2 hours agopublished 2.2.0 2 hours ago

ret

Tokenizes a string that represents a regular expression.

• regex
• regexp
• regular expression
• parser
• tokenizer

fent

published 0.5.0 • a year agopublished 0.5.0 a year ago

stream-json

stream-json is the micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API. I

• scanner
• lexer
• tokenizer
• parser
• django
• stream
• streaming
• json

elazutkin

published 1.8.0 • a year agopublished 1.8.0 a year ago

@lenml/tokenizers

a lightweight no-dependency fork of transformers.js (only tokenizers)

• llama
• llama2
• llama3
• chatgpt
• mistral
• tokenizer
• transformers
• transformers.js
• gpt4
• gpt4o
• gpt3.5
• vicuna
• chatglm
• baichuan

luke_zhang

published 1.0.9 • 9 days agopublished 1.0.9 9 days ago

stream-csv-as-json

Streams CSV files.

• scanner
• lexer
• tokenizer
• parser
• csv
• stream
• streaming

elazutkin

published 1.0.5 • a month agopublished 1.0.5 a month ago

php-parser

Parse PHP code from JS and returns its AST

• php
• php5
• php7
• php8
• parser
• lexer
• tokenizer
• ast

czosel

published 3.1.5 • a year agopublished 3.1.5 a year ago

rich-text-tokenizer

Rich text and markdown tokenization made easy.

• markdown
• rich
• text
• tokenizer

rtkhanas

published 1.0.13 • a year agopublished 1.0.13 a year ago

css-tree-animatable

Fork of css-tree with all but animatable CSS properties removed

• css
• ast
• tokenizer
• parser
• walker
• lexer
• generator
• utils
• syntax
• validation
• animatable
• properties

webanimate

published 1.0.20 • 4 months agopublished 1.0.20 4 months ago

codsen-tokenizer

HTML and CSS lexer aimed at code with fatal errors, accepts mixed coding languages

• ast
• codsen
• css
• html
• lexer
• lint
• parse
• parsing
• processing
• token
• tokeniser
• tokenizer

royston

published 7.0.25 • 2 months agopublished 7.0.25 2 months ago

pratt-parser

TDOP parser

• PRATT
• TDOP
• parser
• tokenizer

arlac77

published 10.0.2 • 3 months agopublished 10.0.2 3 months ago

is-char-suitable-for-html-attr-name

Is given character suitable to be in an HTML attribute's name?

• attribute
• character
• codsen
• emlint
• for
• html
• is
• lexer
• name
• parsing
• suitable
• tag
• tokeniser
• tokenizer

royston

published 4.0.7 • 6 months agopublished 4.0.7 6 months ago

is-html-tag-opening

Does an HTML tag start at given position?

• bracket
• brackets
• cleaning
• code
• codsen
• html
• is
• non-parsing
• opening
• parsing
• tag
• tokenizer
• utility

royston

published 4.0.19 • 4 months agopublished 4.0.19 4 months ago