tokenizer

@jscpd/tokenizer

tokenizer is package from @jscpd used for convert programming code to list of tokens

Installation

npm install @jscpd/tokenizer --save

Usage

import {IOptions,  ITokensMap} from '@jscpd/core';
import {Tokenizer} from '@jscpd/tokenizer';

const tokenizer = new Tokenizer();
const options: IOptions = {};

const maps: ITokensMap[] = tokenizer.generateMaps('source_id', 'let a = "11"', 'javascript', options);

License

MIT © Andrey Kucherenko