You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tokenizer.ts doesn't need a conditional import if you're using Vite (npm run dev), but it is necessary when using (npm run build --debug && npm run preview).
So, why does tokenizer.ts require this conditional check to import GPT3TokenizerImport?
However – the following Just Works™️ when you're in npm run dev in Vite because it can handle mixed import syntax from ES6 module imports and CommonJS:
// $lib/tokenizer.ts
import GPT3Tokenizer from 'gpt3-tokenizer';
const tokenizer = new GPT3Tokenizer({ type: 'gpt3' });
export function getTokens(input: string): number {
const tokens = tokenizer.encode(input);
return tokens.text.length;
}
I hope this helps.
The text was updated successfully, but these errors were encountered:
So, why does tokenizer.ts require this conditional check to import
GPT3TokenizerImport
?The code below (from the GitHub issue mentioned in the video), checks if you're using
CommonJS
or not.However – the following Just Works™️ when you're in
npm run dev
in Vite because it can handle mixed import syntax from ES6 module imports and CommonJS:I hope this helps.
The text was updated successfully, but these errors were encountered: