Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tokenizer.ts doesn't need a conditional import if you're using Vite's default bundler #13

Open
inspiredlabs opened this issue May 14, 2023 · 2 comments

Comments

@inspiredlabs
Copy link

inspiredlabs commented May 14, 2023

tokenizer.ts doesn't need a conditional import if you're using Vite (npm run dev), but it is necessary when using (npm run build --debug && npm run preview).

So, why does tokenizer.ts require this conditional check to import GPT3TokenizerImport?

The code below (from the GitHub issue mentioned in the video), checks if you're using CommonJS or not.

const GPT3Tokenizer: typeof GPT3TokenizerImport =
 typeof GPT3TokenizerImport === 'function'
 ? GPT3TokenizerImport
 : (GPT3TokenizerImport as any).default

However – the following Just Works™️ when you're in npm run dev in Vite because it can handle mixed import syntax from ES6 module imports and CommonJS:

// $lib/tokenizer.ts
import GPT3Tokenizer from 'gpt3-tokenizer';

const tokenizer = new GPT3Tokenizer({ type: 'gpt3' });

export function getTokens(input: string): number {
  const tokens = tokenizer.encode(input);
  return tokens.text.length;
}

I hope this helps.

@inspiredlabs
Copy link
Author

Rich Harris on the state of commonJS, see this: https://youtu.be/AOXq89h8saI?t=298

@inspiredlabs
Copy link
Author

And this: https://youtu.be/xCeYmdukOKI?t=541

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant