js-tokens is a tiny JavaScript tokenizer that uses regular expressions to split a JavaScript source string into an iterable stream of token objects. It is useful for tooling or analysis that needs lexer-level tokenization, including handling invalid JavaScript without throwing (it yields an “Invalid” token when it cannot classify a character).
Project status
- Actively maintained: Evidence suggests ongoing repository activity, with the last upstream push on 2026-06-05 (relative to today, 2026-09-10). However, published updates are less frequent than raw activity.
- Update cadence: The most recent tagged update is v10.0.0 (2025-12-08), with prior notable updates in 2024-11-22 (v9.0.1) and 2024-02 (v9.0.0, v8.0.3). Overall, this looks like roughly annual or semi-annual update spacing, with additional work occurring between tags.
AI summary generated
Recent updates
v10.0.0
Release v10.0.0 primarily migrates the package from CommonJS to ESM. The provided publisher release notes are empty, so most user-visible changes are only visible by inspecting the code diff and repository documentation files.
v9.0.1
v9.0.1 makes a targeted fix to the regular expression literal tokenizer (notably around character class edge cases like `/a[/]/`). The code diff also includes a larger repository/tooling shift to ESM (`package.json` `type` change), and the test stack is migrated from Jest to Vitest with updated CI Node versions.
Breakingv9.0.0
Release v9.0.0 was published, but no release notes were provided by the publisher. As a result, there is insufficient information to determine what changed, whether there are breaking changes, or what developers should migrate.
v8.0.3
v8.0.3 primarily improves js-tokens handling of extremely long inputs. It updates several of the token-matching regular expressions to avoid regex-engine stack overflows, and adds tests plus documentation for the “known failures” edge cases.
v8.0.2
This release changes the shipped TypeScript type definitions for js-tokens and bumps a development dependency. The code diff indicates the main functional difference is how the jsTokens function is exported in TypeScript (export default vs export=).
Breakingv8.0.1
No release notes were provided in the release entry for v8.0.1. The code diff shows a targeted change to js-tokens regular expression literal parsing, along with test and tooling updates (dev dependency bumps and snapshot churn).
Breakingv8.0.0
Release v8.0.0 adds ES2022-related tokenizer support, most notably a new token type for private identifiers (hash-prefixed names). It also adds support for the RegExp literal `d` flag, and updates TypeScript typings and test expectations accordingly. No release notes were provided by the publisher, so the changes below are inferred directly from the code diff.
Featuresv7.0.0
v7.0.0 updates the tokenizer’s internal regexes and expands the test suite. The main behavioral change is new ES2021 token support for logical assignment operators (&&=, ||=, ??=) and numeric separators (underscores in numeric literals), alongside tooling and CI updates.
v6.0.0
v6.0.0 is a major refactor of js-tokens, replacing the previous “export a global RegExp plus helper” approach with a generator-based tokenizer function that yields spec-named token objects. The implementation is now a larger CoffeeScript state machine that supports additional syntax like templates, regex literals, and optional JSX, and it adds a TypeScript type surface.
v5.0.0
Version v5.0.0 was published, but the publisher did not provide any release notes content. As a result, this release summary cannot confirm new features, breaking changes, bug fixes, security or performance updates from the release notes alone.