# First contribution: core tokenizer unit tests now run without NLTK downloads (PR #8134), feedback welcome **URL:** \ **Category:** Agents\ **Tags:** agent-development\ **Created:** [October 27, 2026, 10:58pm UTC](https://community.livekit.io/t/first-contribution-core-tokenizer-unit-tests-now-run-without-nltk-downloads-pr-8134-feedback-welcome/2399 "2026-10-27T22:58:50Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![ilakkmanoharan2025](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@ilakkmanoharan2025](https://community.livekit.io/u/ilakkmanoharan2025)\ **Post date:** [October 27, 2026, 10:58pm UTC](https://community.livekit.io/t/first-contribution-core-tokenizer-unit-tests-now-run-without-nltk-downloads-pr-8134-feedback-welcome/2399/1 "2026-10-27T22:58:50Z")
Hi everyone, I’m Ilak, a software engineer. A LiveKit demo got me exploring the Agents framework, and I just opened my first PR: [test: separate NLTK coverage from hermetic tokenizer unit tests by ilakkmanoharan · Pull Request #8134 · livekit/agents · GitHub](https://github.com/livekit/agents/pull/8134) What it does: - tests/test\_tokenizer.py is marked as a hermetic unit test, but it imported the NLTK plugin and downloaded the Punkt model at collection time. The core tokenizer tests (basic, BlingFire, word, streaming, hyphenation, etc.) no longer import NLTK or need model data. NLTK batch and streaming coverage moved to a dedicated tests/test\_plugin\_nltk.py (run with --plugin nltk). The model download happens in a module-scoped fixture, only when those tests run. The shared sample text lives in a small dependency-free tests/tokenizer\_data.py. CI runs the NLTK tests as a separate, clearly labeled step, and the README documents the command. No production tokenizer code or public API changed. Validation: 63 core tokenizer tests pass with network and NLTK blocked, the 2 NLTK plugin tests pass, and make check is clean. CI on the PR is green. I’d really appreciate feedback on the approach, and pointers to other areas (tests, reliability, docs) where a new contributor could help. Thanks! ---
**Author:** ![Muhammad\_Usman\_Bashir](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/muhammad_usman_bashir/32/1149_2.png) [@Muhammad\_Usman\_Bashir](https://community.livekit.io/u/Muhammad_Usman_Bashir)\ **Post date:** [October 28, 2026, 1:45pm UTC](https://community.livekit.io/t/first-contribution-core-tokenizer-unit-tests-now-run-without-nltk-downloads-pr-8134-feedback-welcome/2399/2 "2026-10-28T13:45:22Z")
@ilakkmanoharan2025, Welcome Ilak, and a good first PR to pick. A test marked hermetic should not import a plugin or download the Punkt model at collection time, so pulling those out is the right call. The split also matches the harness already in the repo: `tests/conftest.py` builds a `--` option for each category and filters tests by their marker, and it uses `--plugin` as its own worked example ([conftest.py](https://github.com/livekit/agents/blob/main/tests/conftest.py)). So moving the NLTK cases behind the `plugin` marker and running them with `--plugin nltk` is the idiomatic pattern here, not a one-off, and the dependency-free `tokenizer_data.py` follows the same instinct. One gate before a code owner can act: the PR still needs the CLA signed (the CLA Assistant bot left the link on it) and one code-owner approval to merge, so clear the CLA first to unblock review. For the where-next ask, keep mining the test-reliability vein you found, since other suites can hide the same import-time side effects. [CONTRIBUTING](https://github.com/livekit/agents/blob/main/CONTRIBUTING.md) asks you to open an issue first to agree scope, run `make check` locally before pushing, and leave `CHANGELOG.md` and the package manifests to the bot, which saves a review round-trip. ---
**Author:** ![ilakkmanoharan2025](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@ilakkmanoharan2025](https://community.livekit.io/u/ilakkmanoharan2025)\ **Post date:** [October 29, 2026, 12:49am UTC](https://community.livekit.io/t/first-contribution-core-tokenizer-unit-tests-now-run-without-nltk-downloads-pr-8134-feedback-welcome/2399/3 "2026-10-29T00:49:10Z")
Thanks, Muhammad! I appreciate the welcome and the clarification on the marker pattern. I’ll sort out the CLA so the PR can move forward. I’ll also look for similar import-time side effects in other test suites and open an issue to agree on scope before starting another contribution. Thanks for the guidance!