Update tokenizer to include tekken implementation #13601

mergennachin · 2025-08-22T15:28:06Z

Include this

meta-pytorch/tokenizers@91140f7

pytorch-bot · 2025-08-22T15:28:09Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13601

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures

As of commit 43635c5 with merge base 8aca197 ():

NEW FAILURES - The following jobs have failed:

pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh)
RuntimeError: Command docker exec -t 0ee6b552440a048912d9b0164615e2141e339105fee6ea4dea7b22f5ae556059 /exec failed with exit code 1
pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
RuntimeError: Command docker exec -t 8942017313bf1345e205e83f70ec750c5f4682659ba5c334215aac67a00dd336 /exec failed with exit code 1
trunk / test-arm-backend (test_models_ethos-u55) / linux-job (gh)
RuntimeError: Command docker exec -t 6e44f802a94fcc57b08b6bdcd0bb4d2359793f04c1945fc1b2429427a28172d5 /exec failed with exit code 1
trunk / test-arm-backend (test_models_ethos-u85) / linux-job (gh)
RuntimeError: Command docker exec -t 3ca53442dc2eb55f6739468f51b8845d96bde5838889375ed64d31799bf19173 /exec failed with exit code 1
trunk / test-arm-backend (test_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 5df7dfc383c7bbbcfd2a640bc54f2d698a1a6b81a342b22b39c41070ea310252 /exec failed with exit code 1
trunk / test-arm-backend (test_pytest_models_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t f8a1e8e76036feeafd2ca54bbd54a8a17ad8041dc6b0d09485cb4cd3ab4082cc /exec failed with exit code 1
trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t bc9e31d103bcd2613c60f69226a1b6db3bfcc414e17f6c10ff5fcfccfdb7179a /exec failed with exit code 1
trunk / test-arm-backend (test_run_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t 691c249eae6b4344adb5940166cec25d9414a3d0f24069aafac179e9c564b6fe /exec failed with exit code 1
trunk / test-arm-backend (test_smaller_stories_llama) / linux-job (gh)
RuntimeError: Command docker exec -t a8d3b414d0b6f69bad058d0f1075ddf629227b5a5dc478fe04c3c240128cce79 /exec failed with exit code 1
trunk / test-arm-cortex-m-size-test (bare_metal) / linux-job (gh)
RuntimeError: Command docker exec -t ff044937e219c6027e9884c487eabd47cf3547f31f194d947d6b38c7cfc89e76 /exec failed with exit code 1
trunk / test-arm-cortex-m-size-test (zephyr-preset) / linux-job (gh)
RuntimeError: Command docker exec -t 61aee095a9ecf60554f87f56e2a208781e09caa675c3bfb58c63b5c42808470b /exec failed with exit code 1
trunk / test-arm-ootb-linux / linux-job (gh)
RuntimeError: Command docker exec -t 92350149712ede6103cb8c62e6ee234f0c7adbddc4a1c297bb50abba416c1138 /exec failed with exit code 1
trunk / test-huggingface-transformers-coreml (qwen3-1.7b|xnnpack|--quantize) / macos-job (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 138

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Update tokenizer to include tekken implementation

43635c5

mergennachin requested review from jackzhxng, larryliu0820 and swolchok as code owners August 22, 2025 15:28

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 22, 2025

mergennachin added the ciflow/trunk label Aug 22, 2025

mergennachin requested a review from kimishpatel August 22, 2025 15:29

mergennachin added the release notes: llm To capture llm specific changes in release notes label Aug 22, 2025

mergennachin requested a review from psiddh August 22, 2025 15:53

jackzhxng approved these changes Aug 22, 2025

View reviewed changes

mergennachin merged commit 45e7810 into main Aug 22, 2025
226 of 240 checks passed

mergennachin deleted the update_tokenizer branch August 22, 2025 17:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update tokenizer to include tekken implementation #13601

Update tokenizer to include tekken implementation #13601

Uh oh!

mergennachin commented Aug 22, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Update tokenizer to include tekken implementation #13601

Update tokenizer to include tekken implementation #13601

Uh oh!

Conversation

mergennachin commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13601

❌ 13 New Failures

Uh oh!

Uh oh!

Uh oh!

mergennachin commented Aug 22, 2025 •

edited

Loading

pytorch-bot bot commented Aug 22, 2025 •

edited

Loading