Skip to content

[LLM Runner] Bridge ExecuTorch tokenizer to llguidance LlgTokenizer #19474

@kirklandsign

Description

@kirklandsign

Create bridge from pytorch/tokenizers Tokenizer to LlgTokenizer*. Provide vocab bytes, tokenize_fn callback. Support BPE + SentencePiece. Ref: kotoba.cpp/common/llguidance.cpp. Depends on: llguidance dependency.

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

Metadata

Metadata

Assignees

Labels

module: llmIssues related to LLM examples and apps, and to the extensions/llm/ codetriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

Status

To triage

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions