Create bridge from pytorch/tokenizers Tokenizer to LlgTokenizer*. Provide vocab bytes, tokenize_fn callback. Support BPE + SentencePiece. Ref: kotoba.cpp/common/llguidance.cpp. Depends on: llguidance dependency.
cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng
Create bridge from pytorch/tokenizers Tokenizer to LlgTokenizer*. Provide vocab bytes, tokenize_fn callback. Support BPE + SentencePiece. Ref: kotoba.cpp/common/llguidance.cpp. Depends on: llguidance dependency.
cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng