Hybrid chunking using OpenAI Tokenizer

[mohammedfaisal]

The documentation explains how to configure and use OpenAI Tokenizer with hybrid chunker in python.

       import tiktoken

        from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer

        tokenizer = OpenAITokenizer(
                   tokenizer=tiktoken.encoding_for_model("gpt-4o"),
                   max_tokens=128 * 1024,  # context window length required for OpenAI tokenizers
        )
        
        chunker = HybridChunker(
                tokenizer=tokenizer,
                merge_peers=True,  # optional, defaults to True
        )
        chunk_iter = chunker.chunk(dl_doc=doc)
        chunks = list(chunk_iter)

How to do the same operation using docling-java ?
The HybridChunkerOptions.builder().tokenizer() seems support only HuggingFace models.

[edeandrea]

I'll have to look at the docling-serve documentation. If it's exposed in the api then we probably just missed it. If it isn't part of the docling-serve API then they will need to add it there first.

[edeandrea]

Just checking quickly, but I don't see anything in https://github.com/docling-project/docling-serve/blob/main/docs/usage.md that describes a way to use an OpenAI transformer.

@dolfim-ibm is that documentation correct?

[edeandrea]

@dolfim-ibm gentle ping? Do you know if its possible via docling-serve to use the OpenAITokenizer with the hybrid chunker? Is that exposed via the rest api?

[edeandrea]

@dosu do you know anything about this in the docling-serve project?

[dosubot]

Based on my research, docling-serve's REST API does not currently expose tokenizer configuration for the HybridChunker. The API focuses on document conversion options (output formats, OCR, table structure, image descriptions, etc.) but there's no chunker, tokenizer, or chunking_options field available.

The OpenAITokenizer functionality exists only at the Python library level in docling_core.transforms.chunker.tokenizer.openai, and this hasn't been wired up to the REST API.

Workaround: Use docling-serve for document conversion, then apply chunking locally:

from docling_core.types.doc.document import DoclingDocument
from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer
from docling.chunking import HybridChunker
import tiktoken

# Load the JSON output from docling-serve
doc = DoclingDocument.load_from_json("output.json")

# Apply HybridChunker with OpenAI tokenizer locally
tokenizer = OpenAITokenizer(
    tokenizer=tiktoken.encoding_for_model("gpt-4o"),
    max_tokens=128 * 1024,
)
chunker = HybridChunker(tokenizer=tokenizer)
chunks = list(chunker.chunk(dl_doc=doc))

So @edeandrea's assessment appears correct — this would need to be added to docling-serve first before it could be exposed in docling-java.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[mohammedfaisal]

I was able to work around this by using the HuggingFace-hosted equivalent of the GPT-4 tokenizer (Xenova/gpt-4) directly in HybridChunkerOptions:

HybridChunkDocumentRequest request = HybridChunkDocumentRequest.builder()
        .source(FileSource.builder()
                .filename("abc.pdf")
                .base64String(Base64.getEncoder().encodeToString(bytes))
                .build())
        .options(ConvertDocumentOptions.builder()
                .toFormat(OutputFormat.MARKDOWN)
                .includeImages(false)
                .build())
        .chunkingOptions(HybridChunkerOptions.builder()
                .tokenizer("Xenova/gpt-4")
                .maxTokens(600)
                .useMarkdownTables(true)
                .mergePeers(true)
                .build())
        .target(InBodyTarget.builder().build())
        .build();

[edeandrea]

Thanks for the update @mohammedfaisal ! Do you mind if I close out this issue?