Late Chunking

Late chunking is another advanced chunking technique. Instead of chunking first and embedding each chunk in isolation, it runs the embedding model over the full document (or as much as the model can handle), then creates chunk embeddings based on token embeddings that already saw the broader context.

Each chunk can carry more meaningful context because the model has seen how its words fit into the larger document. "Bear" in a Ted review means a different kind of bear than "bear" in a nature documentary.

When Should I Use the Fancy Stuff?

• Regular chunking handles most real-world applications effectively.
• Advanced methods (like ColBERT or late chunking) are worth considering when:
  • You need extremely precise search results.
  • You have significant compute and storage resources at your disposal (7 trillion dollars? please?).
  • Standard approaches aren't meeting your accuracy requirements.
  • You're working with complex, nuanced text where context is critical.