[CAMEL-20986] camel-ai: improve chunking support

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.x
• Fix Version/s: 4.8.0
• Component/s: camel-ai, camel-core, camel-langchain4j
• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Otavio Rodolfo Piske
• Created: 16/Jul/24 07:45
• Updated: 03/Sep/24 13:31
• Resolved: 03/Sep/24 13:31

Description

Currently, when loading data into a VectorDB, applications have to implement chunking on their own. This is typically done as a processor (which then uses LangChain4J Document Splitter features or implement their own).

We could make this simpler by implementing a specialized chunking DSL that abstracts this process and simplify loading the data to those databases.

Issue Links

• links to: GitHub Pull Request #14960
• links to: GitHub Pull Request #15061