[CAMEL-8150] camel-hdfs sending message per chunk, not per file

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.14.0
• Fix Version/s: 2.14.2, 2.15.0
• Component/s: camel-hdfs, documentation
• Estimated Complexity: Unknown
• Assignee: Claus Ibsen
• Reporter: Josef Ludvíček
• Created: 12/Dec/14 15:08
• Updated: 17/Dec/14 16:45
• Resolved: 17/Dec/14 07:10

Description

When hdfs consumer picks orinary file, it sends message per data chunk.
Not message per file as one could expect.
In the docs it is mentioned only in option description, which can be overlooked very easily.
It should be explicitly written in the beginning of the component doc page .

Option description from docs

chunkSize

4096

When reading a normal file, this is split into chunks producing a message per chunk.

Sample camel route (workaround could be to add ?fileExist=Append to file component)

 <camel:route id="toFile">
            <from uri="hdfs2:localhost:8020/tmp/camel-test/"/>
            <log message="picked up file from hdfs with name $simple{header.CamelFileName}"/>
            <to uri="file:test-dest"/>
            <log message="file downloaded from hadoop"/>
</camel:route>

Issue Links

• is related to: CAMEL-8040 camel-hdfs2 consumer overwriting data instead of appending them