[CAMEL-18776] camel-hdfs - Fix HdfsNormalFileHandler to handle temporary file path correctly

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.14.7, 3.18.5, 3.20.0
• Component/s: camel-hdfs
• Estimated Complexity: Unknown
• Assignee: Kengo Seki
• Reporter: Kengo Seki
• Created: 30/Nov/22 11:57
• Updated: 29/Mar/23 23:15
• Resolved: 30/Nov/22 17:08

Description

Currently, HDFS consumer doesn't seem to work with normal files.

Given the following file on HDFS,

$ hdfs dfs -ls /tmp/input
Found 1 items
-rw-r--r--   3 sekikn supergroup          4 2022-11-30 20:40 /tmp/input/foo.txt
$ hdfs dfs -cat /tmp/input/foo.txt
bar

I tried to run the following route so as to copy normal files from HDFS to local filesystem.

$ cat put.java 
import org.apache.camel.builder.RouteBuilder;

public class put extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("hdfs:192.168.2.2:9000/tmp/input/?repeatCount=1").log("${body}");
    }
}

But I came across the following error.

$ camel run put.java 

...

2022-11-30 20:46:25.151  WARN 2718863 --- [9000/tmp/input/] apache.camel.component.hdfs.HdfsConsumer : Failed polling endpoint: hdfs://192.168.2.2:9000/tmp/input/?repeatCount=1. Will try again at next poll. Caused by: [org.apache.camel.RuntimeCamelException - java.io.FileNotFoundException: /tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)]
org.apache.camel.RuntimeCamelException: java.io.FileNotFoundException: /tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)
	at org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:98) ~[camel-hdfs-3.19.0.jar:3.19.0]
	at org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:38) ~[camel-hdfs-3.19.0.jar:3.19.0]
	at org.apache.camel.component.hdfs.HdfsFileType.createInputStream(HdfsFileType.java:46) ~[camel-hdfs-3.19.0.jar:3.19.0]
	at org.apache.camel.component.hdfs.HdfsInputStream.createInputStream(HdfsInputStream.java:71) ~[camel-hdfs-3.19.0.jar:3.19.0]
	at org.apache.camel.component.hdfs.HdfsConsumer.asHdfsFile(HdfsConsumer.java:246) ~[camel-hdfs-3.19.0.jar:3.19.0]

...

Caused by: java.io.FileNotFoundException: /tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)
	at java.io.FileInputStream.open0(Native Method) ~[?:?]
	at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
	at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
	at org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:94) ~[camel-hdfs-3.19.0.jar:3.19.0]
	... 27 more

Temporary file was correctly created as follows, but HdfsNormalFileHandler seems to have mistaken its path. The correct one is "/tmp/foo.txt.opened5427239805734101292.hdfs", but there's unnecessary trailing string "/foo.txt.opened" in the log above.

$ ls /tmp/foo.txt*
/tmp/foo.txt.opened5427239805734101292.hdfs
$ cat /tmp/foo.txt.opened5427239805734101292.hdfs 
bar

Issue Links

• links to: GitHub Pull Request #8803