[NIFI-2859] List + Fetch HDFS processors are reading part files from HDFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Component/s: Extensions
• Assignee: Pierre Villard
• Reporter: Mahesh Nayak
• Created: 04/Oct/16 09:28
• Updated: 20/Jul/18 17:47
• Resolved: 15/Feb/17 04:23

Description

Create the following ProcessGroups
GetFile --> PutHdfs --> PutFile
ListHDFS --> FetchHdfs --> putFile
2. Now start both the processGroups
3. Write lots of files into HDFS so that ListHDFS keeps listing and FetchHdfs fetches.
4. An exception is thrown because the processor reads the part file from the putHdfs folder

java.io.FileNotFoundException: File does not exist: /tmp/HDFSProcessorsTest_visjJMcHORUwigw/.ycnVSpBOzEaoTWk_7f37d5af-d4a4-4521-b60d-c3c11ae19669
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)

Note that eventually the file is copied to the output successfully, but at the same time there are some files in the failure/comms failure relationship

Issue Links

• is duplicated by: NIFI-3379 ListHDFS has no File Filter Regex
• is related to: NIFI-4434 ListHDFS applies File Filter also to subdirectory names in recursive search
• links to: GitHub Pull Request #1383