[NIFI-6464] ListHDFS should support fragment attributes with strategies

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.9.2
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Alessandro D'Armiento
• Created: 22/Jul/19 12:46
• Updated: 23/Jul/19 14:52
• Resolved: 23/Jul/19 14:52

Description

Current Situation

ListHDFS doesn't support Fragmentation attributes

Improvement Proposal

• Since the processor works on a 1:N semantic (1 input trigger flowfile, N output flowfiles) it would be nice to support fragmentation attributes (for example for subsequent merge operations)
  • It would be also useful to support different fragmentation strategies, in order to support multiple user cases. For example, it should be possible to select:
    • A "one for all" fragmentation strategy which will create a single fragmentation group. Therefore, all files will have the same fragment.identifier, the same fragment.count, equal to the total number N of listed files, and fragment.index ∈ [0, N).
    • A "per subdir" fragmentation strategy which will create different fragmentation groups, one for each scanned subdirectory of the given path. Therefore, for each subfolder, flowfiles will have a specific fragment.identifier, fragment.count will be, for each flowfile, equal to the number Ni of files in the i-th directory, and fragment.index ∈ [0, Ni).