Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Today the PutHDFS Processor merely places a file into HDFS from NiFi. There are times when we may want to move Files/Directories around on HDFS as part of a workflow. This could be after the PutHDFS processor has placed the file, or from some other trigger.

      Initially we are targeting to take a flow file attribute of an absolute HDFS path, and be able to move it to a target HDFS Path using the Filesystem RENAME API.

        Activity

        Hide
        ggwizdz1 Gray Gwizdz added a comment -

        Ford Motor Company is interested in this functionality, and has developed a patch to fix this need.

        Show
        ggwizdz1 Gray Gwizdz added a comment - Ford Motor Company is interested in this functionality, and has developed a patch to fix this need.
        Hide
        ggwizdz1 Gray Gwizdz added a comment -

        Attached patch demonstrates the ability to move a file from one directory to another on HDFS.

        Show
        ggwizdz1 Gray Gwizdz added a comment - Attached patch demonstrates the ability to move a file from one directory to another on HDFS.
        Hide
        patricker Peter Wicks added a comment -

        Gray Gwizdz, thanks for the contribution. Couple of thoughts, let me know what you think/if you'd be willing to do some additional work:

        • Make Input Folder optional, and if not present use an incoming FlowFile, either it's value or an attribute on it.
        • Allow for Copying along with Moving. This is a common use case I run into where right now I end up Putting to HDFS twice (once for HIVE ingest and once for Archive) where a copy would be a great second step.

        Copy could be a separate ticket, but it might be nice to have a Copy/Move processor all in one.

        Thoughts?

        Show
        patricker Peter Wicks added a comment - Gray Gwizdz , thanks for the contribution. Couple of thoughts, let me know what you think/if you'd be willing to do some additional work: Make Input Folder optional, and if not present use an incoming FlowFile, either it's value or an attribute on it. Allow for Copying along with Moving. This is a common use case I run into where right now I end up Putting to HDFS twice (once for HIVE ingest and once for Archive) where a copy would be a great second step. Copy could be a separate ticket, but it might be nice to have a Copy/Move processor all in one. Thoughts?
        Hide
        ggwizdz1 Gray Gwizdz added a comment -

        Hi Peter,

        Thank you for the recommendations! I agree with your thinking, let me spend some time working through these changes and I'll send another patch once I've made progress.

        Thanks!
        Gray

        Show
        ggwizdz1 Gray Gwizdz added a comment - Hi Peter, Thank you for the recommendations! I agree with your thinking, let me spend some time working through these changes and I'll send another patch once I've made progress. Thanks! Gray
        Hide
        ggwizdz1 Gray Gwizdz added a comment -

        Hi Peter,

        Please see the attached patch that include your recommendations, thanks for the ideas!

        Gray

        Show
        ggwizdz1 Gray Gwizdz added a comment - Hi Peter, Please see the attached patch that include your recommendations, thanks for the ideas! Gray

          People

          • Assignee:
            ggwizdz1 Gray Gwizdz
            Reporter:
            josephxsxn Joseph Niemiec
          • Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development