Hadoop Common
  1. Hadoop Common
  2. HADOOP-8940

Add a resume feature to the copyFromLocal and put commands

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0.1-alpha
    • Fix Version/s: 2.0.1-alpha
    • Component/s: tools
    • Labels:
      None

      Description

      Add a resume feature to the copyFromLocal command. Failures in large transfers result in a great deal of wasted time. For large files, it would be good to be able to continue from the last good block onwards. The file would have to be unavailable to other clients for reads or regular writes until the "resume" process was completed.

        Activity

        Hide
        Adam Muise added a comment -

        Yes, this should probably look like rsync.

        No, sqoop does not support this use case.

        Show
        Adam Muise added a comment - Yes, this should probably look like rsync. No, sqoop does not support this use case.
        Hide
        Eli Collins added a comment -

        Yea, something like rsync or Sqoop for file systems seems more appropriate.

        Show
        Eli Collins added a comment - Yea, something like rsync or Sqoop for file systems seems more appropriate.
        Hide
        Robert Joseph Evans added a comment -

        It almost sounds like you want to turn this into something like rsync. I think it would be much more useful to just add in an rsync command with a simmilar set of features and flags then trying to reinvent it piecemeal. Then it can look at time stamps on the files, and possibly checksums as well, to pick up where it left off on a failure.

        Show
        Robert Joseph Evans added a comment - It almost sounds like you want to turn this into something like rsync. I think it would be much more useful to just add in an rsync command with a simmilar set of features and flags then trying to reinvent it piecemeal. Then it can look at time stamps on the files, and possibly checksums as well, to pick up where it left off on a failure.

          People

          • Assignee:
            Unassigned
            Reporter:
            Adam Muise
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development