Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12469

distcp should not ignore the ignoreFailures option

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: tools/distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      RetriableFileCopyCommand.CopyReadException is double-wrapped via

      1. via RetriableCommand::execute
      2. via CopyMapper#copyFileWithRetry

      before CopyMapper::handleFailure tests

      if (ignoreFailures && exception.getCause() instanceof
                  RetriableFileCopyCommand.CopyReadException
      

      which is always false.

      Orthogonally, ignoring failures should be mutually exclusive with the atomic option otherwise an incomplete dir is eligible for commit defeating the purpose.

        Attachments

        1. HADOOP-12469.000.patch
          2 kB
          Mingliang Liu
        2. HADOOP-12469.001.patch
          7 kB
          Mingliang Liu
        3. HADOOP-12469.002.patch
          7 kB
          Mingliang Liu
        4. HADOOP-12469.003.patch
          8 kB
          Mingliang Liu
        5. HADOOP-12469.004.patch
          7 kB
          Mingliang Liu
        6. HADOOP-12469.005.patch
          7 kB
          Mingliang Liu
        7. HADOOP-12469.005.patch
          7 kB
          Mingliang Liu
        8. HADOOP-12469.006.patch
          6 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                liuml07 Mingliang Liu
                Reporter:
                jira.shegalov Gera Shegalov
              • Votes:
                1 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: