Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13489

DistCp may incorrectly return success status when the underlying Job failed



    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      I was troubleshooting HBASE-14450 where at the end of BackupdistCp#execute(), distcp job was marked unsuccessful (BackupdistCp is a wrapper of DistCp).
      Yet in IncrementalTableBackupProcedure#incrementalCopy(), the return value from copyService.copy() was 0.

      Here is related code from DistCp:

          try {
          } catch (InvalidInputException e) {
            LOG.error("Invalid input: ", e);
            return DistCpConstants.INVALID_ARGUMENT;
          } catch (DuplicateFileException e) {
            LOG.error("Duplicate files in input path: ", e);
            return DistCpConstants.DUPLICATE_INPUT;
          } catch (AclsNotSupportedException e) {
            LOG.error("ACLs not supported on at least one file system: ", e);
            return DistCpConstants.ACLS_NOT_SUPPORTED;
          } catch (XAttrsNotSupportedException e) {
            LOG.error("XAttrs not supported on at least one file system: ", e);
            return DistCpConstants.XATTRS_NOT_SUPPORTED;
          } catch (Exception e) {
            LOG.error("Exception encountered ", e);
            return DistCpConstants.UNKNOWN_ERROR;
          return DistCpConstants.SUCCESS;

      We don't check whether the Job returned by execute() was successful - we rely on all failure cases going through the last catch clause but there may be special case.
      Even if the Job fails, DistCpConstants.SUCCESS is returned.


        1. testIncrementalBackup-8-12-rethrow.txt
          308 kB
          Ted Yu
        2. MapReduceBackupCopyService.java
          13 kB
          Ted Yu
        3. testIncrementalBackup-8-12.txt
          305 kB
          Ted Yu
        4. HADOOP-13489.v3.patch
          1 kB
          Ted Yu
        5. HADOOP-13489.v2.patch
          1 kB
          Ted Yu
        6. TestIncrementalBackup-output.txt
          20 kB
          Ted Yu
        7. HADOOP-13489.v1.patch
          1.0 kB
          Ted Yu

          Issue Links



              • Assignee:
                yuzhihong@gmail.com Ted Yu
                yuzhihong@gmail.com Ted Yu
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: