Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15790

distcp app should fail if m/r job fails

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1, 3.2.0
    • Fix Version/s: None
    • Component/s: tools/distcp
    • Labels:
    • Tags:
      distcp distcpv2

      Description

      I run distcpv2 in a scripted manner. The script checks if the distcp step fails and, if so, aborts the rest of the script. However, I ran into an issue today where the distcp job failed, but my calling script went on its merry way.

      Digging into the code a bit more (at https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java), I think I see the issue: the distcp app is not returning an error exit code to the shell when the distcp job fails. This is a big problem, IMO, as it prevents distcp from being successfully used in a scripted environment. IMO, the code should change like so:

      Before:

      org.apache.hadoop.tools.DistCp.java
      //...
        public int run(String[] argv) {
      //...
          try {
            execute();
          } catch (InvalidInputException e) {
            LOG.error("Invalid input: ", e);
            return DistCpConstants.INVALID_ARGUMENT;
          } catch (DuplicateFileException e) {
            LOG.error("Duplicate files in input path: ", e);
            return DistCpConstants.DUPLICATE_INPUT;
          } catch (Exception e) {
            LOG.error("Exception encountered ", e);
            return DistCpConstants.UNKNOWN_ERROR;
          }
          return DistCpConstants.SUCCESS;
        }
      //...
      

      After:

      org.apache.hadoop.tools.DistCp.java
      //...
        public int run(String[] argv) {
      //...
          Job job = null;
          try {
            job = execute();
          } catch (InvalidInputException e) {
            LOG.error("Invalid input: ", e);
            return DistCpConstants.INVALID_ARGUMENT;
          } catch (DuplicateFileException e) {
            LOG.error("Duplicate files in input path: ", e);
            return DistCpConstants.DUPLICATE_INPUT;
          } catch (Exception e) {
            LOG.error("Exception encountered ", e);
            return DistCpConstants.UNKNOWN_ERROR;
          }
      
          if (job.isSuccessful()) {
            return DistCpConstants.SUCCESS;
          }
          else {
            return DistCpConstants.UNKNOWN_ERROR;
          }
        }
      //...
      

        Attachments

        1. MAPREDUCE-5549-002.patch
          3 kB
          Steve Loughran
        2. MAPREDUCE-5549-001.patch
          2 kB
          Steve Loughran

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                darose David Rosenstrauch
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: