Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HA Branch (HDFS-1623)
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None

      Description

      HADOOP-7896 (on the HA branch) refactored RetryAction from an enum to a class, and also moved the act of sleeping to delay retries from the RetryPolicy implementations into RetryInvocationHandler. RetriableCommand, in the rewritten distcp tool, uses RetryPolicy and associated classes from o.a.h.io.retry. When MAPREDUCE-2765 was merged into the HA branch, RetriableCommand wasn't adjusted accordingly to make use of the new structure of the o.a.h.io.retry classes.

      It's probably generally not kosher for RetriableCommand to be using the RetryPolicy classes at all, since they're not really intended to be used except by RetryInvocationHandler. But, regardless, this JIRA aims to make distcp's use of the o.a.h.io.retry classes functional again.

      1. HADOOP-8116-HDFS-1623.patch
        3 kB
        Aaron T. Myers
      2. HADOOP-8116-HDFS-1623.patch
        3 kB
        Aaron T. Myers

        Activity

        Hide
        Aaron T. Myers added a comment -

        Here's a patch which addresses the issue. I also noticed while fixing this issue that TestCopyMapper wouldn't have worked because HADOOP-8068 results in an additional exception wrapping, which the test wasn't taking into account.

        Show
        Aaron T. Myers added a comment - Here's a patch which addresses the issue. I also noticed while fixing this issue that TestCopyMapper wouldn't have worked because HADOOP-8068 results in an additional exception wrapping, which the test wasn't taking into account.
        Hide
        Aaron T. Myers added a comment -

        Oh, I should've mentioned - the two test cases which will fail without this patch are TestCopyMapper and TestRetriableCommand.

        Show
        Aaron T. Myers added a comment - Oh, I should've mentioned - the two test cases which will fail without this patch are TestCopyMapper and TestRetriableCommand.
        Hide
        Eli Collins added a comment -

        +1 looks great

        Nit, the line where you pull out the nested exception in TestCopyMapper could use a comment indicating why this is necessary

        Also wrt your comment that it's not kosher for RetriableCommand to be using the RetryPolicy classes, file a MR jira for that?

        Show
        Eli Collins added a comment - +1 looks great Nit, the line where you pull out the nested exception in TestCopyMapper could use a comment indicating why this is necessary Also wrt your comment that it's not kosher for RetriableCommand to be using the RetryPolicy classes, file a MR jira for that?
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for the review, Eli. Here's an updated patch which adds the comment you asked for. I'm going to commit this in a moment, and create the JIRA you asked for.

        Show
        Aaron T. Myers added a comment - Thanks a lot for the review, Eli. Here's an updated patch which adds the comment you asked for. I'm going to commit this in a moment, and create the JIRA you asked for.
        Hide
        Aaron T. Myers added a comment -

        I've committed this to the HA branch and filed the following JIRA: MAPREDUCE-3934

        Thanks a lot for the review, Eli.

        Show
        Aaron T. Myers added a comment - I've committed this to the HA branch and filed the following JIRA: MAPREDUCE-3934 Thanks a lot for the review, Eli.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-HAbranch-build #92 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/92/)
        HADOOP-8116. RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896. Contributed by Aaron T. Myers. (Revision 1294729)

        Result = UNSTABLE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294729
        Files :

        • /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/CHANGES.HDFS-1623.txt
        • /hadoop/common/branches/HDFS-1623/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java
        • /hadoop/common/branches/HDFS-1623/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-HAbranch-build #92 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/92/ ) HADOOP-8116 . RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896 . Contributed by Aaron T. Myers. (Revision 1294729) Result = UNSTABLE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294729 Files : /hadoop/common/branches/ HDFS-1623 /hadoop-common-project/hadoop-common/CHANGES. HDFS-1623 .txt /hadoop/common/branches/ HDFS-1623 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java /hadoop/common/branches/ HDFS-1623 /hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java

          People

          • Assignee:
            Aaron T. Myers
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development