Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2968

Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: datanode, namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If there are multiple blocks to be recovered, it ends up translating to N copies of the first block instead of the N different blocks.

      1. hdfs-2968.txt
        6 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          The PBHelper code has:

              for (int i = 0; i < list.size(); i++) {
                recoveringBlocks.add(PBHelper.convert(list.get(0)));
              }
          

          which is obviously wrong

          Show
          Todd Lipcon added a comment - The PBHelper code has: for ( int i = 0; i < list.size(); i++) { recoveringBlocks.add(PBHelper.convert(list.get(0))); } which is obviously wrong
          Hide
          Aaron T. Myers added a comment -

          Good catch, Todd. Do you not think this should also go in 0.23.1?

          Show
          Aaron T. Myers added a comment - Good catch, Todd. Do you not think this should also go in 0.23.1?
          Hide
          Todd Lipcon added a comment -

          0.23.1 doesn't have protobufs in place yet, right?

          Show
          Todd Lipcon added a comment - 0.23.1 doesn't have protobufs in place yet, right?
          Hide
          Todd Lipcon added a comment -

          Attached patch fixes the issue and adds a unit test

          Show
          Todd Lipcon added a comment - Attached patch fixes the issue and adds a unit test
          Hide
          Aaron T. Myers added a comment -

          0.23.1 doesn't have protobufs in place yet, right?

          Whoops! Of course you're right.

          Show
          Aaron T. Myers added a comment - 0.23.1 doesn't have protobufs in place yet, right? Whoops! Of course you're right.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. Good find, Todd.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. Good find, Todd.
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and branch-0.23-PB. Thanks for review, Aaron.

          Show
          Todd Lipcon added a comment - Committed to trunk and branch-0.23-PB. Thanks for review, Aaron.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1821/)
          HDFS-2968. Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1821/ ) HDFS-2968 . Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1747 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1747/)
          HDFS-2968. Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1747 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1747/ ) HDFS-2968 . Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1759 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1759/)
          HDFS-2968. Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832)

          Result = ABORTED
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1759 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1759/ ) HDFS-2968 . Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #959 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/959/)
          HDFS-2968. Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #959 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/959/ ) HDFS-2968 . Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #994 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/994/)
          HDFS-2968. Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #994 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/994/ ) HDFS-2968 . Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery. Contributed by Todd Lipcon. (Revision 1245832) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245832 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have merged this to 0.23.

          Show
          Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development