Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: datanode, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira makes protocol buffers protocol changes required for HDFS-2832. HDFS-2832 will have the following subtasks: protocol buffer changes, protocol changes, followed by jiras for implementation changes.

      1. HDFS-2880.txt
        9 kB
        Suresh Srinivas
      2. HDFS-2880.txt
        10 kB
        Suresh Srinivas
      3. HDFS-2880.txt
        10 kB
        Suresh Srinivas
      4. HDFS-2880.txt
        12 kB
        Suresh Srinivas

        Activity

        Hide
        Suresh Srinivas added a comment -

        Patch makes the following changes protocol buffers:

        1. Registration request can send list of storages each with state and stroage ID.
        2. Heartbeat request is changed to send utilization per storage.
        3. Block report request is changed to send report per storage.
        4. Block received and deleted list is sent per storage.

        The change is limited to protocol buffers and translators alone. The rest of the implementation remains the same with only one storage per datanode.

        Show
        Suresh Srinivas added a comment - Patch makes the following changes protocol buffers: Registration request can send list of storages each with state and stroage ID. Heartbeat request is changed to send utilization per storage. Block report request is changed to send report per storage. Block received and deleted list is sent per storage. The change is limited to protocol buffers and translators alone. The rest of the implementation remains the same with only one storage per datanode.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12512968/HDFS-2880.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
        org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512968/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Updated patch to fix test issue.

        Show
        Suresh Srinivas added a comment - Updated patch to fix test issue.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12513045/HDFS-2880.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513045/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        I think we should also modify BlockReceivedAndDeletedRequestProto using newly defined StorageReceivedDeletedBlocksProto.

        Minor:

        1. In DatanodeStorageProto I will recommend to define optional field after the required field just to follow a convention.
        2. In HeartbeatRequestProto, please rename report to reports because this is a repeated field.
        Show
        Jitendra Nath Pandey added a comment - I think we should also modify BlockReceivedAndDeletedRequestProto using newly defined StorageReceivedDeletedBlocksProto. Minor: In DatanodeStorageProto I will recommend to define optional field after the required field just to follow a convention. In HeartbeatRequestProto, please rename report to reports because this is a repeated field.
        Hide
        Suresh Srinivas added a comment -

        Thanks for the review Jitendra. New patch addresses all the comments.

        Show
        Suresh Srinivas added a comment - Thanks for the review Jitendra. New patch addresses all the comments.
        Hide
        Jitendra Nath Pandey added a comment -

        +1

        Show
        Jitendra Nath Pandey added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12513260/HDFS-2880.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513260/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        This is only protocol layer change. The existing tests cover this. I committed the patch.

        Show
        Suresh Srinivas added a comment - This is only protocol layer change. The existing tests cover this. I committed the patch.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1737/)
        HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1737/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1665/)
        HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1665/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1680 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1680/)
        HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1680 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1680/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #947 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/947/)
        HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #947 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/947/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/980/)
        HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/980/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have merged this to 0.23.

        Show
        Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development