Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
  3. HDFS-2880

Protocol buffer changes to add support multiple storages

        Details

        • Type: Sub-task Sub-task
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 2.0.0-alpha
        • Component/s: datanode, namenode
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          This jira makes protocol buffers protocol changes required for HDFS-2832. HDFS-2832 will have the following subtasks: protocol buffer changes, protocol changes, followed by jiras for implementation changes.

          1. Text File
            HDFS-2880.txt
            12 kB
            Suresh Srinivas
          2. Text File
            HDFS-2880.txt
            10 kB
            Suresh Srinivas
          3. Text File
            HDFS-2880.txt
            10 kB
            Suresh Srinivas
          4. Text File
            HDFS-2880.txt
            9 kB
            Suresh Srinivas

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Suresh Srinivas added a comment -

            Patch makes the following changes protocol buffers:

            1. Registration request can send list of storages each with state and stroage ID.
            2. Heartbeat request is changed to send utilization per storage.
            3. Block report request is changed to send report per storage.
            4. Block received and deleted list is sent per storage.

            The change is limited to protocol buffers and translators alone. The rest of the implementation remains the same with only one storage per datanode.

            Show
            Suresh Srinivas added a comment - Patch makes the following changes protocol buffers: Registration request can send list of storages each with state and stroage ID. Heartbeat request is changed to send utilization per storage. Block report request is changed to send report per storage. Block received and deleted list is sent per storage. The change is limited to protocol buffers and translators alone. The rest of the implementation remains the same with only one storage per datanode.
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12512968/HDFS-2880.txt
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed these unit tests:
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512968/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//console This message is automatically generated.
            Hide
            Permalink
            Suresh Srinivas added a comment -

            Updated patch to fix test issue.

            Show
            Suresh Srinivas added a comment - Updated patch to fix test issue.
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12513045/HDFS-2880.txt
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in .

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513045/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//console This message is automatically generated.
            Hide
            Permalink
            Jitendra Nath Pandey added a comment -

            I think we should also modify BlockReceivedAndDeletedRequestProto using newly defined StorageReceivedDeletedBlocksProto.

            Minor:

            1. In DatanodeStorageProto I will recommend to define optional field after the required field just to follow a convention.
            2. In HeartbeatRequestProto, please rename report to reports because this is a repeated field.
            Show
            Jitendra Nath Pandey added a comment - I think we should also modify BlockReceivedAndDeletedRequestProto using newly defined StorageReceivedDeletedBlocksProto. Minor: In DatanodeStorageProto I will recommend to define optional field after the required field just to follow a convention. In HeartbeatRequestProto, please rename report to reports because this is a repeated field.
            Hide
            Permalink
            Suresh Srinivas added a comment -

            Thanks for the review Jitendra. New patch addresses all the comments.

            Show
            Suresh Srinivas added a comment - Thanks for the review Jitendra. New patch addresses all the comments.
            Hide
            Permalink
            Jitendra Nath Pandey added a comment -

            +1

            Show
            Jitendra Nath Pandey added a comment - +1
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12513260/HDFS-2880.txt
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in .

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513260/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//console This message is automatically generated.
            Hide
            Permalink
            Suresh Srinivas added a comment -

            This is only protocol layer change. The existing tests cover this. I committed the patch.

            Show
            Suresh Srinivas added a comment - This is only protocol layer change. The existing tests cover this. I committed the patch.
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1737/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1737/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #1665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1665/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1665/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #1680 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1680/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1680 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1680/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk #947 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/947/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #947 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/947/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/980/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/980/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            Tsz Wo Nicholas Sze added a comment -

            I have merged this to 0.23.

            Show
            Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

              People

              • Assignee:
                Suresh Srinivas
                Reporter:
                Suresh Srinivas
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development