Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
  3. HDFS-2880

Protocol buffer changes to add support multiple storages

        Details

        • Type: Sub-task
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 2.0.0-alpha
        • Component/s: datanode, namenode
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          This jira makes protocol buffers protocol changes required for HDFS-2832. HDFS-2832 will have the following subtasks: protocol buffer changes, protocol changes, followed by jiras for implementation changes.

          1. Text File
            HDFS-2880.txt
            12 kB
            Suresh Srinivas
          2. Text File
            HDFS-2880.txt
            10 kB
            Suresh Srinivas
          3. Text File
            HDFS-2880.txt
            10 kB
            Suresh Srinivas
          4. Text File
            HDFS-2880.txt
            9 kB
            Suresh Srinivas

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            sureshms Suresh Srinivas added a comment -

            Patch makes the following changes protocol buffers:

            1. Registration request can send list of storages each with state and stroage ID.
            2. Heartbeat request is changed to send utilization per storage.
            3. Block report request is changed to send report per storage.
            4. Block received and deleted list is sent per storage.

            The change is limited to protocol buffers and translators alone. The rest of the implementation remains the same with only one storage per datanode.

            Show
            sureshms Suresh Srinivas added a comment - Patch makes the following changes protocol buffers: Registration request can send list of storages each with state and stroage ID. Heartbeat request is changed to send utilization per storage. Block report request is changed to send report per storage. Block received and deleted list is sent per storage. The change is limited to protocol buffers and translators alone. The rest of the implementation remains the same with only one storage per datanode.
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12512968/HDFS-2880.txt
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed these unit tests:
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512968/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1834//console This message is automatically generated.
            Hide
            Permalink
            sureshms Suresh Srinivas added a comment -

            Updated patch to fix test issue.

            Show
            sureshms Suresh Srinivas added a comment - Updated patch to fix test issue.
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12513045/HDFS-2880.txt
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in .

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513045/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1836//console This message is automatically generated.
            Hide
            Permalink
            jnp Jitendra Nath Pandey added a comment -

            I think we should also modify BlockReceivedAndDeletedRequestProto using newly defined StorageReceivedDeletedBlocksProto.

            Minor:

            1. In DatanodeStorageProto I will recommend to define optional field after the required field just to follow a convention.
            2. In HeartbeatRequestProto, please rename report to reports because this is a repeated field.
            Show
            jnp Jitendra Nath Pandey added a comment - I think we should also modify BlockReceivedAndDeletedRequestProto using newly defined StorageReceivedDeletedBlocksProto. Minor: In DatanodeStorageProto I will recommend to define optional field after the required field just to follow a convention. In HeartbeatRequestProto, please rename report to reports because this is a repeated field.
            Hide
            Permalink
            sureshms Suresh Srinivas added a comment -

            Thanks for the review Jitendra. New patch addresses all the comments.

            Show
            sureshms Suresh Srinivas added a comment - Thanks for the review Jitendra. New patch addresses all the comments.
            Hide
            Permalink
            jnp Jitendra Nath Pandey added a comment -

            +1

            Show
            jnp Jitendra Nath Pandey added a comment - +1
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12513260/HDFS-2880.txt
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in .

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513260/HDFS-2880.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1842//console This message is automatically generated.
            Hide
            Permalink
            sureshms Suresh Srinivas added a comment -

            This is only protocol layer change. The existing tests cover this. I committed the patch.

            Show
            sureshms Suresh Srinivas added a comment - This is only protocol layer change. The existing tests cover this. I committed the patch.
            Hide
            Permalink
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1737/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1737/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #1665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1665/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1665/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #1680 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1680/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1680 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1680/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-trunk #947 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/947/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #947 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/947/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            hudson Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/980/)
            HDFS-2880. Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas.

            suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653
            Files :

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Show
            hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/980/ ) HDFS-2880 . Protobuf chagnes in DatanodeProtocol to add multiple storages. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240653 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            I have merged this to 0.23.

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

              People

              • Assignee:
                sureshms Suresh Srinivas
                Reporter:
                sureshms Suresh Srinivas
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development