Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
  3. HDFS-5406

Send incremental block reports for all storages in a single call

        Details

        • Type: Sub-task Sub-task
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 3.0.0
        • Fix Version/s: 3.0.0, 2.3.0
        • Component/s: datanode
        • Labels:
          None
        • Hadoop Flags:
          Reviewed
        • Target Version/s:

          Description

          Per code review feedback from Tsz Wo Nicholas Sze on HDFS-5390, we can combine all incremental block reports in a single blockReceivedAndDeleted call.

          1. Text File
            h5406.05.patch
            28 kB
            Arpit Agarwal
          2. Text File
            h5406.04.patch
            3 kB
            Arpit Agarwal
          3. Text File
            h5406.02.patch
            5 kB
            Arpit Agarwal
          4. Text File
            h5406.01.patch
            5 kB
            Arpit Agarwal

            Issue Links

            is duplicated by

            Sub-task - The sub-task of the issue HDFS-5593 Test cases for split and combined block reports

            • Major - Major loss of function.
            • Resolved
            is related to

            Sub-task - The sub-task of the issue HDFS-5390 Send one incremental block report per storage directory

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Arpit Agarwal added a comment -

              Patch to combine all incremental reports in a single call to NN.

              Show
              Arpit Agarwal added a comment - Patch to combine all incremental reports in a single call to NN.
              Hide
              Permalink
              Arpit Agarwal added a comment -

              Change target version to 3.0.0 and submit patch since the feature branch has been merged into trunk now.

              Show
              Arpit Agarwal added a comment - Change target version to 3.0.0 and submit patch since the feature branch has been merged into trunk now.
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12616912/h5406.02.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

              org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
              org.apache.hadoop.hdfs.server.datanode.TestBPOfferService

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5709//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5709//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12616912/h5406.02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.server.datanode.TestBPOfferService +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5709//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5709//console This message is automatically generated.
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12618744/h5406.04.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5719//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5719//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618744/h5406.04.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5719//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5719//console This message is automatically generated.
              Hide
              Permalink
              Arpit Agarwal added a comment -

              Updated patch to add new test case for this fix (TestIncrementalBrVariations#testDataNodeDoesNotSplitReports) and also include all tests from HDFS-5593.

              Show
              Arpit Agarwal added a comment - Updated patch to add new test case for this fix ( TestIncrementalBrVariations#testDataNodeDoesNotSplitReports ) and also include all tests from HDFS-5593 .
              Hide
              Permalink
              Hadoop QA added a comment -

              +1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12618796/h5406.05.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 3 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5724//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5724//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618796/h5406.05.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5724//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5724//console This message is automatically generated.
              Hide
              Permalink
              Tsz Wo Nicholas Sze added a comment -

              +1 patch looks good.

              Show
              Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
              Hide
              Permalink
              Arpit Agarwal added a comment -

              Thanks Nicholas, I committed this to trunk.

              Show
              Arpit Agarwal added a comment - Thanks Nicholas, I committed this to trunk.
              Hide
              Permalink
              Hudson added a comment -

              SUCCESS: Integrated in Hadoop-trunk-Commit #4884 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4884/)
              HDFS-5406. Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093)

              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Show
              Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4884 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4884/ ) HDFS-5406 . Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Hide
              Permalink
              Hudson added a comment -

              SUCCESS: Integrated in Hadoop-trunk-Commit #4885 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4885/)
              HDFS-5406. Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110)

              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
              Show
              Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4885 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4885/ ) HDFS-5406 . Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
              Hide
              Permalink
              Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk #423 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/423/)
              HDFS-5406. Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110)

              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
                HDFS-5406. Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093)
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Show
              Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #423 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/423/ ) HDFS-5406 . Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java HDFS-5406 . Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Hide
              Permalink
              Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk #1614 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1614/)
              HDFS-5406. Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110)

              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
                HDFS-5406. Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093)
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Show
              Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1614 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1614/ ) HDFS-5406 . Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java HDFS-5406 . Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Hide
              Permalink
              Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk #1640 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1640/)
              HDFS-5406. Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110)

              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
                HDFS-5406. Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093)
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
              • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Show
              Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1640 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1640/ ) HDFS-5406 . Add file missed in previous commit. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551110 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java HDFS-5406 . Send incremental block reports for all storages in a single call. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551093 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
              Hide
              Permalink
              Arpit Agarwal added a comment -

              Merged to branch-2. Target version set to 2.4.0.

              Show
              Arpit Agarwal added a comment - Merged to branch-2. Target version set to 2.4.0.

                People

                • Assignee:
                  Arpit Agarwal
                  Reporter:
                  Arpit Agarwal
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development