Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
  3. HDFS-5452

Fix TestReplicationPolicy and TestBlocksScheduledCounter

        Details

        • Hadoop Flags:
          Reviewed

          Description

          TestReplicationPolicy: the log level checking is incorrect.

          TestBlocksScheduledCounter: HDFS-5222, which moved block schedule information from DatanodeDescriptor to DatanodeStorageInfo, is incorrect since a replica scheduled to a storage may eventually be written to another storage in the same datanode.

          1. Text File
            h5452_20131101.patch
            11 kB
            Tsz Wo Nicholas Sze
          2. Text File
            h5452_20131103.patch
            11 kB
            Tsz Wo Nicholas Sze

            Issue Links

            is broken by

            Sub-task - The sub-task of the issue HDFS-5222 Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              h5452_20131101.patch: fixes the tests and cleans up some code.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - h5452_20131101.patch: fixes the tests and cleans up some code.
              Hide
              Permalink
              arpitagarwal Arpit Agarwal added a comment -

              +1 for the patch.

              I verified it fixes TestBlocksScheduledCounter. It looks like one test case is still failing in TestReplicationPolicy. We can fix it in a separate Jira.

              java.lang.NullPointerException: null
        at org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy.testChooseTargetWithMoreThanHalfStaleNodes(TestReplicationPolicy.java:611)
              Show
              arpitagarwal Arpit Agarwal added a comment - +1 for the patch. I verified it fixes TestBlocksScheduledCounter. It looks like one test case is still failing in TestReplicationPolicy. We can fix it in a separate Jira. java.lang.NullPointerException: null at org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy.testChooseTargetWithMoreThanHalfStaleNodes(TestReplicationPolicy.java:611)
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              Just checked TestReplicationPolicy. It did not fail in my machine. Not sure why it still fails in your case.

              Running org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.514 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy

Results :

Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - Just checked TestReplicationPolicy. It did not fail in my machine. Not sure why it still fails in your case. Running org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.514 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Results : Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              h5452_20131103.patch: updated with the branch.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - h5452_20131103.patch: updated with the branch.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              Thanks Arpit for reviewing the patch.

              I have committed this.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Arpit for reviewing the patch. I have committed this.
              Hide
              Permalink
              arpitagarwal Arpit Agarwal added a comment -

              Both tests succeed after I synced the source tree.

              Show
              arpitagarwal Arpit Agarwal added a comment - Both tests succeed after I synced the source tree.

                People

                • Assignee:
                  szetszwo Tsz Wo Nicholas Sze
                  Reporter:
                  szetszwo Tsz Wo Nicholas Sze
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development