Details

    • Hadoop Flags:
      Reviewed

      Description

      Block location type is changed from datanode to datanode storage. Balancer needs to handle it.

      1. h5480_20131108.patch
        43 kB
        Tsz Wo Nicholas Sze
      2. h5480_20131108b.patch
        28 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Arpit Agarwal added a comment -

          +1 for the patch. I committed it to branch HDFS-2832. Thanks Nicholas!

          Show
          Arpit Agarwal added a comment - +1 for the patch. I committed it to branch HDFS-2832 . Thanks Nicholas!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h5480_20131108b.patch: fixes all balancer related tests.

          Running org.apache.hadoop.hdfs.server.balancer.TestBalancer
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.961 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancer
          Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.533 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.082 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.416 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.72 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          Running org.apache.hadoop.hdfs.TestBalancerBandwidth
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.708 sec - in org.apache.hadoop.hdfs.TestBalancerBandwidth

          Show
          Tsz Wo Nicholas Sze added a comment - h5480_20131108b.patch: fixes all balancer related tests. Running org.apache.hadoop.hdfs.server.balancer.TestBalancer Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.961 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancer Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.533 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.082 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.416 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.72 sec - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup Running org.apache.hadoop.hdfs.TestBalancerBandwidth Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.708 sec - in org.apache.hadoop.hdfs.TestBalancerBandwidth
          Hide
          Junping Du added a comment -

          Good to know it. Thanks Nicholas!
          So I will mark HDFS-5479 as duplicated and will help to review the patch when it is ready. Thanks!

          Show
          Junping Du added a comment - Good to know it. Thanks Nicholas! So I will mark HDFS-5479 as duplicated and will help to review the patch when it is ready. Thanks!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Junping, I was trying to fix balancer related tests so that we could merge the branch. So I think there is only a little overlap with HDFS-4989.

          The balancer did not have storage information so that I added getDatanodeReportWithStorage. It seems that it may be better to just add datanode information to BlockWithLocations. I will try it.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Junping, I was trying to fix balancer related tests so that we could merge the branch. So I think there is only a little overlap with HDFS-4989 . The balancer did not have storage information so that I added getDatanodeReportWithStorage. It seems that it may be better to just add datanode information to BlockWithLocations. I will try it.
          Hide
          Junping Du added a comment -

          Hi Nicholas, thanks for jira and patch. Do you think this JIRA is overlapped with HDFS-4989? or we have different phases and HDFS-4989 can address more later? Would like to get your comments. Thanks!

          Show
          Junping Du added a comment - Hi Nicholas, thanks for jira and patch. Do you think this JIRA is overlapped with HDFS-4989 ? or we have different phases and HDFS-4989 can address more later? Would like to get your comments. Thanks!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h5480_20131108.patch:

          • adds ClientProtocol.getDatanodeReportWithStorage;
          • adds a storage map to Balancer; and
          • fixes the StorageReport in SimulatedFSDataset.

          Still need to fix unit tests.

          Show
          Tsz Wo Nicholas Sze added a comment - h5480_20131108.patch: adds ClientProtocol.getDatanodeReportWithStorage; adds a storage map to Balancer; and fixes the StorageReport in SimulatedFSDataset. Still need to fix unit tests.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development