Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None

      Description

      DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:

      1. DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
      2. Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
      3. If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent.

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          This patch made the changes suggested above. It also optimizes SimulatedFSDataset#injectBlocks by avoid allocating a new block map.

          Show
          Hairong Kuang added a comment - This patch made the changes suggested above. It also optimizes SimulatedFSDataset#injectBlocks by avoid allocating a new block map.
          Hide
          Steve Loughran added a comment -

          #Does CreateEditsLog have to create a default Configuration() instance, or could createEditsLog() be changed to take a Configuration as a parameter?

          1. It might be useful to think at some point in time to move these MiniDFSCluster and DataNodeCluster classes into a redistributable JAR, as they are handy to anyone trying to set up a test cluster in a single JVM -though we'd have to get rid of all emergency System.exit() calls first.
          Show
          Steve Loughran added a comment - #Does CreateEditsLog have to create a default Configuration() instance, or could createEditsLog() be changed to take a Configuration as a parameter? It might be useful to think at some point in time to move these MiniDFSCluster and DataNodeCluster classes into a redistributable JAR, as they are handy to anyone trying to set up a test cluster in a single JVM -though we'd have to get rid of all emergency System.exit() calls first.
          Hide
          Ravi Phulari added a comment -

          Submitting updated patch .
          Changed path to work with pro split code .

          Show
          Ravi Phulari added a comment - Submitting updated patch . Changed path to work with pro split code .
          Hide
          Ravi Phulari added a comment -

          HADOOP-5556.patch is for 0.21 , patch specific to pro split code.
          successfully ran ant test-core , waiting for Hudson .

          Show
          Ravi Phulari added a comment - HADOOP-5556 .patch is for 0.21 , patch specific to pro split code. successfully ran ant test-core , waiting for Hudson .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417111/HADOOP-5556.patch
          against trunk revision 804918.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/616/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417111/HADOOP-5556.patch against trunk revision 804918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/616/console This message is automatically generated.
          Hide
          Ravi Phulari added a comment -

          Canceling patch .

          Show
          Ravi Phulari added a comment - Canceling patch .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12403474/DataNodeCluster.patch
          against trunk revision 804918.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/617/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12403474/DataNodeCluster.patch against trunk revision 804918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/617/console This message is automatically generated.
          Hide
          Chris Douglas added a comment -

          Reopened as HDFS-555

          Show
          Chris Douglas added a comment - Reopened as HDFS-555

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development