Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: test
    • Labels:
      None

      Description

      Picking up a thread again from MAPREDUCE-987, I've found it very useful to have a CLI driver for running a single-process DFS cluster, particularly when developing features in HDFS clients. For example, being able to spin up a local cluster easily was tremendously useful for correctness testing of HDFS-2834.

      I'd like to contribute a class based on the patch for MAPREDUCE-987 we've been using fairly extensively. Only for DFS, not MR since much has changed MR-side since the original patch.

      1. HDFS-3167.patch
        10 kB
        Henry Robinson
      2. HDFS-3167.1.patch
        10 kB
        Henry Robinson

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          20m 4s 1 Aaron T. Myers 30/Mar/12 22:45
          Patch Available Patch Available Resolved Resolved
          1d 19h 46m 1 Aaron T. Myers 01/Apr/12 18:31
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1038 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1038/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1038 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1038/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1003 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1003/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1003 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1003/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1975 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1975/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = ABORTED
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1975 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1975/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1962 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1962/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1962 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1962/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2037 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2037/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2037 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2037/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Aaron T. Myers made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Resolution Fixed [ 1 ]
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Hank.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Hank.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. I'm going to commit this momentarily.
          Jeff Hammerbacher made changes -
          Link This issue is related to MAPREDUCE-987 [ MAPREDUCE-987 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520769/HDFS-3167.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520769/HDFS-3167.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//console This message is automatically generated.
          Henry Robinson made changes -
          Attachment HDFS-3167.1.patch [ 12520769 ]
          Hide
          Henry Robinson added a comment -

          Removed unused 'stop', and unused default for nn port, also moved all System.*.println to LOG.

          Regarding testing: informally I've been using it internally for a couple of months, and I've worked through the command line options to verify that they behave as expected before creating this patch.

          Show
          Henry Robinson added a comment - Removed unused 'stop', and unused default for nn port, also moved all System.*.println to LOG. Regarding testing: informally I've been using it internally for a couple of months, and I've worked through the command line options to verify that they behave as expected before creating this patch.
          Hide
          Aaron T. Myers added a comment -

          Hey Henry, patch looks pretty good after a quick look. I haven't actually run it, though. Can you comment on what testing you've done?

          A few little things I noticed:

          1. Why do you initialize nameNodePort to 20500? I don't think it will actually get used, since you later specify "0" as the default.
          2. It's not obvious to me why you output some error messages using LOG.info(...), and others using System.err.println(...). Unless there's some good reason, I'd suggest you either be consistent or add a comment explaining what the distinction for using one vs. the other is.
          3. I don't see how stop(...) will ever be called.
          Show
          Aaron T. Myers added a comment - Hey Henry, patch looks pretty good after a quick look. I haven't actually run it, though. Can you comment on what testing you've done? A few little things I noticed: Why do you initialize nameNodePort to 20500? I don't think it will actually get used, since you later specify "0" as the default. It's not obvious to me why you output some error messages using LOG.info(...), and others using System.err.println(...). Unless there's some good reason, I'd suggest you either be consistent or add a comment explaining what the distinction for using one vs. the other is. I don't see how stop(...) will ever be called.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520674/HDFS-3167.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520674/HDFS-3167.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//console This message is automatically generated.
          Aaron T. Myers made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.0.0 [ 12320353 ]
          Target Version/s 2.0.0 [ 12320353 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Hide
          Aaron T. Myers added a comment -

          Marking patch available for Henry so that test-patch runs.

          Show
          Aaron T. Myers added a comment - Marking patch available for Henry so that test-patch runs.
          Henry Robinson made changes -
          Field Original Value New Value
          Attachment HDFS-3167.patch [ 12520674 ]
          Hide
          Henry Robinson added a comment -

          Patch for trunk. Example invocation instructions in the class Javadoc.

          Show
          Henry Robinson added a comment - Patch for trunk. Example invocation instructions in the class Javadoc.
          Henry Robinson created issue -

            People

            • Assignee:
              Henry Robinson
              Reporter:
              Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development