Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: test
    • Labels:
      None

      Description

      Picking up a thread again from MAPREDUCE-987, I've found it very useful to have a CLI driver for running a single-process DFS cluster, particularly when developing features in HDFS clients. For example, being able to spin up a local cluster easily was tremendously useful for correctness testing of HDFS-2834.

      I'd like to contribute a class based on the patch for MAPREDUCE-987 we've been using fairly extensively. Only for DFS, not MR since much has changed MR-side since the original patch.

      1. HDFS-3167.1.patch
        10 kB
        Henry Robinson
      2. HDFS-3167.patch
        10 kB
        Henry Robinson

        Issue Links

          Activity

          Hide
          Henry Robinson added a comment -

          Patch for trunk. Example invocation instructions in the class Javadoc.

          Show
          Henry Robinson added a comment - Patch for trunk. Example invocation instructions in the class Javadoc.
          Hide
          Aaron T. Myers added a comment -

          Marking patch available for Henry so that test-patch runs.

          Show
          Aaron T. Myers added a comment - Marking patch available for Henry so that test-patch runs.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520674/HDFS-3167.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520674/HDFS-3167.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2131//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Hey Henry, patch looks pretty good after a quick look. I haven't actually run it, though. Can you comment on what testing you've done?

          A few little things I noticed:

          1. Why do you initialize nameNodePort to 20500? I don't think it will actually get used, since you later specify "0" as the default.
          2. It's not obvious to me why you output some error messages using LOG.info(...), and others using System.err.println(...). Unless there's some good reason, I'd suggest you either be consistent or add a comment explaining what the distinction for using one vs. the other is.
          3. I don't see how stop(...) will ever be called.
          Show
          Aaron T. Myers added a comment - Hey Henry, patch looks pretty good after a quick look. I haven't actually run it, though. Can you comment on what testing you've done? A few little things I noticed: Why do you initialize nameNodePort to 20500? I don't think it will actually get used, since you later specify "0" as the default. It's not obvious to me why you output some error messages using LOG.info(...), and others using System.err.println(...). Unless there's some good reason, I'd suggest you either be consistent or add a comment explaining what the distinction for using one vs. the other is. I don't see how stop(...) will ever be called.
          Hide
          Henry Robinson added a comment -

          Removed unused 'stop', and unused default for nn port, also moved all System.*.println to LOG.

          Regarding testing: informally I've been using it internally for a couple of months, and I've worked through the command line options to verify that they behave as expected before creating this patch.

          Show
          Henry Robinson added a comment - Removed unused 'stop', and unused default for nn port, also moved all System.*.println to LOG. Regarding testing: informally I've been using it internally for a couple of months, and I've worked through the command line options to verify that they behave as expected before creating this patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520769/HDFS-3167.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520769/HDFS-3167.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2138//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. I'm going to commit this momentarily.
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Hank.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Hank.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2037 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2037/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2037 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2037/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1962 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1962/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1962 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1962/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1975 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1975/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = ABORTED
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1975 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1975/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1003 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1003/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1003 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1003/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1038 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1038/)
          HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1038 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1038/ ) HDFS-3167 . CLI-based driver for MiniDFSCluster. Contributed by Henry Robinson. (Revision 1308159) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1308159 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/HdfsTestDriver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java

            People

            • Assignee:
              Henry Robinson
              Reporter:
              Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development