Accumulo
  1. Accumulo
  2. ACCUMULO-730

Run MR tests using local job tracker

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: test
    • Labels:
      None

      Description

      This can be done by setting the following properties, then calling ToolRunner.run.
      conf.set("mapred.job.tracker", "local");
      conf.set("fs.default.name", "file:///");

      If it doesn't take too much longer to run this way in a junit test, it would be preferable to the manual context creation that is currently being done.

        Activity

        Hide
        Billie Rinaldi added a comment -

        Question: do we need to maintain compatibility with hadoop 0.20.205.0? I'm about to check in a patch for ACCUMULO-730. I decided to use the plain local job tracker with MockAccumulo (instead of MiniAccumuloCluster) since it's a bit more straightforward and we haven't planned to deprecate MockAccumulo yet. Either way involves adding a couple of dependencies to the poms to get it to work for all of 1.0, 1.1, 0.23, and 2.0. The only one added to the top-level pom is commons-httpclient, which seems fine. To get it to work for 0.20.205.0 as well, we would have to add a dependency on jackson. I'm reluctant to do so, but if someone needs 0.20.205.0, I will.

        Show
        Billie Rinaldi added a comment - Question: do we need to maintain compatibility with hadoop 0.20.205.0? I'm about to check in a patch for ACCUMULO-730 . I decided to use the plain local job tracker with MockAccumulo (instead of MiniAccumuloCluster) since it's a bit more straightforward and we haven't planned to deprecate MockAccumulo yet. Either way involves adding a couple of dependencies to the poms to get it to work for all of 1.0, 1.1, 0.23, and 2.0. The only one added to the top-level pom is commons-httpclient, which seems fine. To get it to work for 0.20.205.0 as well, we would have to add a dependency on jackson. I'm reluctant to do so, but if someone needs 0.20.205.0, I will.
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk #639 (See https://builds.apache.org/job/Accumulo-Trunk/639/)
        ACCUMULO-730 converted input and output format tests to use local mr. no more contextfactorysvn status (Revision 1433695)

        Result = SUCCESS
        billie :
        Files :

        • /accumulo/trunk/core/pom.xml
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/ContextFactory.java
        • /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormatTest.java
        • /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloInputFormatTest.java
        • /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormatTest.java
        • /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloRowInputFormatTest.java
        • /accumulo/trunk/examples/simple/pom.xml
        • /accumulo/trunk/examples/simple/src/main/java/org/apache/accumulo/examples/simple/helloworld/InsertWithOutputFormat.java
        • /accumulo/trunk/examples/simple/src/test/java/org/apache/accumulo/examples/simple/filedata/ChunkInputFormatTest.java
        • /accumulo/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk #639 (See https://builds.apache.org/job/Accumulo-Trunk/639/ ) ACCUMULO-730 converted input and output format tests to use local mr. no more contextfactorysvn status (Revision 1433695) Result = SUCCESS billie : Files : /accumulo/trunk/core/pom.xml /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/ContextFactory.java /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormatTest.java /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloInputFormatTest.java /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormatTest.java /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/client/mapreduce/AccumuloRowInputFormatTest.java /accumulo/trunk/examples/simple/pom.xml /accumulo/trunk/examples/simple/src/main/java/org/apache/accumulo/examples/simple/helloworld/InsertWithOutputFormat.java /accumulo/trunk/examples/simple/src/test/java/org/apache/accumulo/examples/simple/filedata/ChunkInputFormatTest.java /accumulo/trunk/pom.xml
        Hide
        Christopher Tubbs added a comment -

        Are the added dependencies only needed for tests? If so, they should probably just be in the test scope, and not the default scope. If they are needed beyond unit tests, then disregard.

        Show
        Christopher Tubbs added a comment - Are the added dependencies only needed for tests? If so, they should probably just be in the test scope, and not the default scope. If they are needed beyond unit tests, then disregard.
        Hide
        Billie Rinaldi added a comment -

        Possibly? They might also be dependencies when a user runs a MR job. I don't have a strong understanding of when you would want to use the provided scope vs. the test scope, however.

        Show
        Billie Rinaldi added a comment - Possibly? They might also be dependencies when a user runs a MR job. I don't have a strong understanding of when you would want to use the provided scope vs. the test scope, however.
        Hide
        Christopher Tubbs added a comment -

        compile/provided mean that the item is needed for your artifact to function properly. The test scope means it is required for your tests to function properly. Typically, you add a dependency when you use a class it contains directly in your code. In the case of the test scope, you'd be using a class it contains directly in your testing code. I don't really know how the artifacts (commons-httpclient and avro) are used, so I can't say which they should be... but since you were mostly editing tests, I assumed it was needed for your test code. If it's not scoped to test, I would expect us to need to roll it into our assembly, though.

        Show
        Christopher Tubbs added a comment - compile/provided mean that the item is needed for your artifact to function properly. The test scope means it is required for your tests to function properly. Typically, you add a dependency when you use a class it contains directly in your code. In the case of the test scope, you'd be using a class it contains directly in your testing code. I don't really know how the artifacts (commons-httpclient and avro) are used, so I can't say which they should be... but since you were mostly editing tests, I assumed it was needed for your test code. If it's not scoped to test, I would expect us to need to roll it into our assembly, though.
        Hide
        Billie Rinaldi added a comment -

        The tests don't use the new dependencies directly, but they fail without them. That's why I thought they should be provided by hadoop.

        Show
        Billie Rinaldi added a comment - The tests don't use the new dependencies directly, but they fail without them. That's why I thought they should be provided by hadoop.
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk #658 (See https://builds.apache.org/job/Accumulo-Trunk/658/)
        ACCUMULO-730 removed documentation / functional test references to InsertWithOutputFormat (Revision 1437528)

        Result = SUCCESS
        billie :
        Files :

        • /accumulo/trunk/docs/examples/README.helloworld
        • /accumulo/trunk/test/system/auto/simple/examples.py
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk #658 (See https://builds.apache.org/job/Accumulo-Trunk/658/ ) ACCUMULO-730 removed documentation / functional test references to InsertWithOutputFormat (Revision 1437528) Result = SUCCESS billie : Files : /accumulo/trunk/docs/examples/README.helloworld /accumulo/trunk/test/system/auto/simple/examples.py
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk-Hadoop-2.0 #16 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/16/)
        ACCUMULO-730 removed documentation / functional test references to InsertWithOutputFormat (Revision 1437528)

        Result = SUCCESS
        billie :
        Files :

        • /accumulo/trunk/docs/examples/README.helloworld
        • /accumulo/trunk/test/system/auto/simple/examples.py
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #16 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/16/ ) ACCUMULO-730 removed documentation / functional test references to InsertWithOutputFormat (Revision 1437528) Result = SUCCESS billie : Files : /accumulo/trunk/docs/examples/README.helloworld /accumulo/trunk/test/system/auto/simple/examples.py

          People

          • Assignee:
            Billie Rinaldi
            Reporter:
            Billie Rinaldi
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development