Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Component/s: tests
    • Labels:
      None

      Description

      We need to provide tests that at least run crunch examples.

      1. BIGTOP-793.text
        2.97 MB
        Johnny Zhang
      2. BIGTOP-793-2.patch.txt
        2.98 MB
        Johnny Zhang
      3. BIGTOP-793-3.patch.txt
        2.98 MB
        Johnny Zhang

        Activity

        Hide
        Roman Shaposhnik added a comment -

        +1 and committed

        Show
        Roman Shaposhnik added a comment - +1 and committed
        Hide
        Johnny Zhang added a comment -

        this is the updated 'git format-patch' with two additional change

        (1) update definition of crunchJar as suggested

        (2) using TestUtils.unpackTestResources to unpack the test resources into HDFS.

        Show
        Johnny Zhang added a comment - this is the updated 'git format-patch' with two additional change (1) update definition of crunchJar as suggested (2) using TestUtils.unpackTestResources to unpack the test resources into HDFS.
        Hide
        Roman Shaposhnik added a comment -

        Johnny,

        a couple of points still.

        1. I think it would be useful to have the location of the jar as a property (sorry if I wasn't clear before). Also, as you can see in my updated patch on BIGTOP-612 the correct location for examples is under /usr/share/doc/crunch*/crunch-examples-*job.jar. So something like the following would probably do us all good:
          -  static String crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar";
          +  static String crunchJar = System.getProperty("org.apache.bigtop.itest.crunch.smoke.crunch.jar", "/usr/share/doc/crunch*/crunch-examples-*job.jar");
          
        2. Somehow it seems that you don't quite manage to unpack your resources into the HDFS and the tests are failing for me on a real cluster. As a matter of fact, since unpacking of resources under a particular location (be it on HDFS or a host filesystem) is such a common operation I filed BIGTOP-799 – feel free to pick it up as part of fixing this one.
        Show
        Roman Shaposhnik added a comment - Johnny, a couple of points still. I think it would be useful to have the location of the jar as a property (sorry if I wasn't clear before). Also, as you can see in my updated patch on BIGTOP-612 the correct location for examples is under /usr/share/doc/crunch*/crunch-examples-*job.jar. So something like the following would probably do us all good: - static String crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar"; + static String crunchJar = System.getProperty("org.apache.bigtop.itest.crunch.smoke.crunch.jar", "/usr/share/doc/crunch*/crunch-examples-*job.jar"); Somehow it seems that you don't quite manage to unpack your resources into the HDFS and the tests are failing for me on a real cluster. As a matter of fact, since unpacking of resources under a particular location (be it on HDFS or a host filesystem) is such a common operation I filed BIGTOP-799 – feel free to pick it up as part of fixing this one.
        Hide
        Johnny Zhang added a comment -

        Roman Shaposhnik Thanks a lot for the comments.
        BIGTOP-793-2.patch.txt is the git format-patch. I make below change also

        (1) add bigtop-tests/test-execution/smokes/crunch/pom.xml
        (2) crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar";
        I build crunch rpm locally and install it, I verify the example jar is under /usr/lib/crunch, not /usr/share/doc/crunch

        # ls /usr/lib/crunch/crunch-examples-*job.jar
        /usr/lib/crunch/crunch-examples-job.jar
        

        please let me know. Thanks.

        Show
        Johnny Zhang added a comment - Roman Shaposhnik Thanks a lot for the comments. BIGTOP-793 -2.patch.txt is the git format-patch. I make below change also (1) add bigtop-tests/test-execution/smokes/crunch/pom.xml (2) crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar"; I build crunch rpm locally and install it, I verify the example jar is under /usr/lib/crunch, not /usr/share/doc/crunch # ls /usr/lib/crunch/crunch-examples-*job.jar /usr/lib/crunch/crunch-examples-job.jar please let me know. Thanks.
        Hide
        Roman Shaposhnik added a comment -

        Johnny, thanks a million for the patch! A couple of notes:

        1. could you please attach your patches as git format-patch – this makes it way easier for me to apply them
        2. it seems that you have forgotten to include the bigtop-tests/test-execution/smokes/crunch/pom.xml
        3. I suggest getting rid of crunchHome/crunchJar and have just a crunchJar with the default being /usr/share/doc/crunch*/crunch-examples-*-job.jar

        Otherwise – looks quite nice!

        Show
        Roman Shaposhnik added a comment - Johnny, thanks a million for the patch! A couple of notes: could you please attach your patches as git format-patch – this makes it way easier for me to apply them it seems that you have forgotten to include the bigtop-tests/test-execution/smokes/crunch/pom.xml I suggest getting rid of crunchHome/crunchJar and have just a crunchJar with the default being /usr/share/doc/crunch*/crunch-examples-*-job.jar Otherwise – looks quite nice!
        Hide
        Johnny Zhang added a comment -

        the patch include 4 crunch test cases. They are all crunch example jobs: WordCount, SecondarySortExample, TotalBytesByIP, AverageBytesByIP. I didn't include WordAggregationHBase since it requires a HBase instance, which is not necessary if we just want to do sanity test for Crunch itself.

        I have run 4 tests on a centos 5.6 VM.

        Show
        Johnny Zhang added a comment - the patch include 4 crunch test cases. They are all crunch example jobs: WordCount, SecondarySortExample, TotalBytesByIP, AverageBytesByIP. I didn't include WordAggregationHBase since it requires a HBase instance, which is not necessary if we just want to do sanity test for Crunch itself. I have run 4 tests on a centos 5.6 VM.

          People

          • Assignee:
            Johnny Zhang
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development