Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Component/s: tests
    • Labels:
      None

      Description

      We need to provide tests that at least run crunch examples.

      1. BIGTOP-793.text
        2.97 MB
        Johnny Zhang
      2. BIGTOP-793-2.patch.txt
        2.98 MB
        Johnny Zhang
      3. BIGTOP-793-3.patch.txt
        2.98 MB
        Johnny Zhang

        Activity

        Roman Shaposhnik created issue -
        Johnny Zhang made changes -
        Field Original Value New Value
        Assignee Johnny Zhang [ dreambird ]
        Hide
        Johnny Zhang added a comment -

        the patch include 4 crunch test cases. They are all crunch example jobs: WordCount, SecondarySortExample, TotalBytesByIP, AverageBytesByIP. I didn't include WordAggregationHBase since it requires a HBase instance, which is not necessary if we just want to do sanity test for Crunch itself.

        I have run 4 tests on a centos 5.6 VM.

        Show
        Johnny Zhang added a comment - the patch include 4 crunch test cases. They are all crunch example jobs: WordCount, SecondarySortExample, TotalBytesByIP, AverageBytesByIP. I didn't include WordAggregationHBase since it requires a HBase instance, which is not necessary if we just want to do sanity test for Crunch itself. I have run 4 tests on a centos 5.6 VM.
        Johnny Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Johnny Zhang made changes -
        Attachment BIGTOP-793.text [ 12555276 ]
        Hide
        Roman Shaposhnik added a comment -

        Johnny, thanks a million for the patch! A couple of notes:

        1. could you please attach your patches as git format-patch – this makes it way easier for me to apply them
        2. it seems that you have forgotten to include the bigtop-tests/test-execution/smokes/crunch/pom.xml
        3. I suggest getting rid of crunchHome/crunchJar and have just a crunchJar with the default being /usr/share/doc/crunch*/crunch-examples-*-job.jar

        Otherwise – looks quite nice!

        Show
        Roman Shaposhnik added a comment - Johnny, thanks a million for the patch! A couple of notes: could you please attach your patches as git format-patch – this makes it way easier for me to apply them it seems that you have forgotten to include the bigtop-tests/test-execution/smokes/crunch/pom.xml I suggest getting rid of crunchHome/crunchJar and have just a crunchJar with the default being /usr/share/doc/crunch*/crunch-examples-*-job.jar Otherwise – looks quite nice!
        Hide
        Johnny Zhang added a comment -

        Roman Shaposhnik Thanks a lot for the comments.
        BIGTOP-793-2.patch.txt is the git format-patch. I make below change also

        (1) add bigtop-tests/test-execution/smokes/crunch/pom.xml
        (2) crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar";
        I build crunch rpm locally and install it, I verify the example jar is under /usr/lib/crunch, not /usr/share/doc/crunch

        # ls /usr/lib/crunch/crunch-examples-*job.jar
        /usr/lib/crunch/crunch-examples-job.jar
        

        please let me know. Thanks.

        Show
        Johnny Zhang added a comment - Roman Shaposhnik Thanks a lot for the comments. BIGTOP-793 -2.patch.txt is the git format-patch. I make below change also (1) add bigtop-tests/test-execution/smokes/crunch/pom.xml (2) crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar"; I build crunch rpm locally and install it, I verify the example jar is under /usr/lib/crunch, not /usr/share/doc/crunch # ls /usr/lib/crunch/crunch-examples-*job.jar /usr/lib/crunch/crunch-examples-job.jar please let me know. Thanks.
        Johnny Zhang made changes -
        Attachment BIGTOP-793-2.patch.txt [ 12555394 ]
        Hide
        Roman Shaposhnik added a comment -

        Johnny,

        a couple of points still.

        1. I think it would be useful to have the location of the jar as a property (sorry if I wasn't clear before). Also, as you can see in my updated patch on BIGTOP-612 the correct location for examples is under /usr/share/doc/crunch*/crunch-examples-*job.jar. So something like the following would probably do us all good:
          -  static String crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar";
          +  static String crunchJar = System.getProperty("org.apache.bigtop.itest.crunch.smoke.crunch.jar", "/usr/share/doc/crunch*/crunch-examples-*job.jar");
          
        2. Somehow it seems that you don't quite manage to unpack your resources into the HDFS and the tests are failing for me on a real cluster. As a matter of fact, since unpacking of resources under a particular location (be it on HDFS or a host filesystem) is such a common operation I filed BIGTOP-799 – feel free to pick it up as part of fixing this one.
        Show
        Roman Shaposhnik added a comment - Johnny, a couple of points still. I think it would be useful to have the location of the jar as a property (sorry if I wasn't clear before). Also, as you can see in my updated patch on BIGTOP-612 the correct location for examples is under /usr/share/doc/crunch*/crunch-examples-*job.jar. So something like the following would probably do us all good: - static String crunchJar = "/usr/lib/crunch/crunch-examples-*job.jar"; + static String crunchJar = System.getProperty("org.apache.bigtop.itest.crunch.smoke.crunch.jar", "/usr/share/doc/crunch*/crunch-examples-*job.jar"); Somehow it seems that you don't quite manage to unpack your resources into the HDFS and the tests are failing for me on a real cluster. As a matter of fact, since unpacking of resources under a particular location (be it on HDFS or a host filesystem) is such a common operation I filed BIGTOP-799 – feel free to pick it up as part of fixing this one.
        Hide
        Johnny Zhang added a comment -

        this is the updated 'git format-patch' with two additional change

        (1) update definition of crunchJar as suggested

        (2) using TestUtils.unpackTestResources to unpack the test resources into HDFS.

        Show
        Johnny Zhang added a comment - this is the updated 'git format-patch' with two additional change (1) update definition of crunchJar as suggested (2) using TestUtils.unpackTestResources to unpack the test resources into HDFS.
        Johnny Zhang made changes -
        Attachment BIGTOP-793-3.patch.txt [ 12555993 ]
        Hide
        Roman Shaposhnik added a comment -

        +1 and committed

        Show
        Roman Shaposhnik added a comment - +1 and committed
        Roman Shaposhnik made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Roman Shaposhnik made changes -
        Parent BIGTOP-612 [ 12559571 ]
        Issue Type Sub-task [ 7 ] Bug [ 1 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        7d 2h 2m 1 Johnny Zhang 29/Nov/12 00:44
        Patch Available Patch Available Resolved Resolved
        5d 22h 14m 1 Roman Shaposhnik 04/Dec/12 22:58
        Resolved Resolved Closed Closed
        199d 56m 1 Roman Shaposhnik 22/Jun/13 00:55

          People

          • Assignee:
            Johnny Zhang
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development