Bigtop
  1. Bigtop
  2. BIGTOP-1030

Develop integration tests for new Spark component

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: tests
    • Labels:
      None

      Description

      We need to have integration tests to support BIGTOP-715

      1. BIGTOP-1030.patch
        16 kB
        Henry Wang
      2. BIGTOP-1030.patch
        17 kB
        Henry Wang
      3. BIGTOP-1030.patch
        17 kB
        Henry Wang
      4. BIGTOP-1030.patch
        17 kB
        Henry Wang
      5. BIGTOP-1030.patch
        17 kB
        Henry Wang
      6. BIGTOP-1030.patch
        20 kB
        Henry Wang
      7. BIGTOP-1030.patch
        17 kB
        Henry Wang
      8. BIGTOP-1030.patch
        17 kB
        Henry Wang
      9. BIGTOP-1030-1.patch
        16 kB
        Henry Wang
      10. SparkSmoke.patch
        13 kB
        Henry Wang

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Thanks Henry - the patch looks good.
          Once I am done with BIGTOP-715 - very close - I will commit this one.

          +1

          Show
          Konstantin Boudnik added a comment - Thanks Henry - the patch looks good. Once I am done with BIGTOP-715 - very close - I will commit this one. +1
          Hide
          Henry Wang added a comment -

          Verified patch works on trunk.

          Show
          Henry Wang added a comment - Verified patch works on trunk.
          Hide
          Konstantin Boudnik added a comment -

          Henry,

          could you please pack kmeans_data.txt file as a test resource? The reason is that spark package doesn't include any example data, hence your test won't work against spark installed from a package.

          Also, running a job against real spark cluster (e.g. master + worker) that is instantiated from the spark package would be a good addition to this test (as opposed to just local).

          Show
          Konstantin Boudnik added a comment - Henry, could you please pack kmeans_data.txt file as a test resource? The reason is that spark package doesn't include any example data, hence your test won't work against spark installed from a package. Also, running a job against real spark cluster (e.g. master + worker) that is instantiated from the spark package would be a good addition to this test (as opposed to just local ).
          Hide
          Konstantin Boudnik added a comment -

          I think the executor pom file needs to include Spark examples.jar as a dependency, otherwise the tests fail, because non of the examples are included in the Spark package.

          Please fix it. In order to test I recommend to create spark package, install it, and the run the tests.

          Show
          Konstantin Boudnik added a comment - I think the executor pom file needs to include Spark examples.jar as a dependency, otherwise the tests fail, because non of the examples are included in the Spark package. Please fix it. In order to test I recommend to create spark package, install it, and the run the tests.
          Hide
          Konstantin Boudnik added a comment -

          Also, you need to enforce HADOOP_CONF_DIR settings, otherwise test can't find Hadoop configurations.

          Show
          Konstantin Boudnik added a comment - Also, you need to enforce HADOOP_CONF_DIR settings, otherwise test can't find Hadoop configurations.
          Hide
          Konstantin Boudnik added a comment - - edited

          Henry, why are you adding current directory to the SPARK_CLASSPATH as in

          export SPARK_CLASSPATH=${EXAMPLE_JAR}:`pwd`
          

          ? What's the point of it? And what's that path?

          Show
          Konstantin Boudnik added a comment - - edited Henry, why are you adding current directory to the SPARK_CLASSPATH as in export SPARK_CLASSPATH=${EXAMPLE_JAR}:`pwd` ? What's the point of it? And what's that path?
          Hide
          Henry Wang added a comment -

          In setUp() I am unpacking the examples jar in the current directory so it can be included in the classpath.

          sh.exec("jar xf " + EXAMPLE_JAR) 
          
          Show
          Henry Wang added a comment - In setUp() I am unpacking the examples jar in the current directory so it can be included in the classpath. sh.exec( "jar xf " + EXAMPLE_JAR)
          Hide
          Konstantin Boudnik added a comment -

          And the current directory is target is something else?

          Show
          Konstantin Boudnik added a comment - And the current directory is target is something else?
          Hide
          Henry Wang added a comment -

          It is target.

          Show
          Henry Wang added a comment - It is target.
          Hide
          Konstantin Boudnik added a comment -

          right, which has all sorts of other things in it... Could you at least unpack this jar under something like target/examples so we can guarantee the content of the classpath?

          Show
          Konstantin Boudnik added a comment - right, which has all sorts of other things in it... Could you at least unpack this jar under something like target/examples so we can guarantee the content of the classpath?
          Hide
          Konstantin Boudnik added a comment -

          I have ran latest patch against the spark package built from the master and all tests are passing, considering that one has a running HDFS, of course

          +1 on the patch, will commit it shortly. Thanks Henry!

          Show
          Konstantin Boudnik added a comment - I have ran latest patch against the spark package built from the master and all tests are passing, considering that one has a running HDFS, of course +1 on the patch, will commit it shortly. Thanks Henry!
          Hide
          Konstantin Boudnik added a comment -

          Committed to master as 95bac80. Thanks!

          Show
          Konstantin Boudnik added a comment - Committed to master as 95bac80. Thanks!

            People

            • Assignee:
              Henry Wang
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development