Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: spark
    • Labels:
      None

      Description

      Add a smoke test for Spark SQL using Spark's examples

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user youngwookim opened a pull request:

        https://github.com/apache/bigtop/pull/30

        BIGTOP-1975. Smoke tests for Spark SQL

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/youngwookim/bigtop BIGTOP-1975

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/30.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #30


        commit a3f88a03385ee55c78b886b63ad53496ac5748e6
        Author: YoungWoo Kim <ywkim@apache.org>
        Date: 2015-08-20T07:45:24Z

        BIGTOP-1975. Smoke tests for Spark SQL


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/30 BIGTOP-1975 . Smoke tests for Spark SQL You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1975 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit a3f88a03385ee55c78b886b63ad53496ac5748e6 Author: YoungWoo Kim <ywkim@apache.org> Date: 2015-08-20T07:45:24Z BIGTOP-1975 . Smoke tests for Spark SQL
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/30#discussion_r37715124

        — Diff: bigtop-tests/smoke-tests/spark/TestSpark.groovy —
        @@ -0,0 +1,63 @@
        +/**
        + * Licensed to the Apache Software Foundation (ASF) under one
        + * or more contributor license agreements. See the NOTICE file
        + * distributed with this work for additional information
        + * regarding copyright ownership. The ASF licenses this file
        + * to you under the Apache License, Version 2.0 (the
        + * "License"); you may not use this file except in compliance
        + * with the License. You may obtain a copy of the License at
        + * <p/>
        + * http://www.apache.org/licenses/LICENSE-2.0
        + * <p/>
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.bigtop.itest.spark
        +
        +import org.junit.BeforeClass
        +import org.junit.AfterClass
        +import static org.junit.Assert.assertNotNull
        +import org.apache.bigtop.itest.shell.Shell
        +import static org.junit.Assert.assertTrue
        +import org.junit.Test
        +import org.apache.bigtop.itest.JarContent
        +import org.apache.bigtop.itest.TestUtils
        +import org.apache.commons.logging.LogFactory
        +import org.apache.commons.logging.Log
        +
        +import static org.apache.bigtop.itest.LogErrorsUtils.logError
        +
        +class TestSpark {
        + static private Log LOG = LogFactory.getLog(Object.class)
        +
        + static Shell sh = new Shell("/bin/bash -s")
        + static final String SPARK_HOME = System.getenv("SPARK_HOME")
        + static final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master yarn-client"
        + static final String TEST_SAPRKSQL_LOG = "/tmp/TestSpark_testSparkSQL.log"
        — End diff –

        Spelling of variable name

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/30#discussion_r37715124 — Diff: bigtop-tests/smoke-tests/spark/TestSpark.groovy — @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * <p/> + * http://www.apache.org/licenses/LICENSE-2.0 + * <p/> + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.bigtop.itest.spark + +import org.junit.BeforeClass +import org.junit.AfterClass +import static org.junit.Assert.assertNotNull +import org.apache.bigtop.itest.shell.Shell +import static org.junit.Assert.assertTrue +import org.junit.Test +import org.apache.bigtop.itest.JarContent +import org.apache.bigtop.itest.TestUtils +import org.apache.commons.logging.LogFactory +import org.apache.commons.logging.Log + +import static org.apache.bigtop.itest.LogErrorsUtils.logError + +class TestSpark { + static private Log LOG = LogFactory.getLog(Object.class) + + static Shell sh = new Shell("/bin/bash -s") + static final String SPARK_HOME = System.getenv("SPARK_HOME") + static final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master yarn-client" + static final String TEST_SAPRKSQL_LOG = "/tmp/TestSpark_testSparkSQL.log" — End diff – Spelling of variable name
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/30#discussion_r37715134

        — Diff: bigtop-tests/smoke-tests/spark/TestSpark.groovy —
        @@ -0,0 +1,63 @@
        +/**
        + * Licensed to the Apache Software Foundation (ASF) under one
        + * or more contributor license agreements. See the NOTICE file
        + * distributed with this work for additional information
        + * regarding copyright ownership. The ASF licenses this file
        + * to you under the Apache License, Version 2.0 (the
        + * "License"); you may not use this file except in compliance
        + * with the License. You may obtain a copy of the License at
        + * <p/>
        + * http://www.apache.org/licenses/LICENSE-2.0
        + * <p/>
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.bigtop.itest.spark
        +
        +import org.junit.BeforeClass
        +import org.junit.AfterClass
        +import static org.junit.Assert.assertNotNull
        +import org.apache.bigtop.itest.shell.Shell
        +import static org.junit.Assert.assertTrue
        +import org.junit.Test
        +import org.apache.bigtop.itest.JarContent
        +import org.apache.bigtop.itest.TestUtils
        +import org.apache.commons.logging.LogFactory
        +import org.apache.commons.logging.Log
        +
        +import static org.apache.bigtop.itest.LogErrorsUtils.logError
        +
        +class TestSpark {
        + static private Log LOG = LogFactory.getLog(Object.class)
        +
        + static Shell sh = new Shell("/bin/bash -s")
        + static final String SPARK_HOME = System.getenv("SPARK_HOME")
        + static final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master yarn-client"
        + static final String TEST_SAPRKSQL_LOG = "/tmp/TestSpark_testSparkSQL.log"
        +
        + @BeforeClass
        + static void setUp() {
        + sh.exec("rm -f " + TEST_SAPRKSQL_LOG)
        — End diff –

        Spelling of variable name

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/30#discussion_r37715134 — Diff: bigtop-tests/smoke-tests/spark/TestSpark.groovy — @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * <p/> + * http://www.apache.org/licenses/LICENSE-2.0 + * <p/> + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.bigtop.itest.spark + +import org.junit.BeforeClass +import org.junit.AfterClass +import static org.junit.Assert.assertNotNull +import org.apache.bigtop.itest.shell.Shell +import static org.junit.Assert.assertTrue +import org.junit.Test +import org.apache.bigtop.itest.JarContent +import org.apache.bigtop.itest.TestUtils +import org.apache.commons.logging.LogFactory +import org.apache.commons.logging.Log + +import static org.apache.bigtop.itest.LogErrorsUtils.logError + +class TestSpark { + static private Log LOG = LogFactory.getLog(Object.class) + + static Shell sh = new Shell("/bin/bash -s") + static final String SPARK_HOME = System.getenv("SPARK_HOME") + static final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master yarn-client" + static final String TEST_SAPRKSQL_LOG = "/tmp/TestSpark_testSparkSQL.log" + + @BeforeClass + static void setUp() { + sh.exec("rm -f " + TEST_SAPRKSQL_LOG) — End diff – Spelling of variable name
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/30#discussion_r37715174

        — Diff: bigtop-tests/smoke-tests/spark/TestSpark.groovy —
        @@ -0,0 +1,63 @@
        +/**
        + * Licensed to the Apache Software Foundation (ASF) under one
        + * or more contributor license agreements. See the NOTICE file
        + * distributed with this work for additional information
        + * regarding copyright ownership. The ASF licenses this file
        + * to you under the Apache License, Version 2.0 (the
        + * "License"); you may not use this file except in compliance
        + * with the License. You may obtain a copy of the License at
        + * <p/>
        + * http://www.apache.org/licenses/LICENSE-2.0
        + * <p/>
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.bigtop.itest.spark
        +
        +import org.junit.BeforeClass
        +import org.junit.AfterClass
        +import static org.junit.Assert.assertNotNull
        +import org.apache.bigtop.itest.shell.Shell
        +import static org.junit.Assert.assertTrue
        +import org.junit.Test
        +import org.apache.bigtop.itest.JarContent
        +import org.apache.bigtop.itest.TestUtils
        +import org.apache.commons.logging.LogFactory
        +import org.apache.commons.logging.Log
        +
        +import static org.apache.bigtop.itest.LogErrorsUtils.logError
        +
        +class TestSpark {
        + static private Log LOG = LogFactory.getLog(Object.class)
        +
        + static Shell sh = new Shell("/bin/bash -s")
        + static final String SPARK_HOME = System.getenv("SPARK_HOME")
        + static final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master yarn-client"
        + static final String TEST_SAPRKSQL_LOG = "/tmp/TestSpark_testSparkSQL.log"
        +
        + @BeforeClass
        + static void setUp()

        { + sh.exec("rm -f " + TEST_SAPRKSQL_LOG) + sh.exec("hdfs dfs -put " + SPARK_HOME + "/examples examples") + logError(sh) + }

        +
        + @AfterClass
        + public static void tearDown()

        { + sh.exec("hdfs dfs -ls") + logError(sh) + sh.exec("hdfs dfs -rmr people* examples") + logError(sh) + }

        +
        + @Test
        + void testSparkSQL() {
        + sh.exec(SPARK_SHELL + " --class org.apache.spark.examples.sql.JavaSparkSQL " + " --jars " + SPARK_HOME + "/lib/spark-examples*.jar > " + TEST_SAPRKSQL_LOG + " 2>&1")
        — End diff –

        Spelling of variable name

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/30#discussion_r37715174 — Diff: bigtop-tests/smoke-tests/spark/TestSpark.groovy — @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * <p/> + * http://www.apache.org/licenses/LICENSE-2.0 + * <p/> + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.bigtop.itest.spark + +import org.junit.BeforeClass +import org.junit.AfterClass +import static org.junit.Assert.assertNotNull +import org.apache.bigtop.itest.shell.Shell +import static org.junit.Assert.assertTrue +import org.junit.Test +import org.apache.bigtop.itest.JarContent +import org.apache.bigtop.itest.TestUtils +import org.apache.commons.logging.LogFactory +import org.apache.commons.logging.Log + +import static org.apache.bigtop.itest.LogErrorsUtils.logError + +class TestSpark { + static private Log LOG = LogFactory.getLog(Object.class) + + static Shell sh = new Shell("/bin/bash -s") + static final String SPARK_HOME = System.getenv("SPARK_HOME") + static final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master yarn-client" + static final String TEST_SAPRKSQL_LOG = "/tmp/TestSpark_testSparkSQL.log" + + @BeforeClass + static void setUp() { + sh.exec("rm -f " + TEST_SAPRKSQL_LOG) + sh.exec("hdfs dfs -put " + SPARK_HOME + "/examples examples") + logError(sh) + } + + @AfterClass + public static void tearDown() { + sh.exec("hdfs dfs -ls") + logError(sh) + sh.exec("hdfs dfs -rmr people* examples") + logError(sh) + } + + @Test + void testSparkSQL() { + sh.exec(SPARK_SHELL + " --class org.apache.spark.examples.sql.JavaSparkSQL " + " --jars " + SPARK_HOME + "/lib/spark-examples*.jar > " + TEST_SAPRKSQL_LOG + " 2>&1") — End diff – Spelling of variable name
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on the pull request:

        https://github.com/apache/bigtop/pull/30#issuecomment-133992885

        Does the JavaSparkSQL example check the correctness of the SQL result? Or are you just going for a basic "no errors were reported" approach?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on the pull request: https://github.com/apache/bigtop/pull/30#issuecomment-133992885 Does the JavaSparkSQL example check the correctness of the SQL result? Or are you just going for a basic "no errors were reported" approach?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user youngwookim commented on the pull request:

        https://github.com/apache/bigtop/pull/30#issuecomment-133997605

        Addressed RJ's comment and updated this PR.

        @rnowling Yes, it just checks that the shell returned 0

        Show
        githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on the pull request: https://github.com/apache/bigtop/pull/30#issuecomment-133997605 Addressed RJ's comment and updated this PR. @rnowling Yes, it just checks that the shell returned 0
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on the pull request:

        https://github.com/apache/bigtop/pull/30#issuecomment-133999120

        +1. LGTM. In the future, maybe we can parse the output log for results but this is a great start.

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on the pull request: https://github.com/apache/bigtop/pull/30#issuecomment-133999120 +1. LGTM. In the future, maybe we can parse the output log for results but this is a great start.
        Hide
        warwithin YoungWoo Kim added a comment -

        Committed! Thanks RJ Nowling for reviewing the patch.

        Show
        warwithin YoungWoo Kim added a comment - Committed! Thanks RJ Nowling for reviewing the patch.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/bigtop/pull/30

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/30

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            warwithin YoungWoo Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development