Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1971

Support Spark SQL CLI with Apache Hive out of the box

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: spark
    • Labels:
      None

      Description

      Spark SQL supports reading and writing Apache Hive tables. To run the spark-sql CLI with Hive support, It's required ${SPARK_HOME}/conf/hive-site.xml file, so, Bigtop should package the hive-site.xml for Spark SQL (with Hive support)

        Activity

        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Without SPARK_HOME/conf/hive-site.xml, "$ ./spark-sql --master yarn" failed:

        ...
        Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        	... 76 more
        Caused by: java.sql.SQLException: Directory /usr/lib/spark/metastore_db cannot be created.
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        	... 73 more
        Caused by: ERROR XBM0H: Directory /usr/lib/spark/metastore_db cannot be created.
        	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        	at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
        	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
        	at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
        	at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
        	... 73 more
        
        

        Users can create own hive-site.xml or symlink existing one but it should work out of the box. Currently Apache Hive on Bigtop just works without additional configuration by default.

        Show
        warwithin YoungWoo Kim added a comment - - edited Without SPARK_HOME/conf/hive-site.xml, "$ ./spark-sql --master yarn" failed: ... Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 76 more Caused by: java.sql.SQLException: Directory /usr/lib/spark/metastore_db cannot be created. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) ... 73 more Caused by: ERROR XBM0H: Directory /usr/lib/spark/metastore_db cannot be created. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source) ... 73 more Users can create own hive-site.xml or symlink existing one but it should work out of the box. Currently Apache Hive on Bigtop just works without additional configuration by default.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user youngwookim opened a pull request:

        https://github.com/apache/bigtop/pull/29

        BIGTOP-1971. Support Spark SQL CLI with Apache Hive out of the box

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/youngwookim/bigtop BIGTOP-1971

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/29.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #29


        commit f5ec457196675f2b7dd8d44df9b87b476645f962
        Author: YoungWoo Kim <ywkim@apache.org>
        Date: 2015-08-20T09:59:30Z

        BIGTOP-1971. Support Spark SQL CLI with Apache Hive out of the box


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/29 BIGTOP-1971 . Support Spark SQL CLI with Apache Hive out of the box You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1971 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit f5ec457196675f2b7dd8d44df9b87b476645f962 Author: YoungWoo Kim <ywkim@apache.org> Date: 2015-08-20T09:59:30Z BIGTOP-1971 . Support Spark SQL CLI with Apache Hive out of the box
        Hide
        warwithin YoungWoo Kim added a comment -

        Send a PR:

        • Create a embedded metastore at /tmp/spark-$ {user.name}

          /metastore_db by default

        Spark SQL CLI, /usr/lib/spark/bin/spark-sql, works on embedded metastore. Now, users can change their own Hive configurations by editing /etc/spark/conf/hive-site.xml

        Show
        warwithin YoungWoo Kim added a comment - Send a PR: Create a embedded metastore at /tmp/spark-$ {user.name} /metastore_db by default Spark SQL CLI, /usr/lib/spark/bin/spark-sql, works on embedded metastore. Now, users can change their own Hive configurations by editing /etc/spark/conf/hive-site.xml
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jayunit100 commented on the pull request:

        https://github.com/apache/bigtop/pull/29#issuecomment-132997131

        Looks good to me if tested. +1

        Show
        githubbot ASF GitHub Bot added a comment - Github user jayunit100 commented on the pull request: https://github.com/apache/bigtop/pull/29#issuecomment-132997131 Looks good to me if tested. +1
        Hide
        warwithin YoungWoo Kim added a comment -

        jay vyas, Thanks you for reviewing the patch! With this patch, Spark SQL's interactive shell would work fine.

        Show
        warwithin YoungWoo Kim added a comment - jay vyas , Thanks you for reviewing the patch! With this patch, Spark SQL's interactive shell would work fine.

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            warwithin YoungWoo Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development