Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1971

Support Spark SQL CLI with Apache Hive out of the box

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: spark
    • Labels:
      None

      Description

      Spark SQL supports reading and writing Apache Hive tables. To run the spark-sql CLI with Hive support, It's required ${SPARK_HOME}/conf/hive-site.xml file, so, Bigtop should package the hive-site.xml for Spark SQL (with Hive support)

        Activity

        Hide
        warwithin YoungWoo Kim added a comment -

        jay vyas, Thanks you for reviewing the patch! With this patch, Spark SQL's interactive shell would work fine.

        Show
        warwithin YoungWoo Kim added a comment - jay vyas , Thanks you for reviewing the patch! With this patch, Spark SQL's interactive shell would work fine.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jayunit100 commented on the pull request:

        https://github.com/apache/bigtop/pull/29#issuecomment-132997131

        Looks good to me if tested. +1

        Show
        githubbot ASF GitHub Bot added a comment - Github user jayunit100 commented on the pull request: https://github.com/apache/bigtop/pull/29#issuecomment-132997131 Looks good to me if tested. +1
        Hide
        warwithin YoungWoo Kim added a comment -

        Send a PR:

        • Create a embedded metastore at /tmp/spark-$ {user.name}

          /metastore_db by default

        Spark SQL CLI, /usr/lib/spark/bin/spark-sql, works on embedded metastore. Now, users can change their own Hive configurations by editing /etc/spark/conf/hive-site.xml

        Show
        warwithin YoungWoo Kim added a comment - Send a PR: Create a embedded metastore at /tmp/spark-$ {user.name} /metastore_db by default Spark SQL CLI, /usr/lib/spark/bin/spark-sql, works on embedded metastore. Now, users can change their own Hive configurations by editing /etc/spark/conf/hive-site.xml
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user youngwookim opened a pull request:

        https://github.com/apache/bigtop/pull/29

        BIGTOP-1971. Support Spark SQL CLI with Apache Hive out of the box

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/youngwookim/bigtop BIGTOP-1971

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/29.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #29


        commit f5ec457196675f2b7dd8d44df9b87b476645f962
        Author: YoungWoo Kim <ywkim@apache.org>
        Date: 2015-08-20T09:59:30Z

        BIGTOP-1971. Support Spark SQL CLI with Apache Hive out of the box


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/29 BIGTOP-1971 . Support Spark SQL CLI with Apache Hive out of the box You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1971 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit f5ec457196675f2b7dd8d44df9b87b476645f962 Author: YoungWoo Kim <ywkim@apache.org> Date: 2015-08-20T09:59:30Z BIGTOP-1971 . Support Spark SQL CLI with Apache Hive out of the box
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Without SPARK_HOME/conf/hive-site.xml, "$ ./spark-sql --master yarn" failed:

        ...
        Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        	... 76 more
        Caused by: java.sql.SQLException: Directory /usr/lib/spark/metastore_db cannot be created.
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
        	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        	... 73 more
        Caused by: ERROR XBM0H: Directory /usr/lib/spark/metastore_db cannot be created.
        	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        	at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
        	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
        	at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
        	at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
        	... 73 more
        
        

        Users can create own hive-site.xml or symlink existing one but it should work out of the box. Currently Apache Hive on Bigtop just works without additional configuration by default.

        Show
        warwithin YoungWoo Kim added a comment - - edited Without SPARK_HOME/conf/hive-site.xml, "$ ./spark-sql --master yarn" failed: ... Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 76 more Caused by: java.sql.SQLException: Directory /usr/lib/spark/metastore_db cannot be created. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) ... 73 more Caused by: ERROR XBM0H: Directory /usr/lib/spark/metastore_db cannot be created. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source) ... 73 more Users can create own hive-site.xml or symlink existing one but it should work out of the box. Currently Apache Hive on Bigtop just works without additional configuration by default.

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            warwithin YoungWoo Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development