Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17918

Default Warehouse location apparently in HDFS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Mac OS X 10.11.6

      Description

      It seems that the default warehouse location in Spark 2.0.1 not only points at an inexistent folder in Macintosh systems (/user/hive/warehouse) - see first INFO - but also such folder is then appended to an HDFS - see the error.

      This was fixed in 2.0.0, as previous issues have reported, but appears again in 2.0.1. Indeed some scripts I was able to run in 2.0.0 now throw such errors: Spark 2.0.0 used to create the spark-warehouse folder within the current directory (which was good) and didn't complain about such weird paths, even because I'm not using Spark though HDFS, but just locally.

      16/10/13 20:47:36 INFO internal.SharedState: Warehouse path is '/user/hive/warehouse'.

      py4j.protocol.Py4JJavaError: An error occurred while calling o32.load.
      : org.apache.spark.SparkException: Unable to create database default as failed to create its directory hdfs://localhost:9000/user/hive/warehouse

      Update #1:
      I was able to reinstall Spark 2.0.0 and the first INFO message clearly states that
      16/10/13 21:06:59 INFO internal.SharedState: Warehouse path is 'file:/<local FS folder>/spark-warehouse'.

      Update #2:
      In both Spark 2.0.0 and 2.0.1 I didn't edit any config file and the like. Everything's default.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Purple Alessio
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: