Uploaded image for project: 'Spatial Information Systems'
  1. Spatial Information Systems
  2. SIS-491

sis-embedded-data database fails to start on Apache Spark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Cannot Reproduce
    • 0.8, 1.0
    • 1.1
    • Metadata
    • None
    • Spark version 2.4.4
      Hadoop version 2.7.6
      Scala version 2.11.12
      Java version 1.8.0_242
      OS: Linux <computer> 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

    Description

      Operation CRS.forCode fails when used within the Apache Spark environment. 

      Steps to reproduce (see full log below):

          1) Start spark-shell with the --packages argument specifying the sis-embedded-data artifact

          2) Execute CRS.forCode ("EPSG:21781") // Swiss coordinate system https://epsg.io/21781

      Result:

          NoSuchAuthorityCodeException (with a prior warning message: Failed to start database 'classpath:SIS_DATA/Databases/SpatialMetadata')

      Expected result:

          Returns a valid CoordinateReferenceSystem for "EPSG:21781".

      Workaround:

          Use CRS.fromWKT instead.

      Note:

      It works fine in a stand-alone Scala program with build.sbt:

      name := "ApacheSISTest"

      version := "0.1"scalaVersion := "2.11.12"

      libraryDependencies = "org.apache.sis.core" % "sis-referencing" % "1.0"

      libraryDependencies = "org.apache.sis.non-free" % "sis-embedded-data" % "1.0"

      and code

      import org.apache.sis.geometry.DirectPosition2D
      import org.apache.sis.referencing.CRS

      object Test extends App {

           val position = new DirectPosition2D (678544.6095273666, 248462.881516265)
           println ("Source: " + position)
           val sourceCRS = CRS.forCode ("EPSG:21781") // Swiss coordinate system https://epsg.io/21781

          val targetCRS = CRS.forCode ("EPSG:4326") // WGS 84 https://epsg.io/4326

          val operation = CRS.findOperation (sourceCRS, targetCRS, null)

          val transform = operation.getMathTransform
          val latlong = transform.transform (position, null) // °N, °E

          println ("Target: " + latlong)
      }

       

      Full Spark log:

      $ spark-shell --master local[*] --verbose --packages org.apache.sis.core:sis-referencing:1.0,org.apache.sis.non-free:sis-embedded-data:1.0
      Using properties file: null
      20/03/08 15:41:42 WARN Utils: Your hostname, <computer> resolves to a loopback address: 127.0.1.1; using 192.168.10.221 instead (on interface wlp4s0)
      20/03/08 15:41:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
      Parsed arguments:
      master local[*]
      deployMode null
      executorMemory null
      executorCores null
      totalExecutorCores null
      propertiesFile null
      driverMemory null
      driverCores null
      driverExtraClassPath null
      driverExtraLibraryPath null
      driverExtraJavaOptions null
      supervise false
      queue null
      numExecutors null
      files null
      pyFiles null
      archives null
      mainClass org.apache.spark.repl.Main
      primaryResource spark-shell
      name Spark shell
      childArgs []
      jars null
      packages org.apache.sis.core:sis-referencing:1.0,org.apache.sis.non-free:sis-embedded-data:1.0
      packagesExclusions null
      repositories null
      verbose true

      Spark properties used, including those specified through
      --conf and those from the properties file null:

      Ivy Default Cache set to: /home/<user>/.ivy2/cache
      The jars for the packages stored in: /home/<user>/.ivy2/jars
      :: loading settings :: url = jar:file:/home/<user>/spark/spark-2.4.4-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
      org.apache.sis.core#sis-referencing added as a dependency
      org.apache.sis.non-free#sis-embedded-data added as a dependency
      :: resolving dependencies :: org.apache.spark#spark-submit-parent-83046344-c80a-4a8a-9d41-e6145875a27a;1.0
      confs: [default]
      found org.apache.sis.core#sis-referencing;1.0 in central
      found org.apache.sis.core#sis-utility;1.0 in central
      found javax.measure#unit-api;1.0 in central
      found org.opengis#geoapi;3.0.1 in central
      found org.apache.sis.core#sis-metadata;1.0 in central
      found org.apache.sis.non-free#sis-embedded-data;1.0 in central
      found org.apache.derby#derby;10.14.2.0 in central
      downloading https://repo1.maven.org/maven2/org/apache/sis/core/sis-referencing/1.0/sis-referencing-1.0.jar ...
      [SUCCESSFUL ] org.apache.sis.core#sis-referencing;1.0!sis-referencing.jar (245ms)
      downloading https://repo1.maven.org/maven2/org/apache/sis/non-free/sis-embedded-data/1.0/sis-embedded-data-1.0.jar ...
      [SUCCESSFUL ] org.apache.sis.non-free#sis-embedded-data;1.0!sis-embedded-data.jar (489ms)
      downloading https://repo1.maven.org/maven2/org/apache/sis/core/sis-utility/1.0/sis-utility-1.0.jar ...
      [SUCCESSFUL ] org.apache.sis.core#sis-utility;1.0!sis-utility.jar (92ms)
      downloading https://repo1.maven.org/maven2/org/apache/sis/core/sis-metadata/1.0/sis-metadata-1.0.jar ...
      [SUCCESSFUL ] org.apache.sis.core#sis-metadata;1.0!sis-metadata.jar (134ms)
      :: resolution report :: resolve 7312ms :: artifacts dl 970ms
      :: modules in use:
      javax.measure#unit-api;1.0 from central in [default]
      org.apache.derby#derby;10.14.2.0 from central in [default]
      org.apache.sis.core#sis-metadata;1.0 from central in [default]
      org.apache.sis.core#sis-referencing;1.0 from central in [default]
      org.apache.sis.core#sis-utility;1.0 from central in [default]
      org.apache.sis.non-free#sis-embedded-data;1.0 from central in [default]
      org.opengis#geoapi;3.0.1 from central in [default]
      ---------------------------------------------------------------------

        modules artifacts
      conf number search dwnlded evicted number dwnlded

      ---------------------------------------------------------------------

      default 7 6 6 0 7 4

      ---------------------------------------------------------------------

      :: problems summary ::
      :::: ERRORS
      unknown resolver null

      unknown resolver null

      unknown resolver null

      :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
      :: retrieving :: org.apache.spark#spark-submit-parent-83046344-c80a-4a8a-9d41-e6145875a27a
      confs: [default]
      7 artifacts copied, 0 already retrieved (10240kB/20ms)
      Main class:
      org.apache.spark.repl.Main
      Arguments:

      Spark config:
      (spark.jars,file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-referencing-1.0.jar,file:///home/<user>/.ivy2/jars/org.apache.sis.non-free_sis-embedded-data-1.0.jar,file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-utility-1.0.jar,file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-metadata-1.0.jar,file:///home/<user>/.ivy2/jars/javax.measure_unit-api-1.0.jar,file:///home/<user>/.ivy2/jars/org.opengis_geoapi-3.0.1.jar,file:///home/<user>/.ivy2/jars/org.apache.derby_derby-10.14.2.0.jar)
      (spark.app.name,Spark shell)
      (spark.ui.showConsoleProgress,true)
      (spark.submit.deployMode,client)
      (spark.master,local[*])
      (spark.repl.local.jars,file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-referencing-1.0.jar,file:///home/<user>/.ivy2/jars/org.apache.sis.non-free_sis-embedded-data-1.0.jar,file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-utility-1.0.jar,file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-metadata-1.0.jar,file:///home/<user>/.ivy2/jars/javax.measure_unit-api-1.0.jar,file:///home/<user>/.ivy2/jars/org.opengis_geoapi-3.0.1.jar,file:///home/<user>/.ivy2/jars/org.apache.derby_derby-10.14.2.0.jar)
      Classpath elements:
      file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-referencing-1.0.jar
      file:///home/<user>/.ivy2/jars/org.apache.sis.non-free_sis-embedded-data-1.0.jar
      file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-utility-1.0.jar
      file:///home/<user>/.ivy2/jars/org.apache.sis.core_sis-metadata-1.0.jar
      file:///home/<user>/.ivy2/jars/javax.measure_unit-api-1.0.jar
      file:///home/<user>/.ivy2/jars/org.opengis_geoapi-3.0.1.jar
      file:///home/<user>/.ivy2/jars/org.apache.derby_derby-10.14.2.0.jar

      20/03/08 15:41:58 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
      20/03/08 15:41:58 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
      Spark context Web UI available at http://192.168.10.221:4042
      Spark context available as 'sc' (master = local[*], app id = local-1583678518329).
      Spark session available as 'spark'.
      Welcome to
      ____ __
      / _/_ ___ ____/ /_
      \ \/ _ \/ _ `/ __/ '/
      /__/ ./,// //\ version 2.4.4
      /_/

      Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
      Type in expressions to have them evaluated.
      Type :help for more information.

      scala> import org.apache.sis.geometry.DirectPosition2D
      import org.apache.sis.geometry.DirectPosition2D

      scala> import org.apache.sis.referencing.CRS
      import org.apache.sis.referencing.CRS

      scala> val position = new DirectPosition2D (678544.6095273666, 248462.881516265)
      position: org.apache.sis.geometry.DirectPosition2D = POINT(678544.6095273666 248462.881516265)

      scala> println ("Source: " + position)
      Source: POINT(678544.6095273666 248462.881516265)

      scala>

      scala> val sourceCRS = CRS.forCode ("EPSG:21781")
      20/03/08 15:43:14 WARN factory: Can not use the EPSG geodetic parameters: Failed to start database 'classpath:SIS_DATA/Databases/SpatialMetadata' with class loader sun.misc.Launcher$AppClassLoader@f2a0b8e, see the next exception for details.
      org.opengis.referencing.NoSuchAuthorityCodeException: No ‘CoordinateReferenceSystem’ object found for code “21781”. However only a subset of the EPSG geodetic dataset has been queried. See http://sis.apache.org/epsg.html for instruction about how to install the full EPSG database.
      at org.apache.sis.referencing.EPSGFactoryFallback.predefined(EPSGFactoryFallback.java:355)
      at org.apache.sis.referencing.EPSGFactoryFallback.createCoordinateReferenceSystem(EPSGFactoryFallback.java:231)
      at org.apache.sis.referencing.CRS.forCode(CRS.java:236)
      ... 49 elided

      scala> val targetCRS = CRS.forCode ("EPSG:4326")
      20/03/08 15:43:17 WARN system: Can not connect to “jdbc/SpatialMetadata”.
      java.sql.SQLException: Failed to start database 'classpath:SIS_DATA/Databases/SpatialMetadata' with class loader sun.misc.Launcher$AppClassLoader@f2a0b8e, see the next exception for details.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
      at org.apache.derby.jdbc.BasicEmbeddedDataSource40.getConnection(Unknown Source)
      at org.apache.derby.jdbc.BasicEmbeddedDataSource40.getConnection(Unknown Source)
      at org.apache.sis.metadata.sql.MetadataSource.connection(MetadataSource.java:490)
      at org.apache.sis.metadata.sql.MetadataSource.install(MetadataSource.java:457)
      at org.apache.sis.metadata.sql.MetadataSource.getProvided(MetadataSource.java:318)
      at org.apache.sis.internal.simple.CitationConstant.delegate(CitationConstant.java:177)
      at org.apache.sis.internal.simple.CitationConstant.getIdentifiers(CitationConstant.java:204)
      at org.apache.sis.internal.metadata.Identifiers.getIdentifier(Identifiers.java:117)
      at org.apache.sis.referencing.ImmutableIdentifier.formatTo(ImmutableIdentifier.java:517)
      at org.apache.sis.io.wkt.Formatter.append(Formatter.java:691)
      at org.apache.sis.io.wkt.Formatter.appendComplement(Formatter.java:849)
      at org.apache.sis.io.wkt.Formatter.append(Formatter.java:738)
      at org.apache.sis.io.wkt.FormattableObject.formatWKT(FormattableObject.java:200)
      at org.apache.sis.io.wkt.FormattableObject.toString(FormattableObject.java:126)
      at scala.runtime.ScalaRunTime$.scala$runtime$ScalaRunTime$$inner$1(ScalaRunTime.scala:332)
      at scala.runtime.ScalaRunTime$.stringOf(ScalaRunTime.scala:337)
      at scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345)
      at $line21.$eval$.$print$lzycompute(<console>:10)
      at $line21.$eval$.$print(<console>:6)
      at $line21.$eval.$print(<console>)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
      at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
      at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
      at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:819)
      at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:691)
      at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
      at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:425)
      at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:285)
      at org.apache.spark.repl.SparkILoop.runClosure(SparkILoop.scala:159)
      at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:182)
      at org.apache.spark.repl.Main$.doMain(Main.scala:78)
      at org.apache.spark.repl.Main$.main(Main.scala:58)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
      at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
      at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
      at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
      at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: ERROR XJ040: Failed to start database 'classpath:SIS_DATA/Databases/SpatialMetadata' with class loader sun.misc.Launcher$AppClassLoader@f2a0b8e, see the next exception for details.
      at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
      ... 65 more
      Caused by: ERROR XCL20: Catalogs at version level 'null' cannot be upgraded to version level '10.12'.
      at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      at org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(Unknown Source)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDictionaryTables(Unknown Source)
      at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
      at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source)
      at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
      at org.apache.derby.impl.db.BasicDatabase$5.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.impl.db.BasicDatabase.bootServiceModule(Unknown Source)
      at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
      at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
      at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
      at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source)
      at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(Unknown Source)
      ... 62 more
      targetCRS: org.opengis.referencing.crs.CoordinateReferenceSystem =
      GeodeticCRS["WGS 84",
      Datum["World Geodetic System 1984",
      Ellipsoid["WGS 84", 6378137.0, 298.257223563]],
      CS[ellipsoidal, 2],
      Axis["Geodetic latitude (φ)", north],
      Axis["Geodetic longitude (λ)", east],
      Unit["degree", 0.017453292519943295],
      Scope["Horizontal component of 3D system."],
      Area["World"],
      BBox[-90.00, -180.00, 90.00, 180.00],
      Id["EPSG", 4326, URI["urn:ogc:def:crs:EPSG::4326"]]]

      scala>

      Attachments

        Activity

          People

            desruisseaux Martin Desruisseaux
            derrickoswald Derrick Oswald
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: