Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2129

NPE thrown while lookup a view

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      scala> hql("create table t1(key int, value string)")
      ...
      scala> hql("create view v as select * from t1")
      ...
      scala> table("v")
      ...
      java.lang.NullPointerException
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:190)
              at org.apache.spark.sql.hive.MetastoreRelation.<init>(HiveMetastoreCatalog.scala:261)
              at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:66)
              at org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:149)
              at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:96)
              at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:96)
              at scala.Option.getOrElse(Option.scala:120)
              at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:96)
              at org.apache.spark.sql.hive.HiveContext$$anon$2.lookupRelation(HiveContext.scala:155)
              at org.apache.spark.sql.SQLContext.table(SQLContext.scala:161)
              at .<init>(<console>:40)
              at .<clinit>(<console>)
              at .<init>(<console>:7)
              at .<clinit>(<console>)
              at $print(<console>)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734)
              at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983)
              at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
              at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604)
              at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568)
              at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760)
              at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:805)
              at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:717)
              at scala.tools.nsc.interpreter.ILoop.processLine$1(ILoop.scala:581)
              at scala.tools.nsc.interpreter.ILoop.innerLoop$1(ILoop.scala:588)
              at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:591)
              at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:882)
              at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837)
              at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837)
              at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
              at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:837)
              at scala.tools.nsc.interpreter.ILoop.main(ILoop.scala:904)
              at xsbt.ConsoleInterface.run(ConsoleInterface.scala:69)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:102)
              at sbt.compiler.AnalyzingCompiler.console(AnalyzingCompiler.scala:77)
              at sbt.Console.sbt$Console$$console0$1(Console.scala:23)
              at sbt.Console$$anonfun$apply$2$$anonfun$apply$1.apply$mcV$sp(Console.scala:24)
              at sbt.Console$$anonfun$apply$2$$anonfun$apply$1.apply(Console.scala:24)
              at sbt.Console$$anonfun$apply$2$$anonfun$apply$1.apply(Console.scala:24)
              at sbt.Logger$$anon$4.apply(Logger.scala:90)
              at sbt.TrapExit$App.run(TrapExit.scala:244)
              at java.lang.Thread.run(Thread.java:745)
      

      This issue can also be triggered by console version of the join_view test case of the HiveCompatibilitySuite:

      == Console version of this test ==
      val q0 = hql("""drop table invites"""); q0.collect()
      val q1 = hql("""drop table invites2"""); q1.collect()
      val q2 = hql("""create table invites (foo int, bar string) partitioned by (ds string)"""); q2.collect()
      val q3 = hql("""create table invites2 (foo int, bar string) partitioned by (ds string)"""); q3.collect()
      val q4 = hql("""set hive.mapred.mode=strict"""); q4.collect()
      val q5 = hql("""-- test join views: see HIVE-1989
      
      create view v as select invites.bar, invites2.foo, invites2.ds from invites join invites2 on invites.ds=invites2.ds"""); q5.collect()
      val q6 = hql("""explain select * from v where ds='2011-09-01'"""); q6.collect()
      val q7 = hql("""drop view v"""); q7.collect()
      val q8 = hql("""drop table invites"""); q8.collect()
      val q9 = hql("""drop table invites2"""); q9.collect()
      

      When running inside HiveCompatibilitySuite, the exception is eaten up by SQLContext.stringOrError.

      When attacking SPARK-2094, we added physical plans for DDL and commands, no longer taking the stringOnError route, and the exception is thrown.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                lian cheng Cheng Lian
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: