[LIVY-572] Log operations fails because of dependencies on Spark

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.7.0
• Component/s: Thriftserver
• Assignee: Marco Gaido
• Reporter: Marco Gaido
• Created: 23/Mar/19 09:57
• Updated: 27/Mar/19 00:43
• Resolved: 27/Mar/19 00:43

Description

When livy.server.thrift.logging.operation.enabled is true, which is by default, all queries run through beeline fail due to:

19/03/23 10:56:35 INFO ThriftBinaryCLIService: Closing the session: SessionHandle [eaf96260-ef2b-438e-9d12-3812f1095201]
Exception in thread "LivyThriftserver-Handler-Pool: Thread-30" java.lang.NoClassDefFoundError: org/apache/spark/sql/Row
	at org.apache.livy.thriftserver.session.ColumnBuffer.toHiveString(ColumnBuffer.java:296)
	at org.apache.livy.thriftserver.session.ColumnBuffer.add(ColumnBuffer.java:183)
	at org.apache.livy.thriftserver.serde.ColumnOrientedResultSet.addRow(ThriftResultSet.scala:108)
	at org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:125)
	at org.apache.livy.thriftserver.LivyOperationManager$$anonfun$getOperationLogRowSet$2.apply(LivyOperationManager.scala:124)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at org.apache.livy.thriftserver.LivyOperationManager.getOperationLogRowSet(LivyOperationManager.scala:124)
	at org.apache.livy.thriftserver.LivyOperationManager.fetchResults(LivyOperationManager.scala:229)
	at org.apache.livy.thriftserver.LivyCLIService.fetchResults(LivyCLIService.scala:353)
	at org.apache.livy.thriftserver.cli.ThriftCLIService.FetchResults(ThriftCLIService.scala:609)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Row
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 21 more

Issue Links

• links to: GitHub Pull Request #162