[HUDI-99] [Docker demo] NoClassDefFoundError: scala/Function1 when querying rt table after upserts

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Usability
• Labels: pull-request-available
• Assignee: Balaji Varadarajan
• Reporter: Bhavani Sudha Saktheeswaran
• Created: 12/Apr/19 06:18
• Updated: 28/Aug/19 17:20
• Resolved: 01/Aug/19 04:55

Description

 

Error showing up in Beeline (Not very useful though)

Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380) 

 

Error seen in HiveServer logs:

java.lang.Exception: java.lang.NoClassDefFoundError: scala/Function1
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489) ~[hadoop-mapreduce-client-common-2.8.4.jar:?]
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549) ~[hadoop-mapreduce-client-common-2.8.4.jar:?]
Caused by: java.lang.NoClassDefFoundError: scala/Function1
at com.uber.hoodie.com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:57) ~[hoodie-hadoop-mr-bundle.jar:?]
at com.uber.hoodie.common.util.SerializationUtils$KryoSerializerInstance.<init>(SerializationUtils.java:85) ~[hoodie-hadoop-mr-bundle.jar:?]
at com.uber.hoodie.common.util.SerializationUtils.lambda$static$0(SerializationUtils.java:37) ~[hoodie-hadoop-mr-bundle.jar:?]
at java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:284) ~[?:1.8.0_181]
at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180) ~[?:1.8.0_181]
at java.lang.ThreadLocal.get(ThreadLocal.java:170) ~[?:1.8.0_181]
at com.uber.hoodie.common.util.SerializationUtils.serialize(SerializationUtils.java:50) ~[hoodie-hadoop-mr-bundle.jar:?]
at com.uber.hoodie.common.util.collection.DiskBasedMap.put(DiskBasedMap.java:169) ~[hoodie-hadoop-mr-bundle.jar:?]
at com.uber.hoodie.common.util.collection.ExternalSpillableMap.put(ExternalSpillableMap.java:169) ~[hoodie-hadoop-mr-bundle.jar:?]
at com.uber.hoodie.common.table.log.HoodieMergedLogRecordScanner.processNextRecord(HoodieMergedLogRecordScanner.java:115) ~[hoodie-hadoop-mr-bundle.jar:?]

Steps to reproduce:

• build local docker image following the instructions
• walk through the demo until ingesting second batch of kafka data.
• Now querying _rt table like below fails with the above error.
  select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'; 

Issue Links

• links to: GitHub Pull Request #641