[GOBBLIN-831] Fix NPE in KafkaWorkUnitPacker when there is no WorkUnit created

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0, 0.16.0
• Assignee: Zhixiong Chen
• Reporter: Zhixiong Chen
• Created: 18/Jul/19 19:10
• Updated: 07/Oct/20 18:43
• Resolved: 14/Aug/20 23:25

Description

INFO Created workunits for 0 topics in 0 seconds
INFO For all topics not pulled in the previous run, estimated avg time to pull a record is 1.0 milliseconds
INFO The total estimated data size is 0.00
ERROR Failed to get work units for job job_VeniceInc_1597274974566
java.lang.RuntimeException: Unexpected throwable caught, 
	at org.apache.gobblin.source.extractor.extract.kafka.KafkaSource.getWorkunits(KafkaSource.java:287)
	at com.linkedin.gobblin.venice.source.VeniceKafkaSource.getWorkunits(VeniceKafkaSource.java:49)
	at com.linkedin.gobblinopal.source.OpalKafkaSourceDecorator.getWorkunits(OpalKafkaSourceDecorator.java:68)
	at org.apache.gobblin.runtime.SourceDecorator.getWorkunitStream(SourceDecorator.java:81)
	at org.apache.gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:410)
	at org.apache.gobblin.azkaban.AzkabanJobLauncher.launchJob(AzkabanJobLauncher.java:321)
	at org.apache.gobblin.azkaban.AzkabanJobLauncher.runRealJob(AzkabanJobLauncher.java:286)
	at org.apache.gobblin.azkaban.AzkabanJobLauncher.run(AzkabanJobLauncher.java:277)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at azkaban.jobtype.HadoopJavaJobRunnerMain.runMethod(HadoopJavaJobRunnerMain.java:260)
	at azkaban.jobtype.HadoopJavaJobRunnerMain.lambda$runMethodAsUser$0(HadoopJavaJobRunnerMain.java:250)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
	at azkaban.jobtype.HadoopJavaJobRunnerMain.runMethodAsUser(HadoopJavaJobRunnerMain.java:242)
	at azkaban.jobtype.HadoopJavaJobRunnerMain.<init>(HadoopJavaJobRunnerMain.java:194)
	at azkaban.jobtype.HadoopJavaJobRunnerMain.main(HadoopJavaJobRunnerMain.java:83)
Caused by: java.lang.NullPointerException
	at org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.getWorkUnitEstSize(KafkaWorkUnitPacker.java:171)
	at org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.getWorkUnitEstLoad(KafkaWorkUnitPacker.java:180)
	at org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.worstFitDecreasingBinPacking(KafkaWorkUnitPacker.java:342)
	at org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaBiLevelWorkUnitPacker.pack(KafkaBiLevelWorkUnitPacker.java:87)
	at org.apache.gobblin.source.extractor.extract.kafka.KafkaSource.getWorkunits(KafkaSource.java:280)
	... 19 more

Issue Links

• is a clone of: GOBBLIN-487 Integrate PasswordManager in R2RestWriterBuilder
• is cloned by: GOBBLIN-1001 Implement TimePartitionGlobFinder
• is cloned by: GOBBLIN-1011 Adjust compaction flow to work with virtual partition
• is cloned by: GOBBLIN-1045 Emit more events in compaction job
• is cloned by: GOBBLIN-1146 Allow configuring autocommit in JDBCWriters
• is cloned by: GOBBLIN-1174 Fail job on FileBasedSource ls invalid source directory
• is cloned by: GOBBLIN-1233 Add case-aware support in WhitelistBlacklist and other small fixes