[GOBBLIN-1962] Gobblin job creation encountered a failure due to a NullPointerException, inability to obtain a job lock for the "Data copy" job.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Component/s: misc
• Assignee: Unassigned
• Reporter: Apekshit Kumar
• Created: 27/Nov/23 04:50
• Updated: 30/Nov/23 04:56

Description

Context :

Data copy back job failed because it's unable to acquire job lock.

Gobblin job creation encountered a failure due to a NullPointerException

 

Code ref-

https://github.com/apache/gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java#L945

 

Error Stacktraces

at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:214) at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:214) at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.call(HelixRetriggeringJobCallable.java:159) at org.apache.gobblin.cluster.GobblinHelixJobScheduler.runJob(GobblinHelixJobScheduler.java:251) at org.apache.gobblin.cluster.GobblinHelixJobScheduler$NonScheduledJobRunner.run(GobblinHelixJobScheduler.java:450) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.NullPointerException at org.apache.gobblin.runtime.AbstractJobLauncher.tryLockJob(AbstractJobLauncher.java:887) at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:202) at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:176) at org.apache.gobblin.cluster.GobblinHelixJobLauncher.<init>(GobblinHelixJobLauncher.java:138) at org.apache.gobblin.cluster.GobblinHelixJobScheduler.buildJobLauncher(GobblinHelixJobScheduler.java:266) at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:201) ... 6 more

 

Job run metastore failure details

mysql> select * from gobblin_job_queue  where queue_id like 'DM-JOB-FINANCIALPACKV2PRD.FPV2-SCORES_1632434443250' order by created_date desc;mysql> select * from gobblin_job_queue  where queue_id like 'DM-JOB-FINANCIALPACKV2PRD.FPV2-SCORES_1632434443250' order by created_date desc;-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| queue_id                                            | job_name                              | deployment_id | failure_exception                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | configs                                                                                                                                                                           | status | job_id | created_date        | updated_date        |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| DM-JOB-FINANCIALPACKV2PRD.FPV2-SCORES_1632434443250 | DM-JOB-FINANCIALPACKV2PRD.FPV2-SCORES |           230 | org.apache.gobblin.runtime.JobException: Failed to run job DM-JOB-FINANCIALPACKV2PRD.FPV2-SCORES at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:214) at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.call(HelixRetriggeringJobCallable.java:159) at org.apache.gobblin.cluster.GobblinHelixJobScheduler.runJob(GobblinHelixJobScheduler.java:251) at org.apache.gobblin.cluster.GobblinHelixJobScheduler$NonScheduledJobRunner.run(GobblinHelixJobScheduler.java:450) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.NullPointerException at org.apache.gobblin.runtime.AbstractJobLauncher.tryLockJob(AbstractJobLauncher.java:887) at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:202) at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:176) at org.apache.gobblin.cluster.GobblinHelixJobLauncher.<init>(GobblinHelixJobLauncher.java:138) at org.apache.gobblin.cluster.GobblinHelixJobScheduler.buildJobLauncher(GobblinHelixJobScheduler.java:266) at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:201) ... 6 more |
{"dataset":

{"batch_id":"20210923150042","name":"financialpackv2prd.fpv2_scores","snapshot_id":"20210923150042"}

,"gobblin":
{"deployment":{"name":"DMP230"}},"namespace":"Chunnel"} | FAILED | NULL   | 2021-09-23 22:00:43 | 2021-09-23 22:40:07 |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

1 row in set (0.00 sec)