[HIVE-21194] DruidStorageHandler should set a version of segment to UTC

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.1.0
• Component/s: Druid integration
• Assignee: Seung-Hyun Cheong
• Reporter: Seung-Hyun Cheong
• Created: 31/Jan/19 15:06
• Updated: 05/Jan/23 15:41
• Resolved: 05/Jan/23 15:41

Description

Exception while running a KillTask

2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z, type=kill, dataSource=upload}]
io.druid.java.util.common.ISE: WTF?! Unused segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00] has version[2019-01-31T01:12:32.289+09:00] > task version[2019-01-30T16:58:29.992Z]
	at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

 

Reason

KillTask compares versions

KillTask.java#L88

if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
  throw new ISE(
  "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
  unusedSegment.getId(),
  unusedSegment.getVersion(),
  myLock.getVersion()
  );
}

  

KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")

TaskLockbox.java#L593

version = DateTimes.nowUtc().toString();

  

Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")

DruidStorageHandler.java#L755

jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new DateTime().toString());

 

 

Suggestion

Because druid uses UTC only, DruidStorageHandler should set a version of segment to UTC.