Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21194

DruidStorageHandler should set a version of segment to UTC

Log workAgile BoardRank to TopRank to BottomAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: Druid integration
    • Labels:
      None

      Description

      Exception while running a KillTask

      2019-01-30T16:58:35,354 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KillTask{id=kill_upload_2018-12-31T00:00:00.000Z_2019-02-05T00:00:00.000Z_2019-02-01T16:52:31.851Z, type=kill, dataSource=upload}]
      io.druid.java.util.common.ISE: WTF?! Unused segment[upload_2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z_2019-01-31T01:12:32.289+09:00] has version[2019-01-31T01:12:32.289+09:00] > task version[2019-01-30T16:58:29.992Z]
      	at io.druid.indexing.common.task.KillTask.run(KillTask.java:94) ~[druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
      	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
      	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
      	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
      

       

      Reason

      KillTask compares versions

      KillTask.java#L88

      if (unusedSegment.getVersion().compareTo(myLock.getVersion()) > 0) {
        throw new ISE(
        "WTF?! Unused segment[%s] has version[%s] > task version[%s]",
        unusedSegment.getId(),
        unusedSegment.getVersion(),
        myLock.getVersion()
        );
      }
      

        

      KillTask version (UTC, e.g. "2019-01-30T16:58:29.992Z")

      TaskLockbox.java#L593

      version = DateTimes.nowUtc().toString();
      

        

      Segment version (UTC+9, e.g. "2019-01-31T01:12:32.289+09:00")

      DruidStorageHandler.java#L755

      jobProperties.put(DruidConstants.DRUID_SEGMENT_VERSION, new DateTime().toString());
      

       

       

      Suggestion

      Because druid uses UTC only, DruidStorageHandler should set a version of segment to UTC.

       

       

       

        Attachments

        1. image-2019-02-01-16-32-17-093.png
          12 kB
          Seung-Hyun Cheong
        2. image-2019-02-01-16-31-56-958.png
          12 kB
          Seung-Hyun Cheong

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              seunghyun.cheong Seung-Hyun Cheong Assign to me
              Reporter:
              seunghyun.cheong Seung-Hyun Cheong

              Dates

              • Created:
                Updated:

                Issue deployment