Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20349

Implement Retry Logic in HiveDruidSplit for Scan Queries

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: Druid integration
    • Labels:
      None

      Description

      while distributing druid scan query we check where the segments are loaded and then each HiveDruidSplit directly queries the historical node.
      There are few cases when we need to retry and refetch the segments.

      1. The segment is loaded on multiple historical nodes and one of them went down. in this case when we do not get response from one segment, we query the next replica.
      2. The segment was loaded onto a realtime task and was handed over, when we query the realtime task has already finished. In this case there is no replica. The Split needs to query the broker again for the location of the segment and then send the query to correct historical node.

      This is also the root cause of failure of druidkafkamini_basic.q test, where the segment handover happens before the scan query is executed.

      Note: This is not a problem when we are directly querying Druid brokers as the broker handles the retry logic.

        Attachments

        1. HIVE-20349.patch
          12 kB
          Nishant Bangarwa
        2. HIVE-20349.5.patch
          15 kB
          Nishant Bangarwa
        3. HIVE-20349.4.patch
          15 kB
          Nishant Bangarwa
        4. HIVE-20349.3.patch
          14 kB
          Nishant Bangarwa
        5. HIVE-20349.2.patch
          15 kB
          Nishant Bangarwa
        6. HIVE-20349.1.patch
          14 kB
          Nishant Bangarwa

          Issue Links

            Activity

              People

              • Assignee:
                nishantbangarwa Nishant Bangarwa
                Reporter:
                nishantbangarwa Nishant Bangarwa
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: