Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18780

Improve schema discovery For Druid Storage Handler

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Druid integration
    • None

    Description

      Currently, Druid Storage adapter issues a Segment metadata Query every time the query is of type Select or Scan. Not only that but then every input split (map) will do the same as well since it is using the same Serde, this is very expensive and put a lot of pressure on the Druid Cluster. The way to fix this is to add the schema out of the calcite plan instead of serializing the query itself as part of the Hive query context.

      Attachments

        1. HIVE-18780.11.patch
          471 kB
          Slim Bouguerra
        2. HIVE-18780.12.patch
          471 kB
          Slim Bouguerra
        3. HIVE-18780.13.patch
          489 kB
          Slim Bouguerra
        4. HIVE-18780.14.patch
          489 kB
          Slim Bouguerra
        5. HIVE-18780.2.patch
          436 kB
          Slim Bouguerra
        6. HIVE-18780.4.patch
          447 kB
          Slim Bouguerra
        7. HIVE-18780.5.patch
          450 kB
          Slim Bouguerra
        8. HIVE-18780.6.patch
          450 kB
          Slim Bouguerra
        9. HIVE-18780.7.patch
          470 kB
          Slim Bouguerra
        10. HIVE-18780.8.patch
          472 kB
          Slim Bouguerra
        11. HIVE-18780.patch
          419 kB
          Slim Bouguerra
        12. HIVE-18780.patch
          419 kB
          Slim Bouguerra

        Issue Links

          contains

          Sub-task - The sub-task of the issue HIVE-14518 Support 'having' translation for Druid GroupBy queries

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. HIVE-18993 Use Druid Expressions

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. HIVE-18957 Upgrade Calcite version to 1.16.0

          • Major - Major loss of function.
          • Closed

          Activity

            People

              bslim Slim Bouguerra
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: