Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15388

HiveParser spends lots of time in parsing queries with lots of "("

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Branch: apache-master (applicable with previous releases as well)

      Queries generated via tools can have lots of "(" for "AND/OR" conditions. This causes huge delays in parsing phase when the number of expressions are high.

      e.g

      SELECT `iata`,
             `airport`,
             `city`,
             `state`,
             `country`,
             `lat`,
             `lon`
      FROM airports
      WHERE ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((`airports`.`airport` = "Thigpen"
                                                                                      OR `airports`.`airport` = "Astoria Regional")
                                                                                     OR `airports`.`airport` = "Warsaw Municipal")
                                                                                    OR `airports`.`airport` = "John F Kennedy Memorial")
                                                                                   OR `airports`.`airport` = "Hall-Miller Municipal")
                                                                                  OR `airports`.`airport` = "Atqasuk")
                                                                                 OR `airports`.`airport` = "William B Hartsfield-Atlanta Intl")
                                                                                OR `airports`.`airport` = "Artesia Municipal")
                                                                               OR `airports`.`airport` = "Outagamie County Regional")
                                                                              OR `airports`.`airport` = "Watertown Municipal")
                                                                             OR `airports`.`airport` = "Augusta State")
                                                                            OR `airports`.`airport` = "Aurora Municipal")
                                                                           OR `airports`.`airport` = "Alakanuk")
                                                                          OR `airports`.`airport` = "Austin Municipal")
                                                                         OR `airports`.`airport` = "Auburn Municipal")
                                                                        OR `airports`.`airport` = "Auburn-Opelik")
                                                                       OR `airports`.`airport` = "Austin-Bergstrom International")
                                                                      OR `airports`.`airport` = "Wausau Municipal")
                                                                     OR `airports`.`airport` = "Mecklenburg-Brunswick Regional")
                                                                    OR `airports`.`airport` = "Alva Regional")
                                                                   OR `airports`.`airport` = "Asheville Regional")
                                                                  OR `airports`.`airport` = "Avon Park Municipal")
                                                                 OR `airports`.`airport` = "Wilkes-Barre/Scranton Intl")
                                                                OR `airports`.`airport` = "Marana Northwest Regional")
                                                               OR `airports`.`airport` = "Catalina")
                                                              OR `airports`.`airport` = "Washington Municipal")
                                                             OR `airports`.`airport` = "Wainwright")
                                                            OR `airports`.`airport` = "West Memphis Municipal")
                                                           OR `airports`.`airport` = "Arlington Municipal")
                                                          OR `airports`.`airport` = "Algona Municipal")
                                                         OR `airports`.`airport` = "Chandler")
                                                        OR `airports`.`airport` = "Altus Municipal")
                                                       OR `airports`.`airport` = "Neil Armstrong")
                                                      OR `airports`.`airport` = "Angel Fire")
                                                     OR `airports`.`airport` = "Waycross-Ware County")
                                                    OR `airports`.`airport` = "Colorado City Municipal")
                                                   OR `airports`.`airport` = "Hazelhurst")
                                                  OR `airports`.`airport` = "Kalamazoo County")
                                                 OR `airports`.`airport` = "Granville")
                                                OR `airports`.`airport` = "Silver Springs")
                                               OR `airports`.`airport` = "Whitford")
                                              OR `airports`.`airport` = "Biddeford Municipal")
                                             OR `airports`.`airport` = "Sugarloaf Regional")
                                            OR `airports`.`airport` = "Barnes Municipal")
                                           OR `airports`.`airport` = "Columbus Municipal")
                                          OR `airports`.`airport` = "Battle Mountain")
                                         OR `airports`.`airport` = "Huron County Memorial")
                                        OR `airports`.`airport` = "New Braunfels Municipal")
                                       OR `airports`.`airport` = "Benson Municipal")
                                      OR `airports`.`airport` = "Curtis")
                                     OR `airports`.`airport` = "Marlboro County")
                                    OR `airports`.`airport` = "Broken Bow Municipal")
                                   OR `airports`.`airport` = "Virginia Tech")
                                  OR `airports`.`airport` = "Bryce Canyon")
                                 OR `airports`.`airport` = "Black River Falls Area")
                                OR `airports`.`airport` = "Boca Raton")
                               OR `airports`.`airport` = "Birchwood")
                              OR `airports`.`airport` = "Baudette International")
                             OR `airports`.`airport` = "Blanding Muni")
                            OR `airports`.`airport` = "Bradley International")
                           OR `airports`.`airport` = "Morrilton Municipal")
                          OR `airports`.`airport` = "Igor I Sikorsky Memorial")
                         OR `airports`.`airport` = "Broadus")
                        OR `airports`.`airport` = "Laurence G Hanscom")
                       OR `airports`.`airport` = "Southwest Michigan Regional")
                      OR `airports`.`airport` = "Bethel")
                     OR `airports`.`airport` = "Bradford Reg")
                    OR `airports`.`airport` = "Scotts Bluff County")
                   OR `airports`.`airport` = "Boeing Field/King County Intl")
                  OR `airports`.`airport` = "Buffalo Municipal")
                 OR `airports`.`airport` = "Meadows")
                OR `airports`.`airport` = "Mobile Downtown")
               OR `airports`.`airport` = "Virgil I Grissom Municipal")
              OR `airports`.`airport` = "Silver Bay Municipal");
      

      This query would take more than minutes to parse and compile.

      Temp Workaround: If all "("s in "where" condition are removed manually, this query would complete in 2-3 seconds.

      Problem exists in older versions as well (but in older versions this can make HS2 unresponsive for large amount of time, as Driver ends up taking global compiler lock in this phase).

        Attachments

        1. hive-15388.stacktrace.txt
          121 kB
          Rajesh Balamohan
        2. HIVE-15388.01.patch
          27 kB
          Pengcheng Xiong
        3. HIVE-15388.02.patch
          58 kB
          Pengcheng Xiong
        4. HIVE-15388.03.patch
          59 kB
          Pengcheng Xiong
        5. HIVE-15388.04.patch
          70 kB
          Pengcheng Xiong
        6. HIVE-15388.05.patch
          70 kB
          Pengcheng Xiong
        7. HIVE-15388.06.patch
          73 kB
          Pengcheng Xiong
        8. HIVE-15388.07.patch
          58 kB
          Pengcheng Xiong
        9. HIVE-15388.08.patch
          59 kB
          Pengcheng Xiong

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                pxiong Pengcheng Xiong
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: