Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19821

Distributed HiveServer2



    • New Feature
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • HiveServer2
    • None


      HS2 deployments often hit OOM issues due to a number of factors: (1) too many concurrent connections, (2) query that scan a large number of partitions have to pull a lot of metadata into memory (e.g. a query reading thousands of partitions requires loading thousands of partitions into memory), (3) very large queries can take up a lot of heap space, especially during query parsing. There are a number of other factors that cause HiveServer2 to run out of memory, these are just some of the more commons ones.

      Distributed HS2 proposes to do all query parsing, compilation, planning, and execution coordination inside a dedicated container. This should significantly decrease memory pressure on HS2 and allow HS2 to scale to a larger number of concurrent users.

      For HoS (and I think Hive-on-Tez) this just requires moving all query compilation, planning, etc. inside the application master for the corresponding Hive session.

      The main benefit here is isolation. A poorly written Hive query cannot bring down an entire HiveServer2 instance and force all other queries to fail.


        1. HIVE-19821_ Distributed HiveServer2.pdf
          101 kB
          Sahil Takiar
        2. HIVE-19821.1.WIP.patch
          122 kB
          Sahil Takiar
        3. HIVE-19821.2.WIP.patch
          122 kB
          Sahil Takiar



            Unassigned Unassigned
            stakiar Sahil Takiar
            0 Vote for this issue
            17 Start watching this issue