Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6507

Create separate Hive Driver instance for each SQL query in HiveThriftServer2

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.0.2, 1.1.1, 1.2.1, 1.3.0
    • None
    • SQL
    • None

    Description

      In the current implementation of HiveThriftServer2, Hive Driver instances are cached and reused among queries. However, Driver is not thread-safe, and may cause racing conditions. In SPARK-4908, we synchronized HiveContext.runHive to avoid this issue, but this affects concurrency negatively, because no two native commands can be executed concurrently. This is pretty bad for heavy commands like ANALYZE.

      Please refer this comment in SPARK-4908 for details.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6482 Remove synchronization of Hive Native commands

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              lian cheng Cheng Lian
              Cheng Lian Cheng Lian
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: