Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7926 long-lived daemons for query fragment execution, I/O and caching
  3. HIVE-10203

Move fragment execution onto a thread pool

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • llap
    • None

    Description

      Currently we use a lot of thread local magic to make things work in llap. MRLegacyInput makes sure we use the same thread in input and processor. The execution spawns a new thread per fragment and kills it afterwards.

      That's being done because of Tez/MapredContext, ExecContext and IOContext. These need to be shared for a single fragment but not across.

      Way to clean this up is probably by creating a HiveInput that can handle the config + state sharing more efficiently than generic Tez code.

      cc sseth gopalv vikram.dixit

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue HIVE-10162 LLAP: Avoid deserializing the plan > 1 times in a single thread

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              hagleitn Gunther Hagleitner
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: