Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      hive.exec.submit.local.task.via.child

      Determines whether local tasks (typically mapjoin hashtable generation phase) runs in separate JVM ("true", recommended) or not. Avoids the overhead of spawning new JVM, but can lead to out-of-memory issues.
      Show
      hive.exec.submit.local.task.via.child Determines whether local tasks (typically mapjoin hashtable generation phase) runs in separate JVM ("true", recommended) or not. Avoids the overhead of spawning new JVM, but can lead to out-of-memory issues.

      Description

      Did some experiments to see if there's a way to speed up unit tests. TestCliDriver seemed to take a lot of time just spinning up/tearing down JVMs. I was also curious to see if running everything on a ram disk would help.

      Results (I ran tests up to authorization_2):

      • Current setup: 40 minutes
      • Single JVM (not using child JVM to run all queries): 8 minutes
      • Single JVM + ram disk: 7 minutes

      So the ram disk didn't help that much. But running tests in single JVM seems worthwhile doing.

        Attachments

        1. HIVE-7271.1.patch
          10 kB
          Gunther Hagleitner
        2. HIVE-7271.2.patch
          33 kB
          Gunther Hagleitner
        3. HIVE-7271.3.patch
          32 kB
          Gunther Hagleitner
        4. HIVE-7271.4.patch
          32 kB
          Gunther Hagleitner
        5. HIVE-7271.5.patch
          32 kB
          Gunther Hagleitner
        6. HIVE-7271.6.patch
          35 kB
          Gunther Hagleitner
        7. HIVE-7271.7.patch
          36 kB
          Gunther Hagleitner

          Issue Links

            Activity

              People

              • Assignee:
                hagleitn Gunther Hagleitner
                Reporter:
                hagleitn Gunther Hagleitner
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: