Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.8.3, 2.9.1, 2.7.6, 3.0.2
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      <!-- markdown -->
      If `HADOOP_CLIENT_SKIP_UNJAR` environment variable is set to true, Apache Hadoop RunJar skips unjar the provided jar.
      Show
      <!-- markdown --> If `HADOOP_CLIENT_SKIP_UNJAR` environment variable is set to true, Apache Hadoop RunJar skips unjar the provided jar.

      Description

      Currently Hadoop's RunJar will unjar the jar provided and look for any jars inside and add them to the classpath. Since most deployments doesn't use jar in jar, but rather uberjars this could be rather time consuming at times and can cause issues related to over consumption of inodes, for something that is in many cases is not used.

      For that purpose there should be an env variable to disable this behavior.

       

      Edit: As requested by Akira Ajisaka in person here is a more detailed description of the issues we are trying to solve with this.

      A good chunk of our workloads are packaged in an uberjar, and are launched as a separate process using the hadoop jar cli. This is has generally been working out pretty well historically, with sub second launch times and good client isolation. Since bumping the host OS to a version patched with Meltdown/Specter patches we do see from time to time load becoming very high even with only a few client processes running and a single unjar process taking up to 10min. 

      While another simple approach would be to abandon using the hadoop jar cli this would most likely take a lot more work than simply disabling unjar for the time being.

        Attachments

        1. HADOOP-15477.001.patch
          1 kB
          Johan Gustavsson
        2. HADOOP-15477.002.patch
          1 kB
          Johan Gustavsson
        3. HADOOP-15477.003.patch
          6 kB
          Johan Gustavsson
        4. HADOOP-15477.004.patch
          6 kB
          Johan Gustavsson

          Activity

            People

            • Assignee:
              johang Johan Gustavsson
              Reporter:
              johang Johan Gustavsson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: