HBase
  1. HBase
  2. HBASE-9003

TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 0.99.2
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is the problem: TableMapReduceUtil#addDependencyJars relies on org.apache.hadoop.util.JarFinder if available to call getJar(). However getJar() uses File.createTempFile() to create a temporary file under hadoop.tmp.dir/target/test-dir. Due HADOOP-9737 the created jar and its content is not purged after the JVM is destroyed. Since most configurations point hadoop.tmp.dir under /tmp the generated jar files get purged by tmpwatch or a similar tool, but boxes that have hadoop.tmp.dir pointing to a different location not monitored by tmpwatch will pile up a collection of jars causing all kind of issues. Since JarFinder#getJar is not a public API from Hadoop (see Alejandro Abdelnur comment on HADOOP-9737) we shouldn't use that as part of TableMapReduceUtil in order to avoid this kind of issues.

      1. HBASE-9003.v0.patch
        2 kB
        Esteban Gutierrez
      2. HBASE-9003.v1.patch
        25 kB
        Esteban Gutierrez
      3. HBASE-9003.v2.patch
        26 kB
        Esteban Gutierrez
      4. HBASE-9003.v2.patch
        26 kB
        stack
      5. HBASE-9003.v3.patch
        7 kB
        Nick Dimiduk

        Activity

        Esteban Gutierrez created issue -
        Esteban Gutierrez made changes -
        Field Original Value New Value
        Assignee Esteban Gutierrez [ esteban ]
        Esteban Gutierrez made changes -
        Attachment HBASE-9003.v0.patch [ 12642385 ]
        Esteban Gutierrez made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Esteban Gutierrez made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Affects Version/s 0.92.2 [ 12319888 ]
        Affects Version/s 0.95.1 [ 12324288 ]
        Affects Version/s 0.94.9 [ 12324431 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Esteban Gutierrez made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Esteban Gutierrez made changes -
        Attachment HBASE-9003.v1.patch [ 12642412 ]
        Esteban Gutierrez made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Esteban Gutierrez made changes -
        Attachment HBASE-9003.v2.patch [ 12642563 ]
        Enis Soztutar made changes -
        Fix Version/s 0.99.1 [ 12328551 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Enis Soztutar made changes -
        Fix Version/s 2.0.0 [ 12327188 ]
        Enis Soztutar made changes -
        Fix Version/s 0.99.2 [ 12328822 ]
        Fix Version/s 0.99.1 [ 12328551 ]
        stack made changes -
        Attachment HBASE-9003.v2.patch [ 12678245 ]
        Nick Dimiduk made changes -
        Attachment HBASE-9003.v3.patch [ 12678379 ]
        stack made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Enis Soztutar made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Esteban Gutierrez
            Reporter:
            Esteban Gutierrez
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development