HBase
  1. HBase
  2. HBASE-9003

TableMapReduceUtil should not rely on org.apache.hadoop.util.JarFinder#getJar

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.92.2, 0.95.1, 0.94.9
    • Fix Version/s: None
    • Component/s: mapreduce
    • Labels:
      None

      Description

      This is the problem: TableMapReduceUtil#addDependencyJars relies on org.apache.hadoop.util.JarFinder if available to call getJar(). However getJar() uses File.createTempFile() to create a temporary file under hadoop.tmp.dir/target/test-dir. Due HADOOP-9737 the created jar and its content is not purged after the JVM is destroyed. Since most configurations point hadoop.tmp.dir under /tmp the generated jar files get purged by tmpwatch or a similar tool, but boxes that have hadoop.tmp.dir pointing to a different location not monitored by tmpwatch will pile up a collection of jars causing all kind of issues. Since JarFinder#getJar is not a public API from Hadoop (see Alejandro Abdelnur comment on HADOOP-9737) we shouldn't use that as part of TableMapReduceUtil in order to avoid this kind of issues.

        Activity

        Hide
        Nick Dimiduk added a comment -

        HADOOP-9426 is intended to expose JarFinder as public API. It was included in addDependencyJars in order to facilitate launching MapReduce jobs from within other MapReduce jobs. This was consumed as part of a now deprecated HCatalog API.

        Show
        Nick Dimiduk added a comment - HADOOP-9426 is intended to expose JarFinder as public API. It was included in addDependencyJars in order to facilitate launching MapReduce jobs from within other MapReduce jobs. This was consumed as part of a now deprecated HCatalog API.
        Hide
        Esteban Gutierrez added a comment -

        Nick Dimiduk besides HCatalog is there any other consumer of JarFinder by the ecosystem? So far I researched is the only one around and for HBase I think it causes unnecessary headaches since nobody expects having 10s of GBs of jars under hadoop.tmp.dir after few days of submitting MR jobs using TableMapReduceUtil

        Show
        Esteban Gutierrez added a comment - Nick Dimiduk besides HCatalog is there any other consumer of JarFinder by the ecosystem? So far I researched is the only one around and for HBase I think it causes unnecessary headaches since nobody expects having 10s of GBs of jars under hadoop.tmp.dir after few days of submitting MR jobs using TableMapReduceUtil
        Hide
        Nick Dimiduk added a comment -

        I'm not aware of any other projects making use of it. The feature has only been available since 0.94.6 and it's a relatively esoteric feature of job submission. Maybe send a note to the user list asking if anyone depends on it? If we're removing support, better sooner than later.

        On the other hand, Hadoop not cleaning up after itself is a Hadoop bug, not HBase. HBase isn't the only project interested in seeing JarFinder promoted to a public API.

        Show
        Nick Dimiduk added a comment - I'm not aware of any other projects making use of it. The feature has only been available since 0.94.6 and it's a relatively esoteric feature of job submission. Maybe send a note to the user list asking if anyone depends on it? If we're removing support, better sooner than later. On the other hand, Hadoop not cleaning up after itself is a Hadoop bug, not HBase. HBase isn't the only project interested in seeing JarFinder promoted to a public API.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #830 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/830/)
        HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599)

        • /hbase/trunk/dev-support/findbugs-exclude.xml
        • /hbase/trunk/dev-support/test-patch.properties
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #830 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/830/ ) HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599) /hbase/trunk/dev-support/findbugs-exclude.xml /hbase/trunk/dev-support/test-patch.properties
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4672 (See https://builds.apache.org/job/HBase-TRUNK/4672/)
        HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599)

        • /hbase/trunk/dev-support/findbugs-exclude.xml
        • /hbase/trunk/dev-support/test-patch.properties
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4672 (See https://builds.apache.org/job/HBase-TRUNK/4672/ ) HBASE-9003 Remove the jamon generated classes from the findbugs analysis (nkeywal: rev 1539599) /hbase/trunk/dev-support/findbugs-exclude.xml /hbase/trunk/dev-support/test-patch.properties

          People

          • Assignee:
            Unassigned
            Reporter:
            Esteban Gutierrez
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development