Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12404

Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class.
      Currently Configuration#parse will call url.openStream to get the InputStream for DocumentBuilder to parse.
      Based on the JDK source code, the calling sequence is
      url.openStream => handler.openConnection.getInputStream => new JarURLConnection => JarURLConnection.connect => factory.get(getJarFileURL(), getUseCaches()) => URLJarFile.getInputStream=>JarFile.getInputStream=>ZipFile.getInputStream
      If URLConnection#getUseCaches is true (by default), URLJarFile will be shared for the same URL. If the shared URLJarFile is closed by other users, all the InputStream returned by URLJarFile#getInputStream will be closed based on the document
      So we saw the following exception in a heavy-load system at rare situation which cause a hive job failed

      2014-10-21 23:44:41,856 ERROR org.apache.hadoop.hive.ql.exec.Task: Ended 
      Job = job_1413909398487_3696 with exception 
      'java.lang.RuntimeException(java.io.IOException: Stream closed)' 
      java.lang.RuntimeException: java.io.IOException: Stream closed 
      at 
      org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2484) 
      at 
      org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2337) 
      at 
      org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2254) 
      at org.apache.hadoop.conf.Configuration.get(Configuration.java:861) 
      at 
      org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2030) 
      at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:479) 
      at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:469) 
      at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:187) 
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:582) 
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) 
      at java.security.AccessController.doPrivileged(Native Method) 
      at javax.security.auth.Subject.doAs(Subject.java:415) 
      at 
      org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j 
      ava:1614) 
      at 
      org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:580) 
      at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:598) 
      at 
      org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExe 
      cHelper.java:288) 
      at 
      org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExe 
      cHelper.java:547) 
      at 
      org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426) 
      at 
      org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136) 
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) 
      at 
      org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) 
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1516) 
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1283) 
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101) 
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) 
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:919) 
      at 
      org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation 
      .java:145) 
      at 
      org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation. 
      java:69) 
      at 
      org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.jav 
      a:200) 
      at java.security.AccessController.doPrivileged(Native Method) 
      at javax.security.auth.Subject.doAs(Subject.java:415) 
      at 
      org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j 
      ava:1614) 
      at 
      org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java: 
      502) 
      at 
      org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java: 
      213) 
      at 
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
      at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
      at 
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 
      145) 
      at 
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 
      615) 
      at java.lang.Thread.run(Thread.java:745) 
      Caused by: java.io.IOException: Stream closed 
      at 
      java.util.zip.InflaterInputStream.ensureOpen(InflaterInputStream.java:67) 
      at 
      java.util.zip.InflaterInputStream.read(InflaterInputStream.java:142) 
      at java.io.FilterInputStream.read(FilterInputStream.java:133) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStr 
      eam.read(XMLEntityManager.java:2902) 
      at 
      com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java: 
      302) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScan 
      ner.java:1753) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntity 
      Scanner.java:1426) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$Frag 
      mentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocu 
      mentScannerImpl.java:606) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNS 
      DocumentScannerImpl.java:117) 
      at 
      com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scan 
      Document(XMLDocumentFragmentScannerImpl.java:510) 
      at 
      com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Co 
      nfiguration.java:848) 
      at 
      com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Co 
      nfiguration.java:777) 
      at 
      com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:1 
      41) 
      at 
      com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:2 
      43) 
      at 
      com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentB 
      uilderImpl.java:347) 
      at 
      javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) 
      at 
      org.apache.hadoop.conf.Configuration.parse(Configuration.java:2325) 
      at 
      org.apache.hadoop.conf.Configuration.parse(Configuration.java:2313) 
      at 
      org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2384)
      

      Also we can save a little bit memory, with JarURLConnection's caches disabled.

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 23s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 1s There were no new javac warning messages.
        +1 javadoc 10m 1s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 9s The applied patch generated 1 new checkstyle issues (total was 176, now 176).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 21s Tests passed in hadoop-common.
            64m 19s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12755293/HADOOP-12404.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f103a70
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 9s The applied patch generated 1 new checkstyle issues (total was 176, now 176). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 21s Tests passed in hadoop-common.     64m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755293/HADOOP-12404.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f103a70 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7631/console This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Thanks for chasing this down zhihai xu !!

        I feel the proper solution is to probably fix the JDK Factory class to intercept the close() method and actually close the stream only if there are no more clients. But considering that it might not be easy to get the JDK fix in, I guess your patch makes sense. I understand unit testing this might be non trivial

        +1, LGTM

        Show
        asuresh Arun Suresh added a comment - Thanks for chasing this down zhihai xu !! I feel the proper solution is to probably fix the JDK Factory class to intercept the close() method and actually close the stream only if there are no more clients. But considering that it might not be easy to get the JDK fix in, I guess your patch makes sense. I understand unit testing this might be non trivial +1, LGTM
        Hide
        zxu zhihai xu added a comment -

        thanks Arun Suresh for the review! Yes, No easy way for the test case because all the functionalities to verify are in JDK. which is not accessible from hadoop. Will commit it tomorrow if no one objects.

        Show
        zxu zhihai xu added a comment - thanks Arun Suresh for the review! Yes, No easy way for the test case because all the functionalities to verify are in JDK. which is not accessible from hadoop. Will commit it tomorrow if no one objects.
        Hide
        zxu zhihai xu added a comment -

        Committed it to branch-2 and trunk! thanks Arun Suresh for the review!

        Show
        zxu zhihai xu added a comment - Committed it to branch-2 and trunk! thanks Arun Suresh for the review!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8485 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8485/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8485 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8485/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #411 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/411/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #411 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/411/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #419 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/419/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #419 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/419/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2331/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2331/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #393 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/393/)
        HADOOP-12404. Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #393 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/393/ ) HADOOP-12404 . Disable caching for JarURLConnection to avoid sharing JarFile with other users when loading resource from URL in Configuration class. Contributed by Zhihai Xu (zxu: rev 88d89267ff6b66e144bfcceb09532191975f2a4a) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        anishek anishek added a comment - - edited

        zhihai xu you mentioned

        So we saw the following exception in a heavy-load system at rare situation which cause a hive job failed

        can you please tell me why would this only occur rarely and that too only with heavy-loaded system. Shouldnt this happen always, if two or more users are using the same cached jar for their jobs and one of them completes before the other has loaded resources / classes from it? Also, I thought multiple jobs would use their own JVM's ?

        Show
        anishek anishek added a comment - - edited zhihai xu you mentioned So we saw the following exception in a heavy-load system at rare situation which cause a hive job failed can you please tell me why would this only occur rarely and that too only with heavy-loaded system. Shouldnt this happen always, if two or more users are using the same cached jar for their jobs and one of them completes before the other has loaded resources / classes from it? Also, I thought multiple jobs would use their own JVM's ?
        Hide
        zxu zhihai xu added a comment -

        anishek We see this issue in hivesever2 logs, all the queries share a JVM in different threads for hiveserver2. The finalize statement will also close the InputStream for ZipFile class so it will also depend on when the garbage collection happens. Normally we see this issue happened several times per day in hiveserver2 which run thousand of queries per day. I thought loading jar files should be very quick that is why it happens rarely. After we disable caching, this issue didn't happen any more. Did you see this issue also? What is your environment?

        Show
        zxu zhihai xu added a comment - anishek We see this issue in hivesever2 logs, all the queries share a JVM in different threads for hiveserver2. The finalize statement will also close the InputStream for ZipFile class so it will also depend on when the garbage collection happens. Normally we see this issue happened several times per day in hiveserver2 which run thousand of queries per day. I thought loading jar files should be very quick that is why it happens rarely. After we disable caching, this issue didn't happen any more. Did you see this issue also? What is your environment?
        Hide
        anishek anishek added a comment -

        zhihai xu we only saw this issue once running tpc queries, we tried to recreate it with multiple queries simultaneously but no luck recreating the issue.

        Show
        anishek anishek added a comment - zhihai xu we only saw this issue once running tpc queries, we tried to recreate it with multiple queries simultaneously but no luck recreating the issue.

          People

          • Assignee:
            zxu zhihai xu
            Reporter:
            zxu zhihai xu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development