Hive
  1. Hive
  2. HIVE-2706

StackOverflowError when using custom UDF after adding archive after adding jars

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive
          after adding jars
          (Kevin Wilfong via Yongqiang He)

          Summary:
          https://issues.apache.org/jira/browse/HIVE-2706

          The issue was that the current thread's classloader and the classloader in the
          conf differed due to the prehook updating only the current thread's classloader
          with new jars. Now, it updates both classloaders, fixing the issue.

          When a custom UDF is used in a query after add an archive, such as a zip file,
          after adding jars, the XMLEncoder enters an infinite loop when serializing the
          map reduce task, as part of sending it to be executed. This results in a stack
          overflow error.

          Test Plan: Verified it fixed the stack overflow error.

          Reviewers: JIRA, heyongqiang, njain

          Reviewed By: heyongqiang

          CC: heyongqiang

          Differential Revision: https://reviews.facebook.net/D1167 (Revision 1229885)

          Result = ABORTED
          heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229885
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars (Kevin Wilfong via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2706 The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue. When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error. Test Plan: Verified it fixed the stack overflow error. Reviewers: JIRA, heyongqiang, njain Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D1167 (Revision 1229885) Result = ABORTED heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229885 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

          Show
          Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1192 (See https://builds.apache.org/job/Hive-trunk-h0.21/1192/)
          HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive
          after adding jars
          (Kevin Wilfong via Yongqiang He)

          Summary:
          https://issues.apache.org/jira/browse/HIVE-2706

          The issue was that the current thread's classloader and the classloader in the
          conf differed due to the prehook updating only the current thread's classloader
          with new jars. Now, it updates both classloaders, fixing the issue.

          When a custom UDF is used in a query after add an archive, such as a zip file,
          after adding jars, the XMLEncoder enters an infinite loop when serializing the
          map reduce task, as part of sending it to be executed. This results in a stack
          overflow error.

          Test Plan: Verified it fixed the stack overflow error.

          Reviewers: JIRA, heyongqiang, njain

          Reviewed By: heyongqiang

          CC: heyongqiang

          Differential Revision: https://reviews.facebook.net/D1167

          heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229885
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1192 (See https://builds.apache.org/job/Hive-trunk-h0.21/1192/ ) HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars (Kevin Wilfong via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2706 The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue. When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error. Test Plan: Verified it fixed the stack overflow error. Reviewers: JIRA, heyongqiang, njain Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D1167 heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229885 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Phabricator added a comment -

          heyongqiang has committed the revision "HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars".

          REVISION DETAIL
          https://reviews.facebook.net/D1167

          COMMIT
          https://reviews.facebook.net/rHIVE1229885

          Show
          Phabricator added a comment - heyongqiang has committed the revision " HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars". REVISION DETAIL https://reviews.facebook.net/D1167 COMMIT https://reviews.facebook.net/rHIVE1229885
          Hide
          Phabricator added a comment -

          heyongqiang has accepted the revision "HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars".

          tests passed

          REVISION DETAIL
          https://reviews.facebook.net/D1167

          Show
          Phabricator added a comment - heyongqiang has accepted the revision " HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars". tests passed REVISION DETAIL https://reviews.facebook.net/D1167
          Hide
          He Yongqiang added a comment -

          committed, thanks Kevin!

          Show
          He Yongqiang added a comment - committed, thanks Kevin!
          Hide
          Phabricator added a comment -

          kevinwilfong updated the revision "HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars".
          Reviewers: JIRA, heyongqiang, njain

          Fixed imports.

          REVISION DETAIL
          https://reviews.facebook.net/D1167

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java

          Show
          Phabricator added a comment - kevinwilfong updated the revision " HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars". Reviewers: JIRA, heyongqiang, njain Fixed imports. REVISION DETAIL https://reviews.facebook.net/D1167 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java
          Hide
          Phabricator added a comment -

          kevinwilfong requested code review of "HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars".
          Reviewers: JIRA

          https://issues.apache.org/jira/browse/HIVE-2706

          The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue.

          When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D1167

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java
          ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/2439/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - kevinwilfong requested code review of " HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2706 The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue. When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1167 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2439/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

            People

            • Assignee:
              Kevin Wilfong
              Reporter:
              Kevin Wilfong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development