Hive
  1. Hive
  2. HIVE-3273

Add avro jars into hive execution classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Build Infrastructure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      avro*.jar should be added to hive execution classpath

        Issue Links

          Activity

          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3273 [jira] Add avro jars into hive execution classpath
          (Zhenxiao Luo via Carl Steinbach)

          Summary:
          HIVE-3273: Add avro jars into hive execution classpath

          avro*.jar should be added to hive execution classpath

          Test Plan: EMPTY

          Reviewers: JIRA, cwsteinbach

          Reviewed By: cwsteinbach

          Differential Revision: https://reviews.facebook.net/D4209 (Revision 1367708)

          Result = ABORTED
          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367708
          Files :

          • /hive/trunk/ql/build.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3273 [jira] Add avro jars into hive execution classpath (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3273 : Add avro jars into hive execution classpath avro*.jar should be added to hive execution classpath Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D4209 (Revision 1367708) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367708 Files : /hive/trunk/ql/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1581 (See https://builds.apache.org/job/Hive-trunk-h0.21/1581/)
          HIVE-3273 [jira] Add avro jars into hive execution classpath
          (Zhenxiao Luo via Carl Steinbach)

          Summary:
          HIVE-3273: Add avro jars into hive execution classpath

          avro*.jar should be added to hive execution classpath

          Test Plan: EMPTY

          Reviewers: JIRA, cwsteinbach

          Reviewed By: cwsteinbach

          Differential Revision: https://reviews.facebook.net/D4209 (Revision 1367708)

          Result = SUCCESS
          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367708
          Files :

          • /hive/trunk/ql/build.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1581 (See https://builds.apache.org/job/Hive-trunk-h0.21/1581/ ) HIVE-3273 [jira] Add avro jars into hive execution classpath (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3273 : Add avro jars into hive execution classpath avro*.jar should be added to hive execution classpath Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D4209 (Revision 1367708) Result = SUCCESS cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367708 Files : /hive/trunk/ql/build.xml
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.10.0 [ 12320745 ]
          Resolution Fixed [ 1 ]
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks Zhenxiao!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
          Carl Steinbach made changes -
          Component/s Build Infrastructure [ 12312593 ]
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Zhenxiao Luo made changes -
          Link This issue is related to HIVE-3301 [ HIVE-3301 ]
          Hide
          Zhenxiao Luo added a comment -

          @Edward:

          I found this error when I am backporting HIVE-895 to an old version of Hive(CDH3), where I did not put avro jars into hive execution classpath, and kept having the following exception:

          java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader

          We do not have this error on current trunk, since Hadoop classpath is not setting correctly in build-common.xml "test" target. I think this should be fixed in HIVE-3274.

          I think fixing this would emphasis that it is not enough to put avro jars in the hive client classpath. Avro jars should be put in hive execution(the mapreduce job) classpath. This would prevent future bugs.

          What do you think about it?

          Thanks,
          Zhenxiao

          Show
          Zhenxiao Luo added a comment - @Edward: I found this error when I am backporting HIVE-895 to an old version of Hive(CDH3), where I did not put avro jars into hive execution classpath, and kept having the following exception: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader We do not have this error on current trunk, since Hadoop classpath is not setting correctly in build-common.xml "test" target. I think this should be fixed in HIVE-3274 . I think fixing this would emphasis that it is not enough to put avro jars in the hive client classpath. Avro jars should be put in hive execution(the mapreduce job) classpath. This would prevent future bugs. What do you think about it? Thanks, Zhenxiao
          Hide
          Edward Capriolo added a comment -

          I can see this happening but when exactly do we get this error?

          Show
          Edward Capriolo added a comment - I can see this happening but when exactly do we get this error?
          Zhenxiao Luo made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zhenxiao Luo made changes -
          Attachment HIVE-3273.1.patch.txt [ 12537094 ]
          Hide
          Zhenxiao Luo added a comment -

          review request submitted:
          https://reviews.facebook.net/D4209

          Show
          Zhenxiao Luo added a comment - review request submitted: https://reviews.facebook.net/D4209
          Hide
          Zhenxiao Luo added a comment -

          Since Hadoop classpath is not setting correctly in build-common.xml "test" target, even currently we did not add avro jars into hive execution classpath, it is still running OK.

          While, we may get class def not found error, if not putting avro jars into the hive mapreduce job execution classpath:

          Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader

          Show
          Zhenxiao Luo added a comment - Since Hadoop classpath is not setting correctly in build-common.xml "test" target, even currently we did not add avro jars into hive execution classpath, it is still running OK. While, we may get class def not found error, if not putting avro jars into the hive mapreduce job execution classpath: Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader
          Zhenxiao Luo made changes -
          Field Original Value New Value
          Link This issue relates to HIVE-3274 [ HIVE-3274 ]
          Zhenxiao Luo created issue -

            People

            • Assignee:
              Zhenxiao Luo
              Reporter:
              Zhenxiao Luo
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development