Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1467

version of hadoop-auth in oozie is wrong

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: debian, rpm
    • Labels:

      Description

      $ rpm -lpq hadoop-2.4.1.726-1.el6.x86_64.rpm | grep hadoop-auth
      /usr/lib/hadoop/hadoop-auth-2.4.1.jar
      /usr/lib/hadoop/hadoop-auth.jar

      $ rpm -lpq oozie-4.0.1.691-1.el6.noarch.rpm | grep hadoop-auth
      /usr/lib/oozie/libtools/hadoop-auth-2.0.2-alpha.jar

      1. BIGTOP-1467.patch
        0.7 kB
        Amir Sanjar
      2. BIGTOP-1467.patch
        1 kB
        Amir Sanjar

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          it's an old issue - tracked above

          Show
          cos Konstantin Boudnik added a comment - it's an old issue - tracked above
          Hide
          rvs Roman Shaposhnik added a comment -

          Yup. The bad news is – Oozie is still a mess from the build perspective. The good news is – this particular issues doesn't actually seem to affect anything.

          Show
          rvs Roman Shaposhnik added a comment - Yup. The bad news is – Oozie is still a mess from the build perspective. The good news is – this particular issues doesn't actually seem to affect anything.
          Hide
          in-chief Xiaoshuang LU added a comment -

          Hi Roman, appears that oozie uses new classes provided by hadoop-auth-2.4.1.jar.

          Show
          in-chief Xiaoshuang LU added a comment - Hi Roman, appears that oozie uses new classes provided by hadoop-auth-2.4.1.jar.
          Hide
          rvs Roman Shaposhnik added a comment -

          Any chance you can tell us how that would break things in practice? Or better yet, any chance you can help work on a patch?

          Show
          rvs Roman Shaposhnik added a comment - Any chance you can tell us how that would break things in practice? Or better yet, any chance you can help work on a patch?
          Hide
          in-chief Xiaoshuang LU added a comment - - edited

          In practice, Oozie will throw something like ClassNotFound and refuse to start, if this jar file version is not correct.

          A workaround is add "-Dhadoop.auth.version=$

          {HADOOP_VERSION}

          " to bigtop-packages/src/common/oozie/do-component-build line 95.

          Show
          in-chief Xiaoshuang LU added a comment - - edited In practice, Oozie will throw something like ClassNotFound and refuse to start, if this jar file version is not correct. A workaround is add "-Dhadoop.auth.version=$ {HADOOP_VERSION} " to bigtop-packages/src/common/oozie/do-component-build line 95.
          Hide
          rvs Roman Shaposhnik added a comment -

          Well, that's why I am curious – I'm pretty sure Oozie works just fine in Bigtop 0.8.0.

          Show
          rvs Roman Shaposhnik added a comment - Well, that's why I am curious – I'm pretty sure Oozie works just fine in Bigtop 0.8.0.
          Hide
          in-chief Xiaoshuang LU added a comment -

          /usr/lib/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://localhost.localdomain:8020 -locallib /usr/lib/oozie/oozie-sharelib.tar.gz
          setting OOZIE_CONFIG=$

          {OOZIE_CONFIG:-/etc/oozie/conf}

          setting CATALINA_BASE=$

          {CATALINA_BASE:-/var/lib/oozie/tomcat-deployment}

          setting CATALINA_TMPDIR=$

          {CATALINA_TMPDIR:-/var/tmp/oozie}

          setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
          setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67
          setting JRE_HOME=$

          {JAVA_HOME}
          setting OOZIE_LOG=/var/log/oozie
          setting CATALINA_PID=/var/run/oozie/oozie.pid
          setting OOZIE_DATA=/hadoop/oozie/data
          setting OOZIE_HTTP_PORT=11000
          setting OOZIE_ADMIN_PORT=11001
          setting JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64
          setting OOZIE_CONFIG=${OOZIE_CONFIG:-/etc/oozie/conf}
          setting CATALINA_BASE=${CATALINA_BASE:-/var/lib/oozie/tomcat-deployment}
          setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
          setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
          setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67
          setting JRE_HOME=${JAVA_HOME}

          setting OOZIE_LOG=/var/log/oozie
          setting CATALINA_PID=/var/run/oozie/oozie.pid
          setting OOZIE_DATA=/hadoop/oozie/data
          setting OOZIE_HTTP_PORT=11000
          setting OOZIE_ADMIN_PORT=11001
          setting JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64
          log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
          log4j:WARN Please initialize the log4j system properly.
          log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
          Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
          at org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName(UserGroupInformation.java:337)
          at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:382)
          at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:132)
          at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
          at org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
          at org.apache.oozie.service.Services.setService(Services.java:354)
          at org.apache.oozie.service.Services.loadServices(Services.java:287)
          at org.apache.oozie.service.Services.init(Services.java:208)
          at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:136)
          at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:52)
          Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
          at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
          ... 10 more

          Show
          in-chief Xiaoshuang LU added a comment - /usr/lib/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://localhost.localdomain:8020 -locallib /usr/lib/oozie/oozie-sharelib.tar.gz setting OOZIE_CONFIG=$ {OOZIE_CONFIG:-/etc/oozie/conf} setting CATALINA_BASE=$ {CATALINA_BASE:-/var/lib/oozie/tomcat-deployment} setting CATALINA_TMPDIR=$ {CATALINA_TMPDIR:-/var/tmp/oozie} setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67 setting JRE_HOME=$ {JAVA_HOME} setting OOZIE_LOG=/var/log/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting OOZIE_DATA=/hadoop/oozie/data setting OOZIE_HTTP_PORT=11000 setting OOZIE_ADMIN_PORT=11001 setting JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64 setting OOZIE_CONFIG=${OOZIE_CONFIG:-/etc/oozie/conf} setting CATALINA_BASE=${CATALINA_BASE:-/var/lib/oozie/tomcat-deployment} setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie} setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67 setting JRE_HOME=${JAVA_HOME} setting OOZIE_LOG=/var/log/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting OOZIE_DATA=/hadoop/oozie/data setting OOZIE_HTTP_PORT=11000 setting OOZIE_ADMIN_PORT=11001 setting JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64 log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName at org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName(UserGroupInformation.java:337) at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:382) at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:132) at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98) at org.apache.oozie.service.Services.setServiceInternal(Services.java:368) at org.apache.oozie.service.Services.setService(Services.java:354) at org.apache.oozie.service.Services.loadServices(Services.java:287) at org.apache.oozie.service.Services.init(Services.java:208) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:136) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:52) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more
          Hide
          rvs Roman Shaposhnik added a comment -

          That script was never actually supported by Bigtop. I was asking more about the core Oozie functionality. Things like starting/stopping the service, using client, etc.

          Show
          rvs Roman Shaposhnik added a comment - That script was never actually supported by Bigtop. I was asking more about the core Oozie functionality. Things like starting/stopping the service, using client, etc.
          Hide
          in-chief Xiaoshuang LU added a comment -

          Since bigtop 0.8 uses hadoop 2.4.1, it does not make any sense to have jar files with version 2.0.2-alpha.

          Show
          in-chief Xiaoshuang LU added a comment - Since bigtop 0.8 uses hadoop 2.4.1, it does not make any sense to have jar files with version 2.0.2-alpha.
          Hide
          asanjar Amir Sanjar added a comment -

          We have experienced the same issue with bigtop 0.8.0 release.I strongly recommand not to have discrepancy with hadoop jar files, specially in production environment.
          Let me know if I could help.

          Show
          asanjar Amir Sanjar added a comment - We have experienced the same issue with bigtop 0.8.0 release.I strongly recommand not to have discrepancy with hadoop jar files, specially in production environment. Let me know if I could help.
          Hide
          cos Konstantin Boudnik added a comment -

          How about a patch, perhaps?

          Show
          cos Konstantin Boudnik added a comment - How about a patch, perhaps?
          Hide
          asanjar Amir Sanjar added a comment -

          sure, I am not familiar with your build process, point me to the build documentation, I would be glad to take a swing at it.

          Show
          asanjar Amir Sanjar added a comment - sure, I am not familiar with your build process, point me to the build documentation, I would be glad to take a swing at it.
          Hide
          cos Konstantin Boudnik added a comment -

          Here's the getting started page https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop

          And the up-to-date info could be found in README files inside of the source code. Please ping user@ list if you feel you need more help.

          Show
          cos Konstantin Boudnik added a comment - Here's the getting started page https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop And the up-to-date info could be found in README files inside of the source code. Please ping user@ list if you feel you need more help.
          Hide
          asanjar Amir Sanjar added a comment -

          please review

          Show
          asanjar Amir Sanjar added a comment - please review
          Hide
          asanjar Amir Sanjar added a comment -

          please review the posted patch. got a clean build

          Show
          asanjar Amir Sanjar added a comment - please review the posted patch. got a clean build
          Hide
          warwithin YoungWoo Kim added a comment -

          Amir Sanjar, Please generate the patch using 'git format-patch' and reattach it. See https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute

          Thanks!

          Show
          warwithin YoungWoo Kim added a comment - Amir Sanjar , Please generate the patch using 'git format-patch' and reattach it. See https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute Thanks!
          Hide
          asanjar Amir Sanjar added a comment -

          updated patch

          Show
          asanjar Amir Sanjar added a comment - updated patch
          Hide
          warwithin YoungWoo Kim added a comment -

          Konstantin Boudnik, It's ready to commit. Please assign Amir Sanjar to this JIRA. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - Konstantin Boudnik , It's ready to commit. Please assign Amir Sanjar to this JIRA. Thanks!
          Hide
          warwithin YoungWoo Kim added a comment -

          Committed. Amir Sanjar, Thank you for your contribution!

          Show
          warwithin YoungWoo Kim added a comment - Committed. Amir Sanjar , Thank you for your contribution!

            People

            • Assignee:
              asanjar Amir Sanjar
              Reporter:
              in-chief Xiaoshuang LU
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development