Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2288

workaround to fix invalid leveldbjni in maven repository

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: hadoop, ppc64le
    • Labels:
      None
    • Environment:

      Ubuntu 15.04 on ppc64le

      Description

      sudo service hadoop-yarn-timelineserver start
      starting timelineserver, logging to /var/log/hadoop-yarn/yarn-yarn-timelineserver-sparkhab24.out

      • Failed to start Hadoop timelineserver. Return value: 1

      And this is the error that is logged:

      2016-01-27 09:26:03,200 FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer
      java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-702278194701913984.8: /tmp/libleveldbjni-64-1-702278194701913984.8: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a Power PC 64-bit platform)]

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        Pushed to branch-1.1 and merged to the master. Thanks Amir

        Show
        cos Konstantin Boudnik added a comment - Pushed to branch-1.1 and merged to the master. Thanks Amir
        Hide
        cos Konstantin Boudnik added a comment -

        Argh, of course. Need sleep. Thanks. It looks good, so I will commit it (assuming it has been tested fully) in the afternoon. This will unblock the production of RC

        Show
        cos Konstantin Boudnik added a comment - Argh, of course. Need sleep. Thanks. It looks good, so I will commit it (assuming it has been tested fully) in the afternoon. This will unblock the production of RC
        Hide
        asanjar Amir Sanjar added a comment -

        it is a linux environment variable

        Show
        asanjar Amir Sanjar added a comment - it is a linux environment variable
        Hide
        cos Konstantin Boudnik added a comment -

        A quick look: where the HOSTTYPE is coming from?

        Show
        cos Konstantin Boudnik added a comment - A quick look: where the HOSTTYPE is coming from?
        Hide
        asanjar Amir Sanjar added a comment -

        I am testing the new patch for any regression now, should be up shortly. thanks for your patient

        Show
        asanjar Amir Sanjar added a comment - I am testing the new patch for any regression now, should be up shortly. thanks for your patient
        Hide
        cos Konstantin Boudnik added a comment -

        At this point, everything else is ready to produce RC1 of the release. Any chance you can have the fix by the end of tomorrow? Alternatively, we can document the work-around and get this fixed in the next release.

        Show
        cos Konstantin Boudnik added a comment - At this point, everything else is ready to produce RC1 of the release. Any chance you can have the fix by the end of tomorrow? Alternatively, we can document the work-around and get this fixed in the next release.
        Hide
        asanjar Amir Sanjar added a comment - - edited

        nope, my patch is not complete yet. Reducing the severity.

        Show
        asanjar Amir Sanjar added a comment - - edited nope, my patch is not complete yet. Reducing the severity.
        Hide
        cos Konstantin Boudnik added a comment - - edited

        Is this really a blocker though? With my RM hat on I don't think so

        Show
        cos Konstantin Boudnik added a comment - - edited Is this really a blocker though? With my RM hat on I don't think so
        Hide
        cos Konstantin Boudnik added a comment -

        hey Amir Sanjar. Anything so far? Thanks!

        Show
        cos Konstantin Boudnik added a comment - hey Amir Sanjar . Anything so far? Thanks!
        Hide
        oflebbe Olaf Flebbe added a comment -

        This is a viable thing to do. To be compatible with docker packages build this should be scripted in do-components-build of hadoop.

        The other possibility is to add a suitable leveldb binding either as a dependency into the package (if it is already available as a package) or to build the shared library within for instance bigtop-utils and provide a suitable extension of the HADOOP_CLASS_PATH property which is already handled by /etc/hadoop/conf/hadoop-env.sh

        Show
        oflebbe Olaf Flebbe added a comment - This is a viable thing to do. To be compatible with docker packages build this should be scripted in do-components-build of hadoop. The other possibility is to add a suitable leveldb binding either as a dependency into the package (if it is already available as a package) or to build the shared library within for instance bigtop-utils and provide a suitable extension of the HADOOP_CLASS_PATH property which is already handled by /etc/hadoop/conf/hadoop-env.sh
        Hide
        cos Konstantin Boudnik added a comment -

        Don't worry. I have cut of the branch, so we aren't blocking the master anymore and can commit it whenever you're ready

        Show
        cos Konstantin Boudnik added a comment - Don't worry. I have cut of the branch, so we aren't blocking the master anymore and can commit it whenever you're ready
        Hide
        asanjar Amir Sanjar added a comment -

        Can't miss 1.1 release. should have a patch proposal by the morning, it is going to be a long night

        Show
        asanjar Amir Sanjar added a comment - Can't miss 1.1 release. should have a patch proposal by the morning, it is going to be a long night
        Hide
        cos Konstantin Boudnik added a comment -

        Do you want to have it in 1.1?

        Show
        cos Konstantin Boudnik added a comment - Do you want to have it in 1.1?
        Hide
        asanjar Amir Sanjar added a comment -

        Issue is caused by Leveldbjni-1.8-all artifact hosted on central maven repository, happened to be platform sensitive, built for x86 !!
        A possible solution:
        As part of bigtop/slaves:ubuntu-15.04 docker image build, build ppc64le version of Leveldini-1.8 liabrary and then update the local maven repository with the new artifact .. i.e "mvn clean install"

        Any other ideas?

        Show
        asanjar Amir Sanjar added a comment - Issue is caused by Leveldbjni-1.8-all artifact hosted on central maven repository, happened to be platform sensitive, built for x86 !! A possible solution: As part of bigtop/slaves:ubuntu-15.04 docker image build, build ppc64le version of Leveldini-1.8 liabrary and then update the local maven repository with the new artifact .. i.e "mvn clean install" Any other ideas?

          People

          • Assignee:
            asanjar Amir Sanjar
            Reporter:
            asanjar Amir Sanjar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development