Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.1.0
    • Fix Version/s: 0.2.0
    • Component/s: general
    • Labels:
      None

      Description

      The Hadoop packages go through some effort (see /usr/lib/hadoop-0.20/bin/hadoop-config.sh) to guess at a good JVM to use if JAVA_HOME isn't set. For example, if both OpenJDK and Oracle JDK are installed, hadoop-config.sh will choose Oracle every time. Other packages that Bigtop integrates, don't follow the same practice (ZK, Pig, etc.). It would be nice to have this common code shared.

      1. 0001-BIGTOP-25.-Add-a-new-bigtop-utils-package-which-prov.patch
        27 kB
        Bruno Mahé
      2. 0002-BIGTOP-25.-Part-2.patch
        5 kB
        Bruno Mahé
      3. bigtop-javahome
        0.6 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Hide
          Bruno Mahé added a comment -

          Second patch to be applied on top of the first one

          Show
          Bruno Mahé added a comment - Second patch to be applied on top of the first one
          Hide
          Bruno Mahé added a comment -

          Thanks a lot!

          • can we use Bigtop as a moniker instead of bigTop or any other kinds of capitalization? when used in mailing list/http URLs lets use bigtop
            -> Good catch! So I checked the case used for the proposal and the website, and fixed the strings to Bigtop
          • you sure about /usr/libexec and not something like /usr/lib/bigtop-utils/ ?
            -> See http://fedoraproject.org/wiki/Packaging:Guidelines#Libexecdir
            Although I am quite tempted to put everything in /usr/lib/bigtop-utils for the sake of consistency.
            Let's check it in as is and we can update it if it proves to be too troublesome.
          • lots of references to HBase in the deb/.../copyright. cut-n-paste errors?
            -> Good catch! It's fixed
          • rpm/bigtop-utils/SPECS/.gitignore has weird content
            -> It's a copy and paste from the hbase one. I put something else
          • would it make more sense to version this package together with Bigtop IOW, BIGTOP_UTILS_BASE_VERSION=BIGTOP_VERSION ?
            -> I like the flexibility of having a separate version number. But on the other hand reusing the bigtop version make things simpler. So I updated it.

          See second patch attached

          Show
          Bruno Mahé added a comment - Thanks a lot! can we use Bigtop as a moniker instead of bigTop or any other kinds of capitalization? when used in mailing list/http URLs lets use bigtop -> Good catch! So I checked the case used for the proposal and the website, and fixed the strings to Bigtop you sure about /usr/libexec and not something like /usr/lib/bigtop-utils/ ? -> See http://fedoraproject.org/wiki/Packaging:Guidelines#Libexecdir Although I am quite tempted to put everything in /usr/lib/bigtop-utils for the sake of consistency. Let's check it in as is and we can update it if it proves to be too troublesome. lots of references to HBase in the deb/.../copyright. cut-n-paste errors? -> Good catch! It's fixed rpm/bigtop-utils/SPECS/.gitignore has weird content -> It's a copy and paste from the hbase one. I put something else would it make more sense to version this package together with Bigtop IOW, BIGTOP_UTILS_BASE_VERSION=BIGTOP_VERSION ? -> I like the flexibility of having a separate version number. But on the other hand reusing the bigtop version make things simpler. So I updated it. See second patch attached
          Hide
          Roman Shaposhnik added a comment - - edited

          Looks good in general. +1. A couple of nitpicks before you commit, though:

          • can we use Bigtop as a moniker instead of bigTop or any other kinds of capitalization? when used in mailing list/http URLs lets use bigtop
          • you sure about /usr/libexec and not something like /usr/lib/bigtop-utils/ ?
          • lots of references to HBase in the deb/.../copyright. cut-n-paste errors?
          • rpm/bigtop-utils/SPECS/.gitignore has weird content
          • would it make more sense to version this package together with Bigtop IOW, BIGTOP_UTILS_BASE_VERSION=BIGTOP_VERSION ?

          P.S. Really liked this line: BuildRoot: %(mktemp -ud %{_tmppath}/%

          {name}

          -%

          {version}

          -%

          {release}

          -XXXXXX)
          And thanks for taking care of Cloudera references. I actually have a JIRA to clean it up even more.

          Show
          Roman Shaposhnik added a comment - - edited Looks good in general. +1. A couple of nitpicks before you commit, though: can we use Bigtop as a moniker instead of bigTop or any other kinds of capitalization? when used in mailing list/http URLs lets use bigtop you sure about /usr/libexec and not something like /usr/lib/bigtop-utils/ ? lots of references to HBase in the deb/.../copyright. cut-n-paste errors? rpm/bigtop-utils/SPECS/.gitignore has weird content would it make more sense to version this package together with Bigtop IOW, BIGTOP_UTILS_BASE_VERSION=BIGTOP_VERSION ? P.S. Really liked this line: BuildRoot: %(mktemp -ud %{_tmppath}/% {name} -% {version} -% {release} -XXXXXX) And thanks for taking care of Cloudera references. I actually have a JIRA to clean it up even more.
          Hide
          Bruno Mahé added a comment -

          Here is a patch to add a new package named bigtop-utils and which will provide some java autodetection code.
          Some notes:

          • I replaced along the way some Cloudera URLs by some BigTop ones
          • I had to enhance the build system so we can build packages which don't have source tarball
          • it put the script on /usr/libexec on RPM and /usr/lib/bigtop-utils on debs (debian does not have /usr/libexec)

          packages were built on fedora and maverick

          Show
          Bruno Mahé added a comment - Here is a patch to add a new package named bigtop-utils and which will provide some java autodetection code. Some notes: I replaced along the way some Cloudera URLs by some BigTop ones I had to enhance the build system so we can build packages which don't have source tarball it put the script on /usr/libexec on RPM and /usr/lib/bigtop-utils on debs (debian does not have /usr/libexec) packages were built on fedora and maverick
          Hide
          Eli Collins added a comment -

          Can't apt/yum/etc... tell you who is providing "java"?

          See HADOOP-6605 for some more background.

          Show
          Eli Collins added a comment - Can't apt/yum/etc... tell you who is providing "java"? See HADOOP-6605 for some more background.
          Hide
          Peter Linnell added a comment -

          +1 for the bigtop-utils approach as well

          RPM:

          1. Update this value to point to the installation of Java on your system
            JAVA_HOME=/usr/java/default

          might be prefereable ?

          Show
          Peter Linnell added a comment - +1 for the bigtop-utils approach as well RPM: Update this value to point to the installation of Java on your system JAVA_HOME=/usr/java/default might be prefereable ?
          Hide
          Bruno Mahé added a comment -

          I'm also +1 on the bigtop-utils approach too.

          So we are two leaning toward the bigtop-utils approach. Let me do it this coming week end.
          Let me assign this ticket to myself then.

          Note: Don't hesitate to continue to provide other alternative ideas.

          Show
          Bruno Mahé added a comment - I'm also +1 on the bigtop-utils approach too. So we are two leaning toward the bigtop-utils approach. Let me do it this coming week end. Let me assign this ticket to myself then. Note: Don't hesitate to continue to provide other alternative ideas.
          Hide
          Roman Shaposhnik added a comment -

          @Bruno

          I'm +1 on the bigtop-utils approach too, but that'll be some work. How about if we, at least, put the following into
          all of our /etc/default/* files for now:

          RPM:

          1. Update this value to point to the installation of Java on your system
            JAVA_HOME=/usr/java/latest

          Debian:

          1. Update this value to point to the installation of Java on your system
            JAVA_HOME=/usr/lib/jvm/java-6-sun
          Show
          Roman Shaposhnik added a comment - @Bruno I'm +1 on the bigtop-utils approach too, but that'll be some work. How about if we, at least, put the following into all of our /etc/default/* files for now: RPM: Update this value to point to the installation of Java on your system JAVA_HOME=/usr/java/latest Debian: Update this value to point to the installation of Java on your system JAVA_HOME=/usr/lib/jvm/java-6-sun
          Hide
          Peter Linnell added a comment -

          Moreover, it does not it give us what we really need which is an environment variable.

          Show
          Peter Linnell added a comment - Moreover, it does not it give us what we really need which is an environment variable.
          Hide
          Bruno Mahé added a comment -

          Can't apt/yum/etc... tell you who is providing "java"?

          They can do that.
          Unfortunately, I haven't seen any intersection in what they provide.

          Show
          Bruno Mahé added a comment - Can't apt/yum/etc... tell you who is providing "java"? They can do that. Unfortunately, I haven't seen any intersection in what they provide.
          Hide
          Bruno Mahé added a comment -

          I am not really comfortable having some java auto-detection code in /etc/default.
          That said, do we really want to auto-detect java instead of using the default one, which is supposedly the one the user want to use? But either way, we would need a way to extract the java home from there.

          Regarding the autodetection code, here are some additional ideas:

          • We put it in a wrapper for each binary that needs it. And ideally open a ticket upstream so they can fix it directly
          • We create a package named something like "bigtop-utils" which would put that file somewhere on the filesystem (ex: /usr/libexec) and make scripts source it. This way, the code would not need to be duplicated, and be easily updated as needed.
          Show
          Bruno Mahé added a comment - I am not really comfortable having some java auto-detection code in /etc/default. That said, do we really want to auto-detect java instead of using the default one, which is supposedly the one the user want to use? But either way, we would need a way to extract the java home from there. Regarding the autodetection code, here are some additional ideas: We put it in a wrapper for each binary that needs it. And ideally open a ticket upstream so they can fix it directly We create a package named something like "bigtop-utils" which would put that file somewhere on the filesystem (ex: /usr/libexec) and make scripts source it. This way, the code would not need to be duplicated, and be easily updated as needed.
          Hide
          Patrick Hunt added a comment -

          Can't apt/yum/etc... tell you who is providing "java"?

          From ZK perspective I'd be fine with you determining the java_home via BT code (e.g. this attachment), or if you updated our existing scripts (zkEnv.sh) to search for it in some other way. (iirc currently we just use "java" unless JAVA_HOME is set)

          Show
          Patrick Hunt added a comment - Can't apt/yum/etc... tell you who is providing "java"? From ZK perspective I'd be fine with you determining the java_home via BT code (e.g. this attachment), or if you updated our existing scripts (zkEnv.sh) to search for it in some other way. (iirc currently we just use "java" unless JAVA_HOME is set)
          Hide
          Roman Shaposhnik added a comment -

          Solving this in its full glory is very important but unlikely for 0.2.0. Thus I'm looking for a Bigtop-level solution
          to unblock our package installation (currently all sorts of things fail because we simply can't do anything without
          JAVA_HOME being set).

          Now, as for Bigtop here's what I would like to propose: we take the attached code and put it into the post-install
          phase of our packages. This code will either find Java and update and explicit setting of JAVA_HOME in an
          appropriate /etc/default/foo file or it will issue a warning during package installation telling user to do it.

          Thoughts?

          Show
          Roman Shaposhnik added a comment - Solving this in its full glory is very important but unlikely for 0.2.0. Thus I'm looking for a Bigtop-level solution to unblock our package installation (currently all sorts of things fail because we simply can't do anything without JAVA_HOME being set). Now, as for Bigtop here's what I would like to propose: we take the attached code and put it into the post-install phase of our packages. This code will either find Java and update and explicit setting of JAVA_HOME in an appropriate /etc/default/foo file or it will issue a warning during package installation telling user to do it. Thoughts?
          Hide
          Steve Loughran added a comment -

          This is surprisingly hard, especially if you want to find a full JDK, rather than just a JRE. It is a painful bootstrapping problem, along with that of deciding which is the JDK to use.

          As it's pre-Java-boot-phase, it ends up being shell script, and then you fall into the differences between bash, Unix sh, cygwin, OS/X.

          What might be good would be to say "use Python" (or similar) for all shell-level operations, so we have a language for the shell-level code that is both standard across platforms and human-readable. The java_home detection could be moved to a python class, and others added, along with tests.

          Another thing to consider is having some RPMs/Debs that explicitly bond to openjdk, oraclejdk or others; people install the one they want. The problem with this approach is that you end up creating two exclusive packages, and that complicates package deployment testing

          Show
          Steve Loughran added a comment - This is surprisingly hard, especially if you want to find a full JDK, rather than just a JRE. It is a painful bootstrapping problem, along with that of deciding which is the JDK to use. As it's pre-Java-boot-phase, it ends up being shell script, and then you fall into the differences between bash, Unix sh, cygwin, OS/X. What might be good would be to say "use Python" (or similar) for all shell-level operations, so we have a language for the shell-level code that is both standard across platforms and human-readable. The java_home detection could be moved to a python class, and others added, along with tests. Another thing to consider is having some RPMs/Debs that explicitly bond to openjdk, oraclejdk or others; people install the one they want. The problem with this approach is that you end up creating two exclusive packages, and that complicates package deployment testing

            People

            • Assignee:
              Bruno Mahé
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development