Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: general
    • Labels:
      None
    • Environment:

      All

      Description

      The Bigtop distribution cannot be build with either JDK 1.6 nor JDK 1.7

      At least solr giraph and hue have a build requirement on JDK 1.7

      On the homepage JDK 1.6 is mentioned as a requirement.

      Otherwise the Distribution cannot be build with JDK 1.7 , since oozie enforces the requirement to 1.6.

      A big mess.

      What scares me that the jenkins http://builds.apache.org/view/All/job/Bigtop-trunk/ seems to work flawless .

      My jenkins build needs heavy patching, how should I proceed ?

      1. oozie17_2.patch
        0.7 kB
        Olaf Flebbe
      2. jdk17.diff
        2 kB
        Olaf Flebbe

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          I've noticed that this patch adds some extra output by printing the size of the matcher's group. It will be removed as a part of BIGTOP-1395

          Show
          cos Konstantin Boudnik added a comment - I've noticed that this patch adds some extra output by printing the size of the matcher's group. It will be removed as a part of BIGTOP-1395
          Hide
          cos Konstantin Boudnik added a comment -

          Appreciate the explanation.
          Actually, it seems that without eval make won't work. See here

          Show
          cos Konstantin Boudnik added a comment - Appreciate the explanation. Actually, it seems that without eval make won't work. See here
          Hide
          oflebbe Olaf Flebbe added a comment -

          Because variables in bigtop.mk are not automatically propagated to the bogtop.bom file. In fact only VERSIONS of components are propagated.

          Yes, in order to get only "make" to work there is no $(eval) needed.

          The other variables are propagated because of code which is triggered by an $(eval ), a GNU make feature and by special code in the gradle file triggered by an regexp.

          I reused some of this code on order to get the code somewhat cleaner.

          Show
          oflebbe Olaf Flebbe added a comment - Because variables in bigtop.mk are not automatically propagated to the bogtop.bom file. In fact only VERSIONS of components are propagated. Yes, in order to get only "make" to work there is no $(eval) needed. The other variables are propagated because of code which is triggered by an $(eval ), a GNU make feature and by special code in the gradle file triggered by an regexp. I reused some of this code on order to get the code somewhat cleaner.
          Hide
          cos Konstantin Boudnik added a comment -

          Olaf Flebbe, I am curious why the following changes were need to be made?

          1. $(eval BIGTOP_BOM += JDK_VERSION=$(JDK_VERSION))
            
          2. the packages.gradle
            ??
          Show
          cos Konstantin Boudnik added a comment - Olaf Flebbe , I am curious why the following changes were need to be made? $(eval BIGTOP_BOM += JDK_VERSION=$(JDK_VERSION)) the packages.gradle ??
          Hide
          cos Konstantin Boudnik added a comment -

          +1 on Roman Shaposhnik said.
          Although, could've been worked around with --author

          Show
          cos Konstantin Boudnik added a comment - +1 on Roman Shaposhnik said. Although, could've been worked around with --author
          Hide
          rvs Roman Shaposhnik added a comment -

          +1 and committed! Thanks a million for all the work.

          One small nit: please submit future patches as git format-patch. This makes it way easier to apply.

          Show
          rvs Roman Shaposhnik added a comment - +1 and committed! Thanks a million for all the work. One small nit: please submit future patches as git format-patch. This makes it way easier to apply.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Had to increase the permgen space for oracle jdk 1.7.0_65 on SLES

          Show
          oflebbe Olaf Flebbe added a comment - Had to increase the permgen space for oracle jdk 1.7.0_65 on SLES
          Hide
          oflebbe Olaf Flebbe added a comment -

          Sigh ... now added patch to grade and make system

          Show
          oflebbe Olaf Flebbe added a comment - Sigh ... now added patch to grade and make system
          Hide
          oflebbe Olaf Flebbe added a comment -

          Patch. even did not break the deprecated "make" system

          Show
          oflebbe Olaf Flebbe added a comment - Patch. even did not break the deprecated "make" system
          Hide
          oflebbe Olaf Flebbe added a comment -

          Konstatin, looks like a cleaner solution to use the bigtop.mk file. Will try.

          Show
          oflebbe Olaf Flebbe added a comment - Konstatin, looks like a cleaner solution to use the bigtop.mk file. Will try.
          Hide
          cos Konstantin Boudnik added a comment -

          Olaf, any change you can take it one step away to implement the full solution, where JDK_VERSION is set in the bigtop.ml file. The it will be automatically propagated into a special file bigtop.bom which gets created for every package build and gets sourced into do-component-build. At this point you will be able to use

          ${WORKDIR}/bin/mkdistro.sh -DjavaVersion=${JDK_VERSION} ...
          

          instead of the currently hard-coded one. Thanks!

          BTW, you don't need to remove previous versions of the patches as it - as a general rule - helps to preserve the history of the discussion and gradual improvements.

          Show
          cos Konstantin Boudnik added a comment - Olaf, any change you can take it one step away to implement the full solution, where JDK_VERSION is set in the bigtop.ml file. The it will be automatically propagated into a special file bigtop.bom which gets created for every package build and gets sourced into do-component-build . At this point you will be able to use ${WORKDIR}/bin/mkdistro.sh -DjavaVersion=${JDK_VERSION} ... instead of the currently hard-coded one. Thanks! BTW, you don't need to remove previous versions of the patches as it - as a general rule - helps to preserve the history of the discussion and gradual improvements.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Use Roman Shaposhnik proposal to solve problem.

          Use -DjavaVersion=1.7 to compile

          Show
          oflebbe Olaf Flebbe added a comment - Use Roman Shaposhnik proposal to solve problem. Use -DjavaVersion=1.7 to compile
          Hide
          oflebbe Olaf Flebbe added a comment -

          Works +1 for me. Uploading new patch with proposed patch by Roman Shaposhnik

          Show
          oflebbe Olaf Flebbe added a comment - Works +1 for me. Uploading new patch with proposed patch by Roman Shaposhnik
          Hide
          oflebbe Olaf Flebbe added a comment -

          Testing Roman Shaposhnik suggestion ...

          Show
          oflebbe Olaf Flebbe added a comment - Testing Roman Shaposhnik suggestion ...
          Hide
          gujilangzi Wenwu Peng added a comment -

          +1 for Roman Shaposhnik point

          Show
          gujilangzi Wenwu Peng added a comment - +1 for Roman Shaposhnik point
          Hide
          cos Konstantin Boudnik added a comment -

          Yup, good point Roman Shaposhnik

          Show
          cos Konstantin Boudnik added a comment - Yup, good point Roman Shaposhnik
          Hide
          rvs Roman Shaposhnik added a comment -

          Wouldn't it be simpler to pass -DjavaVersion=XXX in the do-component-build? In fact, I'd actually suggest making JAVA_VERSION and may be even JAVA_TARGET_VERSION part of bigtop.bom that we auto-generate for do-component-build inclusion.

          Thoughts?

          Show
          rvs Roman Shaposhnik added a comment - Wouldn't it be simpler to pass -DjavaVersion=XXX in the do-component-build? In fact, I'd actually suggest making JAVA_VERSION and may be even JAVA_TARGET_VERSION part of bigtop.bom that we auto-generate for do-component-build inclusion. Thoughts?
          Hide
          oflebbe Olaf Flebbe added a comment -

          I had a full rebuild of bigtop with these patches (oozie, sqoop and datafu) with OJDK 1.7 and the full build job was successfull.

          Show
          oflebbe Olaf Flebbe added a comment - I had a full rebuild of bigtop with these patches (oozie, sqoop and datafu) with OJDK 1.7 and the full build job was successfull.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Patch to bigtop to remove 1.6 dependency

          Show
          oflebbe Olaf Flebbe added a comment - Patch to bigtop to remove 1.6 dependency
          Hide
          oflebbe Olaf Flebbe added a comment -

          Hi,

          (I am working on a bigtop distro for Debian Wheezy. I have no interested in oozie either, only a working build so far. BTW: Could you please consider my spark and datafu patches to the control files from BIGTOP-1381 , since debian build is broken witch this regard)

          it is enforced my the maven enforce plugin . I added a patch to do-component-build to remove these lines as the least invasive thing to change.
          Since I see no policy on which targetJavaVersion to use in bigtop I refrained from changing the Target Java Version.

          Please apply

          • <requireJavaVersion>
          • <version>[$ {javaVersion}.0,${javaVersion}

            .1000}]</version>

          • </requireJavaVersion>

          Thanks
          Olaf

          Show
          oflebbe Olaf Flebbe added a comment - Hi, (I am working on a bigtop distro for Debian Wheezy. I have no interested in oozie either, only a working build so far. BTW: Could you please consider my spark and datafu patches to the control files from BIGTOP-1381 , since debian build is broken witch this regard) it is enforced my the maven enforce plugin . I added a patch to do-component-build to remove these lines as the least invasive thing to change. Since I see no policy on which targetJavaVersion to use in bigtop I refrained from changing the Target Java Version. Please apply <requireJavaVersion> <version>[$ {javaVersion}.0,${javaVersion} .1000}]</version> </requireJavaVersion> Thanks Olaf
          Hide
          cos Konstantin Boudnik added a comment -

          Bigtop trunk is far from flawless, as we have two troubled slaves: sles11 and centos6. The former is more or less fixed, where the latter is likely needs to be flush-out (my task for the weekend).

          Back to your point: we are building everything with OJDK7 nowadays. There was a discussion about it on BIGTOP-1110, IIRC. Now I am looking at http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Oozie/ - please disregard sles11 failures for now - and it seems to be building just fine.

          since oozie enforces the requirement to 1.6

          Could you be more specific on how the aforementioned enforcement is implemented? And please pardon my ignorance with Oozie matters.

          Show
          cos Konstantin Boudnik added a comment - Bigtop trunk is far from flawless, as we have two troubled slaves: sles11 and centos6. The former is more or less fixed, where the latter is likely needs to be flush-out (my task for the weekend). Back to your point: we are building everything with OJDK7 nowadays. There was a discussion about it on BIGTOP-1110 , IIRC. Now I am looking at http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Oozie/ - please disregard sles11 failures for now - and it seems to be building just fine. since oozie enforces the requirement to 1.6 Could you be more specific on how the aforementioned enforcement is implemented? And please pardon my ignorance with Oozie matters.

            People

            • Assignee:
              oflebbe Olaf Flebbe
              Reporter:
              oflebbe Olaf Flebbe
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development