Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1580

Improve Bigtop Toolchain: Versioning of Packages

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 1.0.0
    • Component/s: build, debian, rpm
    • Labels:
      None

      Description

      Right now there is no build numbering (beyond -1 ) for different packaging attempts and bigtop releases.

      There is no workable way to increase build numbering in Bigtop. (At least this is the situation for ubuntu and debian, have not checked RPM).

      This leads to indentical package versions and buildnumbers for different bigtop builds when no package version upgrade has been made. This is not desirable , since it essentially disables package upgrades at all.

      There is a BIGTOP_BUILD_STAMP which could be used to introduce the needed semantics but sadly this BIGTOP_BUILD_STAMP is appended to the PKG_VERSION part of the numbering, not the RELEASE_VERSION part .

      I vote to move BIGTOP_BUILD_STAMP to the "right" place in the package-version: Appending to RELEASE_VERSION

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        Sorry for being a bit dense: but isn't current approach solves exactly what you are looking for? BIGTOP_BUILD_STAMP let you to modify the version of the package as the JIRA's title says, no? E.g. if one doing multiple iterations of a package then PKG_VERSION got increased. RELEASE_VERSION, on the other hand, should be used to something that is a property of the software Bigtop packages. May be I am missing something...?

        Show
        cos Konstantin Boudnik added a comment - Sorry for being a bit dense: but isn't current approach solves exactly what you are looking for? BIGTOP_BUILD_STAMP let you to modify the version of the package as the JIRA's title says, no? E.g. if one doing multiple iterations of a package then PKG_VERSION got increased. RELEASE_VERSION , on the other hand, should be used to something that is a property of the software Bigtop packages. May be I am missing something...?
        Hide
        rvs Roman Shaposhnik added a comment -

        I guess this makes two of us. Olaf Flebbe could you please elaborate?

        Show
        rvs Roman Shaposhnik added a comment - I guess this makes two of us. Olaf Flebbe could you please elaborate?
        Hide
        oflebbe Olaf Flebbe added a comment -

        Ok, let me explain it more carefully:

        In Bigtop packages.gradle enforces this naming and versioning scheme

        "name-${PKG_VERSION}${BIGTOP_BUILD_STAMP}-${RELEASE_VERSION}${RELEASE_DIST}....."
        
        • PKG_NAME is the upstream package name (hive, spark, ...)
        • PKG_VERSION the upstream version number (sometimes sanitized)
        • RELEASE_VERSION is de facto 1
        • RELEASE_DIST (only present in RPM builds) is do distinguish rhel6, rhel7, opensuse etc.

        This gives for instance

        hive-0.14${BIGTOP_BUILD_STAMP}-1
        

        It is policy in RPM and DEB build systems that every packaged version the release version should be incremented in order to differentiate different packaging attempts and to enable packaging upgrades.
        https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version
        http://www.rpm.org/max-rpm/ch-rpm-file-format.html

        For instance a fix for packaging hive should result for instance in an change of hive-0.14-1 to hive-0.14-2 , increasing the "release" part. A new upstream version 0.15 should result in resetting the "release" part 1 to hive-0.15-1

        The ${BIGTOP_BUILD_STAMP} cannot be used to mimic this behavior, because it is placed in the upstream version part, not in the release part of the version number. (The Release Part is not allowed to contain minus a dash "-")

        Unfortunately nobody from the bigtop developers increases RELEASE_VERSION if he/she/it fixes something. The -1 stays forever....

        In the end the DEBIAN / YUM repositories populated with consecutive builds from git the version are not suitable for automatic upgrades.

        I propose :

        Dropping the XXX_RELEASE

        Rational:

        • nobody uses it.
        • If used like the policies intended, handling is clumsy. Since it has to be increased it with every patch, it gives patch conflicts every time different changes to the same packages are developed.

        Use Version scheme name-${PKG_VERSION}-${BIGTOP_BUILD_STAMP}

        Rational:

        • Puts an build stamp into the release part of the version, as policies suggest.

        Use grade to synthesize consecutive build stamp numbers for ${BIGTOP_BUILD_STAMP}

        Rational:

        • In order to get an automagically increased RELEASE part if a package is rebuild, without messing with the upstream version. Enabling upgrades from repo without additional quirks.
        Show
        oflebbe Olaf Flebbe added a comment - Ok, let me explain it more carefully: In Bigtop packages.gradle enforces this naming and versioning scheme "name-${PKG_VERSION}${BIGTOP_BUILD_STAMP}-${RELEASE_VERSION}${RELEASE_DIST}....." PKG_NAME is the upstream package name (hive, spark, ...) PKG_VERSION the upstream version number (sometimes sanitized) RELEASE_VERSION is de facto 1 RELEASE_DIST (only present in RPM builds) is do distinguish rhel6, rhel7, opensuse etc. This gives for instance hive-0.14${BIGTOP_BUILD_STAMP}-1 It is policy in RPM and DEB build systems that every packaged version the release version should be incremented in order to differentiate different packaging attempts and to enable packaging upgrades. https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version http://www.rpm.org/max-rpm/ch-rpm-file-format.html For instance a fix for packaging hive should result for instance in an change of hive-0.14-1 to hive-0.14-2 , increasing the "release" part. A new upstream version 0.15 should result in resetting the "release" part 1 to hive-0.15-1 The ${BIGTOP_BUILD_STAMP } cannot be used to mimic this behavior, because it is placed in the upstream version part, not in the release part of the version number. (The Release Part is not allowed to contain minus a dash "-") Unfortunately nobody from the bigtop developers increases RELEASE_VERSION if he/she/it fixes something. The -1 stays forever.... In the end the DEBIAN / YUM repositories populated with consecutive builds from git the version are not suitable for automatic upgrades. I propose : Dropping the XXX_RELEASE Rational: nobody uses it. If used like the policies intended, handling is clumsy. Since it has to be increased it with every patch, it gives patch conflicts every time different changes to the same packages are developed. Use Version scheme name-${PKG_VERSION}-${BIGTOP_BUILD_STAMP } Rational: Puts an build stamp into the release part of the version, as policies suggest. Use grade to synthesize consecutive build stamp numbers for ${BIGTOP_BUILD_STAMP } Rational: In order to get an automagically increased RELEASE part if a package is rebuild, without messing with the upstream version. Enabling upgrades from repo without additional quirks.
        Hide
        rvs Roman Shaposhnik added a comment -

        Olaf Flebbe I like the simplicity of this. Andrew Bayer and Mark Grover can you chime in with some of your experience with the same?

        Show
        rvs Roman Shaposhnik added a comment - Olaf Flebbe I like the simplicity of this. Andrew Bayer and Mark Grover can you chime in with some of your experience with the same?
        Hide
        abayer Andrew Bayer added a comment -

        Yeah, build numbers or equivalent really shouldn't be in the package version, they should be in the release version.

        Show
        abayer Andrew Bayer added a comment - Yeah, build numbers or equivalent really shouldn't be in the package version, they should be in the release version.
        Hide
        oflebbe Olaf Flebbe added a comment -

        First round. Next would be to touch bigtop.mk, removing xxx_RELEASE and introducing BIGTOP_BUILD_STAMP

        Works for me...

        Show
        oflebbe Olaf Flebbe added a comment - First round. Next would be to touch bigtop.mk, removing xxx_RELEASE and introducing BIGTOP_BUILD_STAMP Works for me...
        Hide
        oflebbe Olaf Flebbe added a comment -

        Cannot change Assignment, BTW.

        Show
        oflebbe Olaf Flebbe added a comment - Cannot change Assignment, BTW.
        Hide
        cos Konstantin Boudnik added a comment -

        Assigned. Not sure why you can make the change yourself, really.

        Show
        cos Konstantin Boudnik added a comment - Assigned. Not sure why you can make the change yourself, really.
        Hide
        mgrover Mark Grover added a comment -

        Apologies for the delayed response. Good idea, Olaf.
        To ensure we have the same vocabulary, I have
        name-version-release which seems to be in line with yours well.

        I have a few questions and would appreciate if you could help answer them.

        1. If we simply use the BIGTOP_BUILD_STAMP variable to populate, what happens if we migrate our Jenkins server or clean out the job and the build numbers get reset? I'd like to think it would be good to have a stop gap field in the package's release identifier that we have complete control over in case the build stamp gets reset.
        2. Also, since we never release packages out of band of a bigtop release. Is it worth considering putting bigtop version in the release identifier? In fact, I would also vouch to keep RELEASE_VERSION which is almost always set to 1 before the bigtop version.

        That allows for us to things like a beta (for say, Bigtop 1.0) in which we may call it Bigtop 1.0 but with RELEASE_VERSION as '0' (which is how most packages would do it as).

        So, in my ideal scheme, hive, for example would look something like
        hive-0.14.1-1.bigtop0.9.158
        hive: name
        0.14.1: upstream version number
        1: RELEASE_VERSION
        bigtop0.9: bigtop release
        158: Build Stamp

        What do you think?

        Show
        mgrover Mark Grover added a comment - Apologies for the delayed response. Good idea, Olaf. To ensure we have the same vocabulary, I have name-version-release which seems to be in line with yours well. I have a few questions and would appreciate if you could help answer them. 1. If we simply use the BIGTOP_BUILD_STAMP variable to populate, what happens if we migrate our Jenkins server or clean out the job and the build numbers get reset? I'd like to think it would be good to have a stop gap field in the package's release identifier that we have complete control over in case the build stamp gets reset. 2. Also, since we never release packages out of band of a bigtop release. Is it worth considering putting bigtop version in the release identifier? In fact, I would also vouch to keep RELEASE_VERSION which is almost always set to 1 before the bigtop version. That allows for us to things like a beta (for say, Bigtop 1.0) in which we may call it Bigtop 1.0 but with RELEASE_VERSION as '0' (which is how most packages would do it as). So, in my ideal scheme, hive, for example would look something like hive-0.14.1-1.bigtop0.9.158 hive: name 0.14.1: upstream version number 1: RELEASE_VERSION bigtop0.9: bigtop release 158: Build Stamp What do you think?
        Hide
        oflebbe Olaf Flebbe added a comment -

        Management summary:
        This patch enables a release workflow, but does not implement the workflow.

        <too long>

        Regarding your second suggestion: This versioning scheme is not upgradeable since any "Beta" is old than any "Release", since comparing is left to right.

        Test yourself for instance on centos with rpmdevtools

        $ rpmdev-vercmp  hive-0.14-1.bigtop0.9-99 hive-0.14-0.bigtop1.0-100
        hive-0.14-1.bigtop0.9-99 > hive-0.14-0.bigtop1.0-100
        

        Regarding your last (unnumbered) suggestion:

        IMHO Placing a release number of the whole distro into the release is a good thing as anyone does understand where a package comes from.

        Regarding your first question:

        Using the jenkins build number was only intended as a possible strictly monotonic increasing token which can be inserted automatically. I am suggesting not to hard code it to a Jenkins variable rather to be able to use it. If you have to switch jenkins and want to be able to get increasing release numbers beyond this change, you are free to add an offset to the jenkins build number.

        Regarding the whole picture:

        To be honest: The patch suggested is to get my things done, quickly. It will not fix the world.

        Fixing the CI and Release process is more or less equivalent to chooseing a proper text token to be inserted into the Release portion of the versioning string.
        I tried to introduce the least friction as possible by misusing an already existing variable as a token to be inserted to the release.

        Let me describe my workflow by example:

        I did a full compile of Bigtop until it behaved somewhat reasonable, increasing BIGTOP_BUILD_STAMP at each full recompile, getting hadoop-2.4.1-1, hive-0.14-1 .. hadoop-2.4.1-2, hive-0.14-2 ...
        After l I decided to freeze and only to fix broken packages (for instance hive) I only recompile hive getting version numbers like hadop-2.4.1-2 and hive-0.14-3 .. hadop-2.4.1-2 and hive-0.14-4. After fixing all problems, I brand these set of packages as a repository, shipping it without changing anything.

        So at each step the user/test can update installation with an apt-get upgrade (And I presume it would work with yum update as well)

        Next Release I will start with a full recompile with "5" .

        Thats my workflow, the bigtop workflow in Jenkins could be:

        Doing a nightly full recompile on git master with

        grade clean
        BIGTOP_BUILD_STAMP="nightly-${BUILD_NUMBER}" grade apt / yum
        

        Doing a release is a build from a git branch bigtop0.8 running every checking

        grade clean
        BIGTOP_BUILD_STAMP="bigtop0.8-${BUILD_NUMBER}" grade apt / yum
        

        until stable.

        If you like to test it yourself. the patch has to be updated, since there is a problem with default numbering.

        And yes: We should name BIGTOP_BUILD_STAMP differently.

        Show
        oflebbe Olaf Flebbe added a comment - Management summary: This patch enables a release workflow, but does not implement the workflow. <too long> Regarding your second suggestion: This versioning scheme is not upgradeable since any "Beta" is old than any "Release", since comparing is left to right. Test yourself for instance on centos with rpmdevtools $ rpmdev-vercmp hive-0.14-1.bigtop0.9-99 hive-0.14-0.bigtop1.0-100 hive-0.14-1.bigtop0.9-99 > hive-0.14-0.bigtop1.0-100 Regarding your last (unnumbered) suggestion: IMHO Placing a release number of the whole distro into the release is a good thing as anyone does understand where a package comes from. Regarding your first question: Using the jenkins build number was only intended as a possible strictly monotonic increasing token which can be inserted automatically. I am suggesting not to hard code it to a Jenkins variable rather to be able to use it. If you have to switch jenkins and want to be able to get increasing release numbers beyond this change, you are free to add an offset to the jenkins build number. Regarding the whole picture: To be honest: The patch suggested is to get my things done, quickly. It will not fix the world. Fixing the CI and Release process is more or less equivalent to chooseing a proper text token to be inserted into the Release portion of the versioning string. I tried to introduce the least friction as possible by misusing an already existing variable as a token to be inserted to the release. Let me describe my workflow by example: I did a full compile of Bigtop until it behaved somewhat reasonable, increasing BIGTOP_BUILD_STAMP at each full recompile, getting hadoop-2.4.1-1, hive-0.14-1 .. hadoop-2.4.1-2, hive-0.14-2 ... After l I decided to freeze and only to fix broken packages (for instance hive) I only recompile hive getting version numbers like hadop-2.4.1-2 and hive-0.14-3 .. hadop-2.4.1-2 and hive-0.14-4. After fixing all problems, I brand these set of packages as a repository, shipping it without changing anything. So at each step the user/test can update installation with an apt-get upgrade (And I presume it would work with yum update as well) Next Release I will start with a full recompile with "5" . Thats my workflow, the bigtop workflow in Jenkins could be: Doing a nightly full recompile on git master with grade clean BIGTOP_BUILD_STAMP= "nightly-${BUILD_NUMBER}" grade apt / yum Doing a release is a build from a git branch bigtop0.8 running every checking grade clean BIGTOP_BUILD_STAMP= "bigtop0.8-${BUILD_NUMBER}" grade apt / yum until stable. If you like to test it yourself. the patch has to be updated, since there is a problem with default numbering. And yes: We should name BIGTOP_BUILD_STAMP differently.
        Hide
        rvs Roman Shaposhnik added a comment -

        After reading this, I'm comfortable with the change that Olaf Flebbe is proposing in his latest patch. Mark Grover it seems that none of what's being proposed in this patch is a deal breaker for you, correct?

        Show
        rvs Roman Shaposhnik added a comment - After reading this, I'm comfortable with the change that Olaf Flebbe is proposing in his latest patch. Mark Grover it seems that none of what's being proposed in this patch is a deal breaker for you, correct?
        Hide
        rvs Roman Shaposhnik added a comment -

        Olaf Flebbe could you, please, update the patch and also provide the above comment in BUILDING.txt (it would need to be created)?

        Show
        rvs Roman Shaposhnik added a comment - Olaf Flebbe could you, please, update the patch and also provide the above comment in BUILDING.txt (it would need to be created)?
        Hide
        oflebbe Olaf Flebbe added a comment -

        updated patch. Have to test it myself , since I changed it a little bit before uploading.

        Show
        oflebbe Olaf Flebbe added a comment - updated patch. Have to test it myself , since I changed it a little bit before uploading.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Added a BUILDING.txt file, fixed bigtop_toolchain/README.md removed obsolete jdk6.pp

        Show
        oflebbe Olaf Flebbe added a comment - Added a BUILDING.txt file, fixed bigtop_toolchain/README.md removed obsolete jdk6.pp
        Hide
        oflebbe Olaf Flebbe added a comment -

        please review

        Show
        oflebbe Olaf Flebbe added a comment - please review
        Hide
        cos Konstantin Boudnik added a comment -

        Per conversation on the dev@ marking this as PA

        Show
        cos Konstantin Boudnik added a comment - Per conversation on the dev@ marking this as PA
        Hide
        cos Konstantin Boudnik added a comment -

        I think it all is good. One small nit (which might be legit, but I guess I need more coffee still...)
        There's an empty entry in BOM_map
        BIGTOP_BUILD_STAMP: ''
        followed by this change

        -def final BIGTOP_BUILD_STAMP = System.getenv('BIGTOP_BUILD_STAMP') ?: BOM_map['BIGTOP_BUILD_STAMP']
        +def final BIGTOP_BUILD_STAMP = System.getenv('BIGTOP_BUILD_STAMP') ? System.getenv('BIGTOP_BUILD_STAMP') : 1
        

        Would it achieve the same effect if we just set the initial value of {[1}} to that key above and keep the assignment statement?

        Show
        cos Konstantin Boudnik added a comment - I think it all is good. One small nit (which might be legit, but I guess I need more coffee still...) There's an empty entry in BOM_map BIGTOP_BUILD_STAMP: '' followed by this change -def final BIGTOP_BUILD_STAMP = System.getenv('BIGTOP_BUILD_STAMP') ?: BOM_map['BIGTOP_BUILD_STAMP'] +def final BIGTOP_BUILD_STAMP = System.getenv('BIGTOP_BUILD_STAMP') ? System.getenv('BIGTOP_BUILD_STAMP') : 1 Would it achieve the same effect if we just set the initial value of {[1}} to that key above and keep the assignment statement?
        Hide
        cos Konstantin Boudnik added a comment -

        Olaf, you we can settle the question above I think it'd good to go.

        Show
        cos Konstantin Boudnik added a comment - Olaf, you we can settle the question above I think it'd good to go.
        Hide
        cos Konstantin Boudnik added a comment -

        Guys, shall we finish this for 1.0?

        Show
        cos Konstantin Boudnik added a comment - Guys, shall we finish this for 1.0?
        Hide
        cos Konstantin Boudnik added a comment -

        Anyone?

        Show
        cos Konstantin Boudnik added a comment - Anyone?
        Hide
        cos Konstantin Boudnik added a comment -

        Olaf Flebbe, if you can address my comment above, perhaps we can get in before 1.0? Thanks!

        Show
        cos Konstantin Boudnik added a comment - Olaf Flebbe , if you can address my comment above, perhaps we can get in before 1.0? Thanks!
        Hide
        oflebbe Olaf Flebbe added a comment -

        yep, there was an issue with this line. Have to refill my glass of wine first ...

        Show
        oflebbe Olaf Flebbe added a comment - yep, there was an issue with this line. Have to refill my glass of wine first ...
        Hide
        oflebbe Olaf Flebbe added a comment -

        Patch does not apply any more, have to rework it first.

        Show
        oflebbe Olaf Flebbe added a comment - Patch does not apply any more, have to rework it first.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Konstantin Boudnik Didn't realize the fine points of the ?: operator. Yep, will incorporate your change, updating soon

        Show
        oflebbe Olaf Flebbe added a comment - Konstantin Boudnik Didn't realize the fine points of the ?: operator. Yep, will incorporate your change, updating soon
        Hide
        oflebbe Olaf Flebbe added a comment -

        Patch updated. Former patch accidentially included other changes.

        Show
        oflebbe Olaf Flebbe added a comment - Patch updated. Former patch accidentially included other changes.
        Hide
        cos Konstantin Boudnik added a comment -

        +1 and have tested building both deb and rpms. Thanks!
        I will commit it in a couple of hours, unless you do it before

        Show
        cos Konstantin Boudnik added a comment - +1 and have tested building both deb and rpms. Thanks! I will commit it in a couple of hours, unless you do it before
        Hide
        cos Konstantin Boudnik added a comment -

        Pushed to the master. Thanks Olaf!

        Show
        cos Konstantin Boudnik added a comment - Pushed to the master. Thanks Olaf!

          People

          • Assignee:
            oflebbe Olaf Flebbe
            Reporter:
            oflebbe Olaf Flebbe
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development