Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1917

Simplify gradle creating apt/yum repositories for better CI

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: build
    • Labels:
      None

      Description

      I propose to simplify the packages.gradle in order to better support the new docker based CI .

      So all the target-yum / target-apt tagets have been removed and a simple repository creation is done by gradle yum/apt which can be run after all packages have been done. Complexity of gradle is removed as a side effect.

      In the CI the only thing to be done is to schedule a "gradle yum/apt" .

      Maybe we have to discuss to remove the dependencies at all in order to generate anything if something failed before.

      1. BIGTOP-1917.1.patch
        5 kB
        Olaf Flebbe
      2. BIGTOP-1917.2.patch
        5 kB
        Olaf Flebbe

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        A couple of comments:

        • any reason to move apt and yum tasks code all the way to the bottom of the file from where they were? Makes them kinda separated from the rest of their dependencies... But it might be my personal preference
        • It seems that new tasks do not place a success flag file eg
          touchTargetFile(BOM_map["${variable}_TARGET_YUM"])
        • in the new version there's no way anymore to build apt/yum repo just for a few components explicitly. Is it intentional?
        Show
        cos Konstantin Boudnik added a comment - A couple of comments: any reason to move apt and yum tasks code all the way to the bottom of the file from where they were? Makes them kinda separated from the rest of their dependencies... But it might be my personal preference It seems that new tasks do not place a success flag file eg touchTargetFile(BOM_map[ "${variable}_TARGET_YUM" ]) in the new version there's no way anymore to build apt/yum repo just for a few components explicitly. Is it intentional?
        Hide
        oflebbe Olaf Flebbe added a comment -

        First I have to admit, that I uploaded an outdated version of this patch. I accidently did not remove the dependencies to "rpm/deb".

        Perhaps this answers your comments.

        First, the new targets are independent from all the targets above, but logically last. So I placed them at the bottom.

        Second, I see no point in having a sucessflag, since repository creation is very fast and can be done multiple times without problems. No need to block it, and furthermore if one likes to force it, one has to remove dotfiles ...

        Last, ... yes that was a bug with .1. ... now it is possible.

        Show
        oflebbe Olaf Flebbe added a comment - First I have to admit, that I uploaded an outdated version of this patch. I accidently did not remove the dependencies to "rpm/deb". Perhaps this answers your comments. First, the new targets are independent from all the targets above, but logically last. So I placed them at the bottom. Second, I see no point in having a sucessflag, since repository creation is very fast and can be done multiple times without problems. No need to block it, and furthermore if one likes to force it, one has to remove dotfiles ... Last, ... yes that was a bug with .1. ... now it is possible.
        Hide
        evans_ye Evans Ye added a comment -

        I'd be +1 for removing successful tag since once I've encountered an issue that is the repo won't update when I rebuild packages.

        Show
        evans_ye Evans Ye added a comment - I'd be +1 for removing successful tag since once I've encountered an issue that is the repo won't update when I rebuild packages.
        Hide
        cos Konstantin Boudnik added a comment -

        I see. I still fail to see how I can build a separate hadoop-apt now? Could you please clarify? Other than that the patch look good! Thank you!

        Show
        cos Konstantin Boudnik added a comment - I see. I still fail to see how I can build a separate hadoop-apt now? Could you please clarify? Other than that the patch look good! Thank you!
        Hide
        oflebbe Olaf Flebbe added a comment - - edited

        gradle hadoop-deb apt

        Show
        oflebbe Olaf Flebbe added a comment - - edited gradle hadoop-deb apt
        Hide
        cos Konstantin Boudnik added a comment -

        So, what are we saving here then? Do you think it might be cleaner to have hadoop-apt that will depends on hadoop-deb? It looks like a bit better UX, no?

        Show
        cos Konstantin Boudnik added a comment - So, what are we saving here then? Do you think it might be cleaner to have hadoop-apt that will depends on hadoop-deb? It looks like a bit better UX, no?
        Hide
        oflebbe Olaf Flebbe added a comment -

        The point is to assembly all the packages and build the repository once , not rebuilding repository for every package, which may break consistency in the repo.

        The point dot tag files are implementing this workflow introduce a shortcut if already installed. But unfortunately the workflow breaks down often (see comment from Evans Ye ) with the need to do a rm build/*/.apt since the repository does not update any more. This is plain ugly. I would call the build target a lot better UX.

        I propose to remove the dot files and remove dozens of unnecessary targets, overwhelming random users.

        Building a random package is a sane task, building (and signing!) a repository with all available packages is a task, but "not build a random package and install this single package in a repository." The patch to sign the packages is pending... signing a repository may be a manual step , since you'll have to provide a pass phrase.

        Show
        oflebbe Olaf Flebbe added a comment - The point is to assembly all the packages and build the repository once , not rebuilding repository for every package, which may break consistency in the repo. The point dot tag files are implementing this workflow introduce a shortcut if already installed. But unfortunately the workflow breaks down often (see comment from Evans Ye ) with the need to do a rm build/*/.apt since the repository does not update any more. This is plain ugly. I would call the build target a lot better UX. I propose to remove the dot files and remove dozens of unnecessary targets, overwhelming random users. Building a random package is a sane task, building (and signing!) a repository with all available packages is a task, but "not build a random package and install this single package in a repository." The patch to sign the packages is pending... signing a repository may be a manual step , since you'll have to provide a pass phrase.
        Hide
        cos Konstantin Boudnik added a comment -

        Thanks for the explanation - makes quite a bit of sense!

        Show
        cos Konstantin Boudnik added a comment - Thanks for the explanation - makes quite a bit of sense!
        Hide
        cos Konstantin Boudnik added a comment -

        +1 LGTM

        Show
        cos Konstantin Boudnik added a comment - +1 LGTM
        Hide
        oflebbe Olaf Flebbe added a comment -

        I am away from keyboard for a few days starting right now ...

        Show
        oflebbe Olaf Flebbe added a comment - I am away from keyboard for a few days starting right now ...
        Hide
        cos Konstantin Boudnik added a comment -

        I will commit it in a bit then, don't worry. Thanks!

        Show
        cos Konstantin Boudnik added a comment - I will commit it in a bit then, don't worry. Thanks!
        Hide
        cos Konstantin Boudnik added a comment -

        Pushed to the master. Thanks Olaf!

        Show
        cos Konstantin Boudnik added a comment - Pushed to the master. Thanks Olaf!

          People

          • Assignee:
            oflebbe Olaf Flebbe
            Reporter:
            oflebbe Olaf Flebbe
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development