Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-5550

Simplify binding artifacts and build metadata

    Details

      Description

      Right now we build special tarballs for the perl and wrapped python bindings of the messaging api in the release.sh script.

      First, what motivates this? Wouldn't be simpler from a packaging standpoint to simply build these when you build the C++ parts of the messaging api, using just the one cpp source tarball?

      Second, this logic belongs in the cpp subtree, not in the release.sh script. We can call into any "make special dist variant" targets we may need from the release script.

        Activity

        Hide
        justi9 Justin Ross added a comment -

        Darryl, any comments? I'm sure you have packages that depend on these. I want to remove them, because they are strictly subsets of the cpp tarball.

        Show
        justi9 Justin Ross added a comment - Darryl, any comments? I'm sure you have packages that depend on these. I want to remove them, because they are strictly subsets of the cpp tarball.
        Hide
        justi9 Justin Ross added a comment -

        A few more thoughts:

        The release script does this:

        if [ "PERL" == "$PERL" ]; then
          mkdir qpid-${VER}/perl-qpid-${VER}
          cp qpid-${VER}/cpp/bindings/qpid/perl/perl.i \
             qpid-${VER}/cpp/bindings/qpid/perl/LICENSE \
             qpid-${VER}/cpp/bindings/qpid/perl/Makefile.PL \
             qpid-${VER}/cpp/bindings/qpid/perl/t/*.t \
             qpid-${VER}/perl-qpid-${VER}
        ...
        

        That last bit flattens the structure of the perl tree. Is that for a reason? It strikes me as fragile and unnecessary.

        If we must have these extra tarballs (I'm waiting for the rationale), they should be as close as possible to a simple subtree copy.

        Show
        justi9 Justin Ross added a comment - A few more thoughts: The release script does this: if [ "PERL" == "$PERL" ]; then mkdir qpid-${VER}/perl-qpid-${VER} cp qpid-${VER}/cpp/bindings/qpid/perl/perl.i \ qpid-${VER}/cpp/bindings/qpid/perl/LICENSE \ qpid-${VER}/cpp/bindings/qpid/perl/Makefile.PL \ qpid-${VER}/cpp/bindings/qpid/perl/t/*.t \ qpid-${VER}/perl-qpid-${VER} ... That last bit flattens the structure of the perl tree. Is that for a reason? It strikes me as fragile and unnecessary. If we must have these extra tarballs (I'm waiting for the rationale), they should be as close as possible to a simple subtree copy.
        Hide
        mcpierce Darryl L. Pierce added a comment - - edited

        The goal there was to move the files needed by upstream packaging sources into a directory with a name that matches the release. Both Debian and Fedora expect, after exploding the sources, to find the top-level directory be named:

        ${PROJECT}-${VERSION}

        The above could be simplified to just be:

        cp -r qpid-${VER}/cpp/bindings/perl/* qpid-${VER}/perl-qpid-${VER}

        rather than explicitly copying each individual file. That would meet the simple subtree requirement.

        Regarding a rationale for having them, it makes it simpler for packages to work with only the bindings if there's a source bundle that contains just what's needed to build it; i.e., the Swig descriptor, license and language-only sources. Ted and I had talked about this a few years ago, that there's enough of a separation between the language bindings and the core C++ codebase that we could conceivably decouple releasing bindings from the core C++ code.

        Show
        mcpierce Darryl L. Pierce added a comment - - edited The goal there was to move the files needed by upstream packaging sources into a directory with a name that matches the release. Both Debian and Fedora expect, after exploding the sources, to find the top-level directory be named: ${PROJECT}-${VERSION} The above could be simplified to just be: cp -r qpid-${VER}/cpp/bindings/perl/* qpid-${VER}/perl-qpid-${VER} rather than explicitly copying each individual file. That would meet the simple subtree requirement. Regarding a rationale for having them, it makes it simpler for packages to work with only the bindings if there's a source bundle that contains just what's needed to build it; i.e., the Swig descriptor, license and language-only sources. Ted and I had talked about this a few years ago, that there's enough of a separation between the language bindings and the core C++ codebase that we could conceivably decouple releasing bindings from the core C++ code.
        Hide
        mcpierce Darryl L. Pierce added a comment -

        I would be all for moving the tarball building to the CMake environment rather than in the release.sh script, definitely.

        Show
        mcpierce Darryl L. Pierce added a comment - I would be all for moving the tarball building to the CMake environment rather than in the release.sh script, definitely.
        Hide
        justi9 Justin Ross added a comment -

        As to packaging, it still seems to me that it would be simpler, given our current architecture, to build each binding as a subpackage of the top-level package that provides the core API implementation.

        Anyhow, yes, it would be great to move these into cmake, and to simplify them a bit on the way.

        Show
        justi9 Justin Ross added a comment - As to packaging, it still seems to me that it would be simpler, given our current architecture, to build each binding as a subpackage of the top-level package that provides the core API implementation. Anyhow, yes, it would be great to move these into cmake, and to simplify them a bit on the way.
        Hide
        mcpierce Darryl L. Pierce added a comment -

        From a packaging perspective, trying to build them all in one package would be a headache since, if a bug is reported in any one part, we would have to rebuild the world to put out even a small fix.

        Show
        mcpierce Darryl L. Pierce added a comment - From a packaging perspective, trying to build them all in one package would be a headache since, if a bug is reported in any one part, we would have to rebuild the world to put out even a small fix.
        Hide
        justi9 Justin Ross added a comment -

        That would make sense if these represented a substantial amount of code, but they are just swig wrappers. What's a pain in the ass is maintaining redundant-but-not-quite the same tarballs in the release infrastructure.

        I've spent some time looking at this, and I've decided to escalate this one. This isn't worth it. I'm removing the perl-qpid and python-qpid_messaging synthetic tarballs. The perl and python packaging will have to adjust to using the authoritative tarball directly.

        Show
        justi9 Justin Ross added a comment - That would make sense if these represented a substantial amount of code, but they are just swig wrappers. What's a pain in the ass is maintaining redundant-but-not-quite the same tarballs in the release infrastructure. I've spent some time looking at this, and I've decided to escalate this one. This isn't worth it. I'm removing the perl-qpid and python-qpid_messaging synthetic tarballs. The perl and python packaging will have to adjust to using the authoritative tarball directly.
        Hide
        justi9 Justin Ross added a comment -

        I should address a couple points above.

        • You mention the possibility of decoupling the binding source. That is not the case now, and it's unlikely to ever be the case.
        • Having the source tarball match the package name is only a convenience (for instance, it matches the default %prepare behavior in rpm). That is not sufficient reason to generate these frankenballs.
        • If you want to simulate these reduced tarballs, it should be done outside of the qpid project.
        Show
        justi9 Justin Ross added a comment - I should address a couple points above. You mention the possibility of decoupling the binding source. That is not the case now, and it's unlikely to ever be the case. Having the source tarball match the package name is only a convenience (for instance, it matches the default %prepare behavior in rpm). That is not sufficient reason to generate these frankenballs. If you want to simulate these reduced tarballs, it should be done outside of the qpid project.
        Hide
        gemmellr Robbie Gemmell added a comment -

        Just thought I'd chime in to say that I would echo Justin's comments.

        We had a related discussion on the list years ago, indicating we should only maintain a single source archive [for a given component]; if for none of the other good reasons, then simply because we don't test them all equally. I mentioned this when there was indication of creating some of these additional artifacts originally, and I think they subsequently got removed though have since come back. There is also the confusion around what bits go with what else: do people use python-qpid_messaging-0.26.tar.gz or qpid-python-0.26.tar.gz for example? (I know the answer is obvious if you inspect them, but it isn't from the name). Once you download python-qpid_messaging-0.26.tar.gz, how do you use it without also downloading one of the two other source archives that also happen to include the same content?

        We obviously still have multiple source artifacts for particular components, since we have the 'full release' as well as various other source archives, so these aren't alone in that. However, we are currently working to overhaul the build system for the current Java tree to allow later splitting it up and moving towards the major components being independently releasable. This could then also help us in stopping having multiple artifacts containing the elements of the current C++ tree as well for example.

        Show
        gemmellr Robbie Gemmell added a comment - Just thought I'd chime in to say that I would echo Justin's comments. We had a related discussion on the list years ago, indicating we should only maintain a single source archive [for a given component] ; if for none of the other good reasons, then simply because we don't test them all equally. I mentioned this when there was indication of creating some of these additional artifacts originally, and I think they subsequently got removed though have since come back. There is also the confusion around what bits go with what else: do people use python-qpid_messaging-0.26.tar.gz or qpid-python-0.26.tar.gz for example? (I know the answer is obvious if you inspect them, but it isn't from the name). Once you download python-qpid_messaging-0.26.tar.gz, how do you use it without also downloading one of the two other source archives that also happen to include the same content? We obviously still have multiple source artifacts for particular components, since we have the 'full release' as well as various other source archives, so these aren't alone in that. However, we are currently working to overhaul the build system for the current Java tree to allow later splitting it up and moving towards the major components being independently releasable. This could then also help us in stopping having multiple artifacts containing the elements of the current C++ tree as well for example.

          People

          • Assignee:
            mcpierce Darryl L. Pierce
            Reporter:
            justi9 Justin Ross
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development