Bigtop
  1. Bigtop
  2. BIGTOP-1381

Datafu and Spark .deb build is broken

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: build, debian, rpm
    • Labels:
      None
    • Environment:

      Ubuntu 14.04

      Description

      With the following command to build datafu package:
      >gradle datafu-deb
      I just got below error messages in the stage of creating source package:
      ...
      datafu-1.0.0/tools/jarjar-1.1.jar
      dpkg-buildpackage: source package datafu
      dpkg-buildpackage: source version 1.0.0-1
      dpkg-buildpackage: source distribution stable
      dpkg-buildpackage: source changed by Bigtop <dev@bigtop.apache.org>
      dpkg-source --before-build datafu-1.0.0
      dpkg-source: error: source package has two conflicting values - pig-udf-datafu and datafu
      dpkg-buildpackage: error: dpkg-source --before-build datafu-1.0.0 gave error exit status 255
      :datafu-sdeb FAILED

      FAILURE: Build failed with an exception.

      • Where:
        Script '/data/bigtop/packages.gradle' line: 254
      1. debnames.diff
        0.9 kB
        Olaf Flebbe
      2. BIGTOP-1381.patch
        6 kB
        Xiaomin Zhang

        Activity

        Hide
        Xiaomin Zhang added a comment -

        I found that packages.gradle ignores XXX_PKG_NAME variable defined in bigtop.mk. And datafu has different value for DATAFU_PKG_NAME and DATAFU_NAME. It seems this causes the build failure.
        Let me see if I can create a patch for packages.gradle

        Show
        Xiaomin Zhang added a comment - I found that packages.gradle ignores XXX_PKG_NAME variable defined in bigtop.mk. And datafu has different value for DATAFU_PKG_NAME and DATAFU_NAME. It seems this causes the build failure. Let me see if I can create a patch for packages.gradle
        Hide
        Wenwu Peng added a comment -

        is the same cause when run gradle datafu-rpm? if no, will file other jira to track.

        :datafu-download
        File /root/bigtop/dl/v1.0.0.tar.gz appears to be already downloaded. Exiting...
        :datafu-tar
        Nothing to do. Exiting...
        :datafu-srpm
        Nothing to do. Exiting...
        :datafu-rpm
        error: cannot open /root/bigtop/output/datafu/datafu-1.0.0-1.el6.src.rpm: No such file or directory
        :datafu-rpm FAILED

        FAILURE: Build failed with an exception.

        Show
        Wenwu Peng added a comment - is the same cause when run gradle datafu-rpm? if no, will file other jira to track. :datafu-download File /root/bigtop/dl/v1.0.0.tar.gz appears to be already downloaded. Exiting... :datafu-tar Nothing to do. Exiting... :datafu-srpm Nothing to do. Exiting... :datafu-rpm error: cannot open /root/bigtop/output/datafu/datafu-1.0.0-1.el6.src.rpm: No such file or directory :datafu-rpm FAILED FAILURE: Build failed with an exception.
        Hide
        Xiaomin Zhang added a comment -

        I tried datafu-rpm build with gradle, and it shows exactly the same issue.
        So I just update the title of this issue.

        Show
        Xiaomin Zhang added a comment - I tried datafu-rpm build with gradle, and it shows exactly the same issue. So I just update the title of this issue.
        Hide
        Xiaomin Zhang added a comment -

        There's an assignment to XXX_PKG_NAME in packages.gradle, but it should be directly loaded from bigtop.mk (if it's defined) instead:
        BOM_map[variable + '_PKG_NAME'] = "$

        {variable}

        _NAME"
        All components could be built except datafu, because it defines different value for DATAFU_NAME and DATAFU_PKG_NAME in bigtop.mk:
        DATAFU_NAME=datafu
        DATAFU_PKG_NAME=pig-udf-datafu
        And this is the root cause of the build failure.
        This patch tries to fix this issue by introducing another variable PKG_FINAL_NAME to behave the same as makefile
        Please review it. Thanks.

        Show
        Xiaomin Zhang added a comment - There's an assignment to XXX_PKG_NAME in packages.gradle, but it should be directly loaded from bigtop.mk (if it's defined) instead: BOM_map [variable + '_PKG_NAME'] = "$ {variable} _NAME" All components could be built except datafu, because it defines different value for DATAFU_NAME and DATAFU_PKG_NAME in bigtop.mk: DATAFU_NAME=datafu DATAFU_PKG_NAME=pig-udf-datafu And this is the root cause of the build failure. This patch tries to fix this issue by introducing another variable PKG_FINAL_NAME to behave the same as makefile Please review it. Thanks.
        Hide
        Xiaomin Zhang added a comment -

        patch is submitted as attached. Could anyone review it?
        Thanks.

        Show
        Xiaomin Zhang added a comment - patch is submitted as attached. Could anyone review it? Thanks.
        Hide
        Olaf Flebbe added a comment - - edited

        AFAIK it is a debian convention that the name of the source package has to be the same as the name at the upstream provider.

        So the debian/control file is incorrect (and spark too) appending patches proposed by me

        Show
        Olaf Flebbe added a comment - - edited AFAIK it is a debian convention that the name of the source package has to be the same as the name at the upstream provider. So the debian/control file is incorrect (and spark too) appending patches proposed by me
        Hide
        Olaf Flebbe added a comment -

        Patches to the control files

        Show
        Olaf Flebbe added a comment - Patches to the control files
        Hide
        Olaf Flebbe added a comment -

        Updated patch

        Show
        Olaf Flebbe added a comment - Updated patch
        Hide
        Konstantin Boudnik added a comment -

        Olaf Flebbe, could you please combine the two separate patches together?
        Also, looking at the spark patch: just out of curiosity - why the name of the source package will break the debian build?

        Show
        Konstantin Boudnik added a comment - Olaf Flebbe , could you please combine the two separate patches together? Also, looking at the spark patch: just out of curiosity - why the name of the source package will break the debian build?
        Hide
        Olaf Flebbe added a comment -

        Konstantin Boudnik : The source package name and the name of the package in the changelog must be the same. This is enforced by debuild

        See Debian Policies chapter 4.4 format of changelog.
        https://www.debian.org/doc/debian-policy/ch-source.html

        Since the package name in debian/changelog is autogenerated from the bigtop package name (spark) it cannot be named spark-core in debian/control.

        I am curious why this worked on ubuntu.

        Uploading updated patch...

        Show
        Olaf Flebbe added a comment - Konstantin Boudnik : The source package name and the name of the package in the changelog must be the same. This is enforced by debuild See Debian Policies chapter 4.4 format of changelog. https://www.debian.org/doc/debian-policy/ch-source.html Since the package name in debian/changelog is autogenerated from the bigtop package name (spark) it cannot be named spark-core in debian/control. I am curious why this worked on ubuntu. Uploading updated patch...
        Hide
        Olaf Flebbe added a comment -

        Use consistent name for source package name and changelog

        Show
        Olaf Flebbe added a comment - Use consistent name for source package name and changelog
        Hide
        Konstantin Boudnik added a comment -

        Makes sense. +1 from, but I'd like a second opinion from ppl with better deb knowledge to look at it from the backward compatibility perspective. Sean Mackrory, could you take a look as well?

        BTW, I am pretty sure it isn't a gradle build specific problem, so I will rename the ticket.

        Show
        Konstantin Boudnik added a comment - Makes sense. +1 from, but I'd like a second opinion from ppl with better deb knowledge to look at it from the backward compatibility perspective. Sean Mackrory , could you take a look as well? BTW, I am pretty sure it isn't a gradle build specific problem, so I will rename the ticket.
        Hide
        Peter Linnell added a comment -

        @Cos, where is the "backwards compatibility" issue ? Upgrading from older versions ?

        Show
        Peter Linnell added a comment - @Cos, where is the "backwards compatibility" issue ? Upgrading from older versions ?
        Hide
        Konstantin Boudnik added a comment -

        Yup, should've been clearer. Thanks!

        Show
        Konstantin Boudnik added a comment - Yup, should've been clearer. Thanks!
        Hide
        Konstantin Boudnik added a comment -

        I presume this isn't going to 0.8.0 as I have seen this issue in the offical build. Moving to 0.9.0

        Show
        Konstantin Boudnik added a comment - I presume this isn't going to 0.8.0 as I have seen this issue in the offical build. Moving to 0.9.0
        Hide
        Olaf Flebbe added a comment -

        Konstantin Boudnik Could you please apply this patch? This breaks Debian badly. It does not change binary compatibility or anything user related, since it is only the names of the source packages are affected.

        Show
        Olaf Flebbe added a comment - Konstantin Boudnik Could you please apply this patch? This breaks Debian badly. It does not change binary compatibility or anything user related, since it is only the names of the source packages are affected.
        Hide
        Olaf Flebbe added a comment - - edited

        Ooops, someone else have patched this issue differently ...

        Show
        Olaf Flebbe added a comment - - edited Ooops, someone else have patched this issue differently ...
        Hide
        Olaf Flebbe added a comment -

        It seems to be solved differently. Fine for me. Resolving Issue

        Show
        Olaf Flebbe added a comment - It seems to be solved differently. Fine for me. Resolving Issue
        Hide
        Olaf Flebbe added a comment -

        not neccessary any more, solved differently

        Show
        Olaf Flebbe added a comment - not neccessary any more, solved differently

          People

          • Assignee:
            Olaf Flebbe
            Reporter:
            Xiaomin Zhang
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development