Bigtop
  1. Bigtop
  2. BIGTOP-509

make all is failing because flume-1.0.0-incubating.tar.gz does not exist in APACHE_MIRROR

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: general
    • Labels:
    • Environment:

      all

      Description

      bigtop.mk is attempting to download http://apache.osuosl.org/flume/flume-1.0.0-incubating//flume-1.0.0-incubating.tar.gz

      However, it looks like flume-1.0.0 has been removed from the mirror site. Instead there is http://apache.osuosl.org/flume/flume-1.1.0-incubating//flume-1.1.0-incubating.tar.gz

      I have not submitted a patch as flume-1.1.0 has not been certified/tested on bigtop-0.3.0.

        Activity

        Hide
        Roman Shaposhnik added a comment -

        I think the real fix for these types of issues would be to enhance our package.mk logic and try archive.apache.org IFF the mirror fails. Note that http://archive.apache.org/dist/flume/flume-1.0.0-incubating//flume-1.0.0-incubating.tar.gz is still there.

        Show
        Roman Shaposhnik added a comment - I think the real fix for these types of issues would be to enhance our package.mk logic and try archive.apache.org IFF the mirror fails. Note that http://archive.apache.org/dist/flume/flume-1.0.0-incubating//flume-1.0.0-incubating.tar.gz is still there.
        Hide
        Roman Shaposhnik added a comment -

        Ron, assigning to you – feel free to take a stab at it!

        Show
        Roman Shaposhnik added a comment - Ron, assigning to you – feel free to take a stab at it!
        Hide
        Ron Bogdanoff added a comment -

        Ok will do!

        Show
        Ron Bogdanoff added a comment - Ok will do!
        Hide
        Ron Bogdanoff added a comment -

        Attached is BIGTOP-509.patch.diff

        This is a simple fix that will check that the component tar.gz exists in the mirror site. If it does not, it will use the apache archive site instead for the download of the component

        This is a simple patch that will work for now - I will improve on it based of feedback from everyone (e.g. also check archive site and abort if component does not exists there too, provide mechanizm to check several sites in some order, etc.)

        Changed...
        bigtop.mk
        added *_DOWNLOAD_PATH var for each component to make maintenance easier
        added *_ARCHIVE var for each component

        package.mk
        after setting DOWNLOAD_URL, check it for HTTP 200 using curl --head (this only returns header, not content - good for just checking HTTP return code)
        if not HTTP 200, replace DOWNLOAD_URL with the ARCHIVE url

        NOTE: if the tar.gz also does not exist in ARCHIVE, the make will fail as it did before. Will improve on this in another patch, but, for now, using ARCHIVE as a fallback to the MIRROR will be good enough!!

        Show
        Ron Bogdanoff added a comment - Attached is BIGTOP-509 .patch.diff This is a simple fix that will check that the component tar.gz exists in the mirror site. If it does not, it will use the apache archive site instead for the download of the component This is a simple patch that will work for now - I will improve on it based of feedback from everyone (e.g. also check archive site and abort if component does not exists there too, provide mechanizm to check several sites in some order, etc.) Changed... bigtop.mk added *_DOWNLOAD_PATH var for each component to make maintenance easier added *_ARCHIVE var for each component package.mk after setting DOWNLOAD_URL, check it for HTTP 200 using curl --head (this only returns header, not content - good for just checking HTTP return code) if not HTTP 200, replace DOWNLOAD_URL with the ARCHIVE url NOTE: if the tar.gz also does not exist in ARCHIVE, the make will fail as it did before. Will improve on this in another patch, but, for now, using ARCHIVE as a fallback to the MIRROR will be good enough!!
        Hide
        Ron Bogdanoff added a comment -

        Attched suggested patch

        Show
        Ron Bogdanoff added a comment - Attched suggested patch
        Hide
        Ron Bogdanoff added a comment -

        attached suggested patch

        Show
        Ron Bogdanoff added a comment - attached suggested patch
        Hide
        Roman Shaposhnik added a comment -

        Patch looks ok, but it seems that it will introduce one extra URL request on each attempted download. Won't it be more efficient to do something like this in package.mk:

        ... (cd $(DL_DIR) && curl --retry 5 -# -L -k -o $($(PKG)_TARBALL_DST) $($(PKG)_DOWNLOAD_URL) || 
                             curl --retry 5 -# -L -k -o $($(PKG)_TARBALL_DST) $($(PKG)_ARCHIVE_DOWNLOAD_URL))
        
        Show
        Roman Shaposhnik added a comment - Patch looks ok, but it seems that it will introduce one extra URL request on each attempted download. Won't it be more efficient to do something like this in package.mk: ... (cd $(DL_DIR) && curl --retry 5 -# -L -k -o $($(PKG)_TARBALL_DST) $($(PKG)_DOWNLOAD_URL) || curl --retry 5 -# -L -k -o $($(PKG)_TARBALL_DST) $($(PKG)_ARCHIVE_DOWNLOAD_URL))
        Hide
        Ron Bogdanoff added a comment -

        Hi Roman,

        I thought about putting code up in the # Download part of package.mk as well. The reason I put the code further down in package.mk where I test $(2)_DOWNLOAD_URL and replace its value if necessary, is DOWNLOAD_URL is also used in # Helper targets, specifically in $(1)-info: - if I replace the value of $(2)_DOWNLOAD_URL, and you do a make info, the " Will download from URL: $$($(2)_DOWNLOAD_URL) " will display the true URL where the tar will really be downloaded from. If I had changed just the #Download part of package.mk, the info target will report the incorrect download URL in the event we need to get the tar from ARCHIVE.

        True, there is an extra head only http request, but I thought it was a small price to pay to make $(2)_DOWNLOAD_URL contain the real url we will download from so other part of the code can still reference it and get the correct value.

        In any event, please let me know which you prefer I can make the change you suggest if you like

        Regards
        Ron

        Show
        Ron Bogdanoff added a comment - Hi Roman, I thought about putting code up in the # Download part of package.mk as well. The reason I put the code further down in package.mk where I test $(2)_DOWNLOAD_URL and replace its value if necessary, is DOWNLOAD_URL is also used in # Helper targets, specifically in $(1)-info: - if I replace the value of $(2)_DOWNLOAD_URL, and you do a make info, the " Will download from URL: $$($(2)_DOWNLOAD_URL) " will display the true URL where the tar will really be downloaded from. If I had changed just the #Download part of package.mk, the info target will report the incorrect download URL in the event we need to get the tar from ARCHIVE. True, there is an extra head only http request, but I thought it was a small price to pay to make $(2)_DOWNLOAD_URL contain the real url we will download from so other part of the code can still reference it and get the correct value. In any event, please let me know which you prefer I can make the change you suggest if you like Regards Ron
        Hide
        Roman Shaposhnik added a comment -

        Ron, makes perfect sense! I committed your original patch to the trunk.

        Show
        Roman Shaposhnik added a comment - Ron, makes perfect sense! I committed your original patch to the trunk.

          People

          • Assignee:
            Ron Bogdanoff
            Reporter:
            Ron Bogdanoff
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development