Bigtop
  1. Bigtop
  2. BIGTOP-24

Errors downloading tarballs from dist.apache.org via curl

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: General
    • Labels:
      None

      Description

      We're seeing weird problems with downloading tarballs from dist.apache.org (and possibly other sites) barfing partway through. I'm going to try to use a dedicated mirror rather than dist.a.o so that we get more reliable d/ls.

      1. BIGTOP-24.diff
        3 kB
        Andrew Bayer

        Activity

        Hide
        Andrew Bayer added a comment -

        Oh, definitely agreed - but the immediate problem here wasn't the internet connection, it was that dist.apache.org kept crapping out during downloads, while I had no problems downloading from anywhere else. =) And yeah, we should open another JIRA for wrapping curl calls in retry logic.

        Show
        Andrew Bayer added a comment - Oh, definitely agreed - but the immediate problem here wasn't the internet connection, it was that dist.apache.org kept crapping out during downloads, while I had no problems downloading from anywhere else. =) And yeah, we should open another JIRA for wrapping curl calls in retry logic.
        Hide
        Andrei Savu added a comment -

        A reliable mirror will not make the internet connection reliable but I guess the solution is good enough for now.

        Show
        Andrei Savu added a comment - A reliable mirror will not make the internet connection reliable but I guess the solution is good enough for now.
        Hide
        Andrew Bayer added a comment -

        We do need a retry loop added, but for right now, I'm fine with just using a consistent, functional mirror, so that we can actually build in the first place. =) I've tested and all the tarballs do download via apache.osuosl.org.

        Show
        Andrew Bayer added a comment - We do need a retry loop added, but for right now, I'm fine with just using a consistent, functional mirror, so that we can actually build in the first place. =) I've tested and all the tarballs do download via apache.osuosl.org.
        Hide
        Andrei Savu added a comment -

        We experienced a similar problem in Whirr. I suggest that rather than changing the mirror a more permanent fix would be to add a retry loop around curl (it seems like it doesn't automatically retry partial downloads).

        Show
        Andrei Savu added a comment - We experienced a similar problem in Whirr. I suggest that rather than changing the mirror a more permanent fix would be to add a retry loop around curl (it seems like it doesn't automatically retry partial downloads).
        Hide
        Andrew Bayer added a comment -

        I eyeballed all the tarballs - I'm in the process of testing pig 0.9.0 for BIGTOP-23 right now, but once I verify that, I'll do a full build of everything to make sure they fetch.

        Show
        Andrew Bayer added a comment - I eyeballed all the tarballs - I'm in the process of testing pig 0.9.0 for BIGTOP-23 right now, but once I verify that, I'll do a full build of everything to make sure they fetch.
        Hide
        Bruno Mahé added a comment -

        +1 LGTM.provided bigtop has been able to pull all the tarballs without issue with that mirror

        Show
        Bruno Mahé added a comment - +1 LGTM.provided bigtop has been able to pull all the tarballs without issue with that mirror
        Hide
        Andrew Bayer added a comment -

        Patch changes the download location for the various Apache tarballs to use the Makefile-defined APACHE_MIRROR variable, which I've also changed to default to use apache.osuosl.org, which I generally find to be faster than ibiblio.

        I should note that this approach could cause problems if a Bigtop build is trying to find a version of one of the component projects that is no longer available in the main Apache download/mirror area, i.e., a particularly old version. But for now, I don't think that's an issue we really need to worry about, and we can cross that bridge if/when we get to it.

        Show
        Andrew Bayer added a comment - Patch changes the download location for the various Apache tarballs to use the Makefile-defined APACHE_MIRROR variable, which I've also changed to default to use apache.osuosl.org, which I generally find to be faster than ibiblio. I should note that this approach could cause problems if a Bigtop build is trying to find a version of one of the component projects that is no longer available in the main Apache download/mirror area, i.e., a particularly old version. But for now, I don't think that's an issue we really need to worry about, and we can cross that bridge if/when we get to it.

          People

          • Assignee:
            Andrew Bayer
            Reporter:
            Andrew Bayer
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development