Bigtop
  1. Bigtop
  2. BIGTOP-24

Errors downloading tarballs from dist.apache.org via curl

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: general
    • Labels:
      None

      Description

      We're seeing weird problems with downloading tarballs from dist.apache.org (and possibly other sites) barfing partway through. I'm going to try to use a dedicated mirror rather than dist.a.o so that we get more reliable d/ls.

      1. BIGTOP-24.diff
        3 kB
        Andrew Bayer

        Activity

        Andrew Bayer created issue -
        Andrew Bayer made changes -
        Field Original Value New Value
        Attachment BIGTOP-24.diff [ 12490870 ]
        Hide
        Andrew Bayer added a comment -

        Patch changes the download location for the various Apache tarballs to use the Makefile-defined APACHE_MIRROR variable, which I've also changed to default to use apache.osuosl.org, which I generally find to be faster than ibiblio.

        I should note that this approach could cause problems if a Bigtop build is trying to find a version of one of the component projects that is no longer available in the main Apache download/mirror area, i.e., a particularly old version. But for now, I don't think that's an issue we really need to worry about, and we can cross that bridge if/when we get to it.

        Show
        Andrew Bayer added a comment - Patch changes the download location for the various Apache tarballs to use the Makefile-defined APACHE_MIRROR variable, which I've also changed to default to use apache.osuosl.org, which I generally find to be faster than ibiblio. I should note that this approach could cause problems if a Bigtop build is trying to find a version of one of the component projects that is no longer available in the main Apache download/mirror area, i.e., a particularly old version. But for now, I don't think that's an issue we really need to worry about, and we can cross that bridge if/when we get to it.
        Hide
        Bruno Mahé added a comment -

        +1 LGTM.provided bigtop has been able to pull all the tarballs without issue with that mirror

        Show
        Bruno Mahé added a comment - +1 LGTM.provided bigtop has been able to pull all the tarballs without issue with that mirror
        Hide
        Andrew Bayer added a comment -

        I eyeballed all the tarballs - I'm in the process of testing pig 0.9.0 for BIGTOP-23 right now, but once I verify that, I'll do a full build of everything to make sure they fetch.

        Show
        Andrew Bayer added a comment - I eyeballed all the tarballs - I'm in the process of testing pig 0.9.0 for BIGTOP-23 right now, but once I verify that, I'll do a full build of everything to make sure they fetch.
        Hide
        Andrei Savu added a comment -

        We experienced a similar problem in Whirr. I suggest that rather than changing the mirror a more permanent fix would be to add a retry loop around curl (it seems like it doesn't automatically retry partial downloads).

        Show
        Andrei Savu added a comment - We experienced a similar problem in Whirr. I suggest that rather than changing the mirror a more permanent fix would be to add a retry loop around curl (it seems like it doesn't automatically retry partial downloads).
        Hide
        Andrew Bayer added a comment -

        We do need a retry loop added, but for right now, I'm fine with just using a consistent, functional mirror, so that we can actually build in the first place. =) I've tested and all the tarballs do download via apache.osuosl.org.

        Show
        Andrew Bayer added a comment - We do need a retry loop added, but for right now, I'm fine with just using a consistent, functional mirror, so that we can actually build in the first place. =) I've tested and all the tarballs do download via apache.osuosl.org.
        Hide
        Andrei Savu added a comment -

        A reliable mirror will not make the internet connection reliable but I guess the solution is good enough for now.

        Show
        Andrei Savu added a comment - A reliable mirror will not make the internet connection reliable but I guess the solution is good enough for now.
        Hide
        Andrew Bayer added a comment -

        Oh, definitely agreed - but the immediate problem here wasn't the internet connection, it was that dist.apache.org kept crapping out during downloads, while I had no problems downloading from anywhere else. =) And yeah, we should open another JIRA for wrapping curl calls in retry logic.

        Show
        Andrew Bayer added a comment - Oh, definitely agreed - but the immediate problem here wasn't the internet connection, it was that dist.apache.org kept crapping out during downloads, while I had no problems downloading from anywhere else. =) And yeah, we should open another JIRA for wrapping curl calls in retry logic.
        Andrew Bayer made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Mark Thomas made changes -
        Workflow jira [ 12627643 ] no-reopen-closed, patch-avail [ 12631080 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12631080 ] patch-available, re-open possible [ 12666196 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        2h 5m 1 Andrew Bayer 19/Aug/11 00:37
        Resolved Resolved Closed Closed
        312d 16h 18m 1 Roman Shaposhnik 26/Jun/12 16:56

          People

          • Assignee:
            Andrew Bayer
            Reporter:
            Andrew Bayer
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development