Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3872

Add support for caching PyPi packages

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Infrastructure
    • Labels:

      Description

      As part of the ASF migration we're moving away from checking python packages into the repository, and instead downloading them via PyPi. We don't want to hit the PyPi servers for every build, so instead we should support a local cache or repository.

        Activity

        Hide
        dknupp David Knupp added a comment -

        For what it's worth, I had some experience setting up a local cheeseshop [1] at my last job. We chose devpi [2] and it worked well for us.

        [1] https://wiki.python.org/moin/CheeseShop
        [2] http://doc.devpi.net/latest/

        Show
        dknupp David Knupp added a comment - For what it's worth, I had some experience setting up a local cheeseshop [1] at my last job. We chose devpi [2] and it worked well for us. [1] https://wiki.python.org/moin/CheeseShop [2] http://doc.devpi.net/latest/
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-3872: allow providing PyPi mirror for python packages

        We still rely on the python.org json API, which doesn't seem to be
        mirrored (instead there's a html-based index format implemented by
        the mirrors).

        The mirror can be provided by setting the PYPI_MIRROR environment
        variable. The default is "https://pypi.python.org".

        Change-Id: Ibc11f010332c0225121c86c9930e35c7ac01409c
        Reviewed-on: http://gerrit.cloudera.org:8080/4770
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-3872 : allow providing PyPi mirror for python packages We still rely on the python.org json API, which doesn't seem to be mirrored (instead there's a html-based index format implemented by the mirrors). The mirror can be provided by setting the PYPI_MIRROR environment variable. The default is "https://pypi.python.org". Change-Id: Ibc11f010332c0225121c86c9930e35c7ac01409c Reviewed-on: http://gerrit.cloudera.org:8080/4770 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins —

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development