Uploaded image for project: 'Apache Open Climate Workbench'
  1. Apache Open Climate Workbench
  2. CLIMATE-866

Create ocw conda package against Python3.x as well as Python 2.7

    Details

    • Type: Task
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: conda
    • Labels:
      None

      Description

      Hi Alex Goodman, I recently tried to use the conda package to install OCW. I am using Miniconda3 and therefore Python3. When I run the conda install, I get the following

      lmcgibbn@LMC-032857 /usr/local/climate(master) $ conda install -c agoodman ocw
      Fetching package metadata .........
      Solving package specifications: ....
      
      The following specifications were found to be in conflict:
        - ocw
        - python 3.5*
      Use "conda info <package>" to see the dependencies for each package.
      

      This is because the packages are built using Python 2.7 as per https://anaconda.org/agoodman/ocw/files.

      Seeing as all of our CI and smoke testing now runs off of Python 3.X, I wonder if you are able to push an update to the conda files which compile against Python3?

        Issue Links

          Activity

          Hide
          lewismc Lewis John McGibbney added a comment -

          Ibrahim Jarif can you please look into adding the generation of conda packages to the existing CI suite using https://conda-forge.github.io?
          Would be really cool if we could publish a Python 3 compatible conda package inline with the existing Ci workflow.

          Show
          lewismc Lewis John McGibbney added a comment - Ibrahim Jarif can you please look into adding the generation of conda packages to the existing CI suite using https://conda-forge.github.io? Would be really cool if we could publish a Python 3 compatible conda package inline with the existing Ci workflow.
          Hide
          agoodman Alex Goodman added a comment -

          Hi Lewis John McGibbney,

          Yep, this can easily be done by adding an additional flag to specify python versions when executing conda build. I'll get to this right away tomorrow morning. AFAIK this shouldn't warrant any changes to the recipe files but it will nonetheless make maintaining the packages more cumbersome if we wish to support both python2 and 3. Perhaps I should just make a python or bash shell script that automates a lot of this since it's mostly just a matter of chaining a bunch of conda build commands together.

          Show
          agoodman Alex Goodman added a comment - Hi Lewis John McGibbney , Yep, this can easily be done by adding an additional flag to specify python versions when executing conda build . I'll get to this right away tomorrow morning. AFAIK this shouldn't warrant any changes to the recipe files but it will nonetheless make maintaining the packages more cumbersome if we wish to support both python2 and 3. Perhaps I should just make a python or bash shell script that automates a lot of this since it's mostly just a matter of chaining a bunch of conda build commands together.
          Hide
          lewismc Lewis John McGibbney added a comment -

          Would be great, I appreciate it. My feeling is that we should be moving everything on to Python 3.

          Show
          lewismc Lewis John McGibbney added a comment - Would be great, I appreciate it. My feeling is that we should be moving everything on to Python 3.
          Hide
          jarifibrahim Ibrahim Jarif added a comment -

          Hi Lewis John McGibbney, https://conda-forge.github.io contains a collection of recipes.
          Right now we have to run `conda install -c agoodman podaacpy`. If we use `conda-forge` then we'll have to run `conda install -c conda-forge ocw`.

          We have 2 possible ways to integrate conda build into our existing CI workflow.
          1. We could use Travis CI to build conda recipe and then deploy the package on Anaconda on tagged commits (Just like PodaacPy). The `ocw` package will be available under agoodman channel.

          2. We could setup `ocw` package on `conda-forge`. This might be a bit complicated as `conda-forge` creates a repo for each recipe. Lets call the repo for ocw as `ocw-feedstock`. We will need someone to maintain the `ocw-feedstock` repo. Every time the recipe gets updated the person who owns the `ocw-feedstock` repo needs to send a PR to `conda-forge`. Only then the conda package will be updated on Anaconda. The updating of recipe from `apache/climate` repo to `ocw-feedstock` could be automated. But the PR that has to be sent manually to `conda-forge`.
          We might need something like this if we choose to go with the second option https://gist.github.com/domenic/ec8b0fc8ab45f39403dd

          Show
          jarifibrahim Ibrahim Jarif added a comment - Hi Lewis John McGibbney , https://conda-forge.github.io contains a collection of recipes. Right now we have to run `conda install -c agoodman podaacpy`. If we use `conda-forge` then we'll have to run `conda install -c conda-forge ocw`. We have 2 possible ways to integrate conda build into our existing CI workflow. 1. We could use Travis CI to build conda recipe and then deploy the package on Anaconda on tagged commits (Just like PodaacPy). The `ocw` package will be available under agoodman channel. 2. We could setup `ocw` package on `conda-forge`. This might be a bit complicated as `conda-forge` creates a repo for each recipe. Lets call the repo for ocw as `ocw-feedstock`. We will need someone to maintain the `ocw-feedstock` repo. Every time the recipe gets updated the person who owns the `ocw-feedstock` repo needs to send a PR to `conda-forge`. Only then the conda package will be updated on Anaconda. The updating of recipe from `apache/climate` repo to `ocw-feedstock` could be automated. But the PR that has to be sent manually to `conda-forge`. We might need something like this if we choose to go with the second option https://gist.github.com/domenic/ec8b0fc8ab45f39403dd
          Hide
          lewismc Lewis John McGibbney added a comment -

          Any preferences Alex Goodman?

          Show
          lewismc Lewis John McGibbney added a comment - Any preferences Alex Goodman ?
          Hide
          agoodman Alex Goodman added a comment -

          I like option 2. I know we were originally planning on hosting the recipe files on some sort of apache repo in the future (CLIMATE-836), but in hindsight this is probably the best approach. I actually knew of the conda-forge but initially thought the hassle of needing to make a github PR in order to update the packages might be too much of a hinderance when attempting to prepare for new releases. However I really like that the build process itself is automated through CI services which would ultimately make maintaining the packages much easier since the only thing we actually need to maintain is the meta.yaml file for ocw (and any other dependencies not already on the conda-forge). I believe the build is automatically done for both python2 and 3 as well. The initial PRs to upload our recipes will probably take some time but it seems in practice they are very fast about merging PRs for simple updates.

          Let me know if you agree, and if so I'd be glad to set this all in motion. I was originally going to make the python3 packages today for my anaconda channel but if this is not urgent, we can automatically defer this until after our recipes are pushed to conda-forge. Let me know your thoughts Lewis John McGibbney.

          Show
          agoodman Alex Goodman added a comment - I like option 2. I know we were originally planning on hosting the recipe files on some sort of apache repo in the future ( CLIMATE-836 ), but in hindsight this is probably the best approach. I actually knew of the conda-forge but initially thought the hassle of needing to make a github PR in order to update the packages might be too much of a hinderance when attempting to prepare for new releases. However I really like that the build process itself is automated through CI services which would ultimately make maintaining the packages much easier since the only thing we actually need to maintain is the meta.yaml file for ocw (and any other dependencies not already on the conda-forge). I believe the build is automatically done for both python2 and 3 as well. The initial PRs to upload our recipes will probably take some time but it seems in practice they are very fast about merging PRs for simple updates. Let me know if you agree, and if so I'd be glad to set this all in motion. I was originally going to make the python3 packages today for my anaconda channel but if this is not urgent, we can automatically defer this until after our recipes are pushed to conda-forge. Let me know your thoughts Lewis John McGibbney .
          Hide
          agoodman Alex Goodman added a comment -

          Also, I would like to add this point: With conda-forge, it wouldn't be necessary to maintain the recipe files themselves in the main ocw repo anymore (apache/climate repo), since updating them would be a matter of merging a PR in the ocw-feedstock repo. Something I also just realized upon closer inspection is that you can manually specify the package maintainers (by github username) in the recipe files which I believe grants write access to the feedstock repo, meaning we can merge updates to the conda recipe files on our own terms (or bypass PRs altogether and directly push our commits if need be).

          Show
          agoodman Alex Goodman added a comment - Also, I would like to add this point: With conda-forge, it wouldn't be necessary to maintain the recipe files themselves in the main ocw repo anymore (apache/climate repo), since updating them would be a matter of merging a PR in the ocw-feedstock repo. Something I also just realized upon closer inspection is that you can manually specify the package maintainers (by github username) in the recipe files which I believe grants write access to the feedstock repo, meaning we can merge updates to the conda recipe files on our own terms (or bypass PRs altogether and directly push our commits if need be).
          Hide
          lewismc Lewis John McGibbney added a comment -

          An example of this can be seen on Podaacpy where we have packages for Pythong 2.7, 3.3, 3.4 and 3.5
          https://pypi.python.org/pypi/podaacpy/1.4.0

          Show
          lewismc Lewis John McGibbney added a comment - An example of this can be seen on Podaacpy where we have packages for Pythong 2.7, 3.3, 3.4 and 3.5 https://pypi.python.org/pypi/podaacpy/1.4.0
          Hide
          agoodman Alex Goodman added a comment -

          One minor point, I only submitted a build for Python 2.7 on conda-forge for now since version 1.1 isn't Python 3 compatible. When we are ready to release 1.2.0 we can then allow for Python 3 builds. Additionally AFAIK Python 3.4 and 3.5 can be built by conda-forge, but not 3.3.

          Show
          agoodman Alex Goodman added a comment - One minor point, I only submitted a build for Python 2.7 on conda-forge for now since version 1.1 isn't Python 3 compatible. When we are ready to release 1.2.0 we can then allow for Python 3 builds. Additionally AFAIK Python 3.4 and 3.5 can be built by conda-forge, but not 3.3.
          Hide
          lewismc Lewis John McGibbney added a comment -

          This is addressed now that the OCW packing is working through conda-forge Alex Goodman is that correct?

          Show
          lewismc Lewis John McGibbney added a comment - This is addressed now that the OCW packing is working through conda-forge Alex Goodman is that correct?
          Hide
          agoodman Alex Goodman added a comment -

          Lewis John McGibbney mostly correct. We will have to do a little bit of trickery to exclude the esgf modules from the runtime dependencies when building the recipe for python 3, but it shouldn't be too difficult.

          Show
          agoodman Alex Goodman added a comment - Lewis John McGibbney mostly correct. We will have to do a little bit of trickery to exclude the esgf modules from the runtime dependencies when building the recipe for python 3, but it shouldn't be too difficult.

            People

            • Assignee:
              agoodman Alex Goodman
              Reporter:
              lewismc Lewis John McGibbney
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development