Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-13314

Revise recommendations to manage Python pipeline dependencies.

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.38.0
    • sdk-py-core, website

    Description

      The page https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ recommends managing Python dependencies via requirements files.

      This approach is currently inefficient in light of introduction and adoption of PEP-517 by some packages, see: https://lists.apache.org/thread/trljnxo39c0cmff790yff3h8n5okqt3q and the rest of the thread, and does not mention Custom Containers or SDK prebuilding workflows.

      We should revise it and document best practices.

      Attachments

        Issue Links

          Activity

            Anand Inguva Anand Inguva added a comment -
            1. Suggest users to provide a requirements file with strict versions pinned to the dependencies.
            2. Including documentation on Custom containers. 
            3. Include pre-building workflows as well. 
            Anand Inguva Anand Inguva added a comment - Suggest users to provide a requirements file with strict versions pinned to the dependencies. Including documentation on Custom containers.  Include pre-building workflows as well. 

            People

              Anand Inguva Anand Inguva
              tvalentyn Valentyn Tymofieiev
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h 20m
                  8h 20m