Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12025

Environment specific dependency provisioning for Dataflow

    XMLWordPrintableJSON

Details

    Description

      Currently Dataflow (Runner v2) artifacts are staged to a single GCS location and all artifacts are downloaded to all SDK Harnesses using a bare bones artifact retrieval service.

      We need to implement proper artifact staging/retrieval services for Dataflow and separate out dependencies by environment.

      Doc: https://docs.google.com/document/d/1jq8TgiUxvQlJCpY7Q38MtZahX_XbVAdcUcQ_qXLKfgE/edit

       

      Assigning to Heejong who hopes to look into this.

      Attachments

        Issue Links

          is duplicated by

          Sub-task - The sub-task of the issue BEAM-9455 Environment-sensitive provisioning for Dataflow

          • P1 - Critical bug: data loss, total loss of function, or loss of testing signal due to test failures or flakiness
          • Open
          links to

          Web Link GitHub Pull Request #14398

          Activity

            People

              heejong Heejong Lee
              chamikara Chamikara Madhusanka Jayalath
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m