Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11779

Dataflow runner v2 tests occasionally fails with misconfigured sharding logic

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-dataflow
    • None

    Description

      Dataflow runner v2 test sometimes mysteriously fails data loss error message. This is due to misconfigured sharding logic from backend with combination of shuffle_mode=service and use_runner_v2 logic.

      Example failing pipeline: https://pantheon.corp.google.com/dataflow/jobs/us-central1/2021-02-08_10_20_51-4474446822883759937?project=apache-beam-testing

      Attachments

        Activity

          People

            Unassigned Unassigned
            yichi Yichi Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 9h
                9h