Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7063

Stage Dataflow runner harness in Dataflow FnApi Python test suites that run using an unreleased SDK.

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: sdk-py-core, test-failures
    • Labels:
      None

      Description

      Seems like this was the first post-commit failure: https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Python_Verify/7855/

      (the one before this also failed but that looks like a flake)

       

       

      Looking at failed tests I noticed that some of the were due to failed Dataflow jobs. I noticed three issues.

      (1) Error in Dataflow jobs "This handler is only capable of dealing with urn:beam:sideinput:materialization:multimap:0.1 materializations but was asked to handle beam:side_input:multimap:v1 for PCollectionView with tag side0-FilterOutSpammers."

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_17_57_01-16397932959358575026?project=apache-beam-testing

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_18_07_11-1342017143600540529?project=apache-beam-testing

       

      (2)  BigQuery job failures (possibly this is transient)

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_18_25_22-16437560120793921616?project=apache-beam-testing

       

      (3) Some batch Dataflow jobs timed out and were cancelled (possibly also transient).

       

      (1) is probably due to https://github.com/apache/beam/commit/684f8130284a7c7979773300d04e5473ca0ac8f3 

       

      cc: Ahmet Altay Robert Burke Alan Myrvold

        Attachments

          Activity

            People

            • Assignee:
              tvalentyn Valentyn Tymofieiev
              Reporter:
              chamikara Chamikara Jayalath
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 10m
                5h 10m