Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10708

InteractiveRunner cannot execute pipeline with cross-language transform

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: P2
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: cross-language
    • Labels:
      None

      Description

      The InteractiveRunner crashes when given a pipeline that includes a cross-language transform.

      Here's the example I tried to run in a jupyter notebook:

      p = beam.Pipeline(InteractiveRunner())
      pc = (p | SqlTransform("""SELECT
                  CAST(1 AS INT) AS `id`,
                  CAST('foo' AS VARCHAR) AS `str`,
                  CAST(3.14  AS DOUBLE) AS `flt`"""))
      df = interactive_beam.collect(pc)
      

      The problem occurs when pipeline_fragment.py creates a copy of the pipeline by writing it to proto and reading it back. Reading it back fails because some of the pipeline is not written in Python.

        Attachments

          Activity

            People

            • Assignee:
              ningk Ning Kang
              Reporter:
              bhulette Brian Hulette

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5.5h
                5.5h

                  Issue deployment