Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10708

InteractiveRunner cannot execute pipeline with cross-language transform

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • P2
    • Resolution: Unresolved
    • None
    • None
    • cross-language
    • None

    Description

      The InteractiveRunner crashes when given a pipeline that includes a cross-language transform.

      Here's the example I tried to run in a jupyter notebook:

      p = beam.Pipeline(InteractiveRunner())
pc = (p | SqlTransform("""SELECT
            CAST(1 AS INT) AS `id`,
            CAST('foo' AS VARCHAR) AS `str`,
            CAST(3.14  AS DOUBLE) AS `flt`"""))
df = interactive_beam.collect(pc)

      The problem occurs when pipeline_fragment.py creates a copy of the pipeline by writing it to proto and reading it back. Reading it back fails because some of the pipeline is not written in Python.

      Attachments

        Activity

          People

            Unassigned Unassigned
            bhulette Brian Hulette
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 49h 40m
                49h 40m