Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7979

Avro incompatibilities with Spark 2.2 and Spark 2.3

    XMLWordPrintableJSON

Details

    Description

      Much of the code that depends on Avro (notably the wrappers built with BeamSQL but also some connectors) require a version > 1.8.x

      This library is not present in Spark 2.2 and Spark 2.3 clusters, which are meant to be supported. These pipelines will fail with ClassNotFoundException / MethodNotFoundExceptions.

      Spark 2.4+ should be unaffected.

      Relocating or vendoring is probably not appropriate, since Avro is frequently exposed in the API through parameters and potentially in generated specific records.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-5164 ParquetIOIT fails on Spark and Flink

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Activity

            People

              Unassigned Unassigned
              rskraba Ryan Skraba
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: