Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5439

Support Spark 3 and drop SparkShim

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • spark
    • None
    • Reviewed

    Description

      Support Pig-on-Spark to run on spark3.

      Initial version would only run up to Spark 3.2.4 and not on 3.3 or 3.4.
      This is due to log4j mismatch.

      After moving to log4j2 (PIG-5426), we can move Spark to 3.3 or higher.

      So far, not all unit/e2e tests pass with the proposed patch but at least compilation goes through.

      Attachments

        1. pig-5439-v02.patch
          91 kB
          Koji Noguchi
        2. pig-5439-v01.patch
          90 kB
          Koji Noguchi

        Issue Links

          Activity

            People

              knoguchi Koji Noguchi
              knoguchi Koji Noguchi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: