Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-519

Document the need for Avro dependency shading/relocation for custom payloads, need for spark-avro

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.1
    • Component/s: Docs, Usability
    • Labels:
      None

      Description

      In https://github.com/apache/incubator-hudi/pull/1005 we are migrating Hudi to Spark 2.4.4. As part of this migration, we also had to migrate Hudi to use Avro 1.8.2 (required by spark), while Hive still uses older version of Avro.

      This has resulted in the need to shade Avro in hadoop-mr-bundle. This has implications on users of Hudi, who implement custom record payloads. They would have start shading Avro in there custom jars, similar to how it shaded in hadoop-mr-bundle.

      This Jira is to track the documentation of this caveat in release notes, and if needed at other places like website etc.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                xleesf leesf
                Reporter:
                uditme Udit Mehrotra
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: