Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-254

Provide mechanism for installing hudi-spark-bundle onto an existing spark installation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: Spark Integration
    • Labels:
      None

      Description

      A lot of discussions around this kicked off from https://github.com/apache/incubator-hudi/issues/869 

      Breaking down into phases, when we drop the hudi-spark-bundle*.jar onto the `jars` folder 

       

      a) Writing data via Hudi datasource should work 

      b) Spark datasource reads should work

       

      c)  a + Hive Sync should work

      d) SparkSQL on Hive synced table works 

       

      Start with Spark 2.3 (current demo setup) and then proceed to 2.4 and iron out issues.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vinoth Vinoth Chandar
                Reporter:
                vinoth Vinoth Chandar
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m