Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21187

Complete support for remaining Spark data types in Arrow Converters

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 3.1.0
    • PySpark, SQL
    • None

    Description

      This is to track adding the remaining type support in Arrow Converters. Currently, only primitive data types are supported. '

      Remaining types:

      • Date
      • Timestamp
      • Complex: Struct, Array, Map
      • Decimal
      • Binary
      • Categorical when converting from Pandas

      Some things to do before closing this out:

      • Look to upgrading to Arrow 0.7 for better Decimal support (can now write values as BigDecimal)
      • Need to add some user docs
      • Make sure Python tests are thorough
      • Check into complex type support mentioned in comments by leif, should we support mulit-indexing?

      Attachments

        Issue Links

          is part of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-22216 Improving PySpark/Pandas interoperability

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23836 Support returning StructType to the level support in GroupedMap Arrow's "scalar" UDFS (or similar)

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27834 Make separate PySpark/SparkR vectorization configurations

          • Major - Major loss of function.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-26759 Arrow optimization in SparkR's interoperability

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-32285 Add PySpark support for nested timestamps with arrow

          • Major - Major loss of function.
          • In Progress

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-33489 Support null for conversion from and to Arrow type

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              bryanc Bryan Cutler
              bryanc Bryan Cutler
              Votes:
              3 Vote for this issue
              Watchers:
              39 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: