This is to track adding the remaining type support in Arrow Converters. Currently, only primitive data types are supported. '
- Complex: Struct,
Array, Arrays of Date/Timestamps, Map Decimal Binary
- Categorical when converting from Pandas
Some things to do before closing this out:
Look to upgrading to Arrow 0.7 for better Decimal support (can now write values as BigDecimal) Need to add some user docs Make sure Python tests are thorough
- Check into complex type support mentioned in comments by Leif Mortenson, should we support mulit-indexing?
|Add MapType Support for Arrow in PySpark||Reopened||Unassigned|