[SPARK-5180] Data source API improvement (Spark 1.5)

• Type: Story
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Cheng Lian
• Reporter: Yin Huai
• Created: 09/Jan/15 21:14
• Updated: 13/Aug/15 07:31
• Resolved: 13/Aug/15 07:31

Issue Links

• is related to: SPARK-5166 Stabilize Spark SQL APIs
• is related to: SPARK-9932 Data source API improvement (Spark 1.6)

Sub-Tasks

• Adding support for defining schema in foreign DDL commands.: Resolved
• Persistent data source tables: Resolved
• Partitioning support for tables created by the data source API: Resolved
• Improve the performance of metadata operations: Resolved
• Document data source API: Resolved
• Write support for the data source API: Resolved
• Python API for the write support of the data source API: Resolved
• In memory data cache should be invalidated after insert into/overwrite: Resolved
• Preinsert casting and renaming rule is needed in the Analyzer: Resolved
• Finalize DDL and write support APIs: Resolved
• Add common string filters to data sources: Resolved
• FSBasedRelation interface tweaks: Resolved
• Do not use FloatType in partition column inference: Resolved
• Replace the hash map in DynamicPartitionWriterContainer.outputWriterForRow with java.util.HashMap: Resolved
• Reduce memory consumption for dynamic partition insert: Resolved
• Move all internal data source related classes out of sources package: Resolved
• Speed up path construction in DynamicPartitionWriterContainer.outputWriterForRow: Resolved
• DataFrame partitionBy memory pressure scales extremely poorly: Closed
• Conversion is applied twice on partitioned data sources: Resolved