[PIG-966] Proposed rework for LoadFunc, StoreFunc, and Slice/r interfaces

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: impl
• Hadoop Flags: Incompatible change
• Assignee: Alan Gates
• Reporter: Alan Gates
• Created: 18/Sep/09 20:10
• Updated: 14/May/10 06:45
• Resolved: 01/May/10 00:55

Description

I propose that we rework the LoadFunc, StoreFunc, and Slice/r interfaces significantly. See http://wiki.apache.org/pig/LoadStoreRedesignProposal for full details

Issue Links

• incorporates: PIG-1174 Creation of output path should be done by storage function
• is blocked by: PIG-879 Pig should provide a way for input location string in load statement to be passed as-is to the Loader
• is blocked by: PIG-1216 New load store design does not allow Pig to validate inputs and outputs up front

Sub-Tasks

• load-store-redesign branch: change SampleLoader and subclasses to work with new LoadFunc interface: Closed
• ReversibleLoadStoreFunc interface should be removed to enable different load and store implementation classes to be used in a reversible manner: Closed
• change merge join and merge join indexer to work with new LoadFunc interface: Closed
• Update sources to reflect recent changes in load-store interfaces: Closed
• Fix unit tests corresponding to source changes so far: Closed
• Handle compressed file formats -- Gz, BZip with the new proposal: Closed
• Make streaming work with the new load-store interfaces: Closed
• Move splitable logic from pig latin to InputFormat: Closed
• Using TableInputFormat in HBaseStorage: Closed
• Temporarily disable failed unit test in load-store-redesign branch which have external dependency: Closed
• Enhance HBaseStorage-- Make it support loading row key and implement StoreFunc: Closed
• allow store to overwrite existing directroy: Closed
• \: Resolved
• Make StoreFunc an abstract class and create a mirror interface called StoreFuncInterface: Closed