[HIVE-15844] Make ReduceSinkOperator independent of Acid

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: Transactions
• Target Version/s: 2.2.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 08/Feb/17 01:07
• Updated: 15/Aug/17 22:16
• Resolved: 01/Mar/17 19:26

Description

1. both FileSinkDesk and ReduceSinkDesk have special code path for Update/Delete operations. It is not always set correctly for ReduceSink. ReduceSinkDeDuplication is one place where it gets lost. Even when it isn't set correctly, elsewhere (SemanticAnalyzer.getPartitionColsFromBucketColsForUpdateDelete()) we set ROW_ID to be the partition column of the ReduceSinkOperator and UDFToInteger special cases it to extract bucketId from ROW_ID. We need to modify Explain Plan to record Write Type (i.e. insert/update/delete) to make sure we have tests that can catch errors here.
2. Add some validation at the end of the plan to make sure that RSO/FSO which represent the end of the pipeline and write to acid table have WriteType set (to something other than default).
3. We don't seem to have any tests where number of buckets is > number of reducers. Add those.

Issue Links

• is related to: HIVE-15756 Update/deletes on ACID table throws ArrayIndexOutOfBoundsException
• is related to: HIVE-17328 Remove special handling for Acid tables wherever possible
• relates to: HIVE-16077 UPDATE/DELETE fails with numBuckets > numReducers
• links to: Review Board