[PIG-3770] Enhance DBStorage to make it more flexible (should batch statements?, rollback on job failure, support command line arguments etc.)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: piggybank
• Assignee: Unassigned
• Reporter: Nezih Yigitbasi
• Created: 18/Feb/14 23:22
• Updated: 04/Feb/16 01:04

Description

First of all, the TestDBStorage unit test is broken. It doesn't even run the DBStorage store logic. I debugged it and added logs to find out that putNext is not even called. The reason this unit test doesn't fail is that the verification loop at the end of the testWriteToDB method that traverses the result set simply doesn't do any verification since the result set is empty (since DBStorage store logic is not called at all) and it doesn't enter that for loop. (If it could run it would fail as the verification logic is also broken: see that the orders in the expNames, expRations, and expDates do not even match). This has to be fixed.

I propose to improve DBStorage with the following changes:

• fix the problems with the unit test described above to make it work, and make it more comprehensive (the unit test currently only inserts three records, this test has to be made more comprehensive)
• use command line options in the constructor like other Pig store functions (PigStorage, HBaseStorage, etc.) to make DBStorage more flexible. With this change it would be easy to implement PIG-3597
• DBStorage supports rollbacks on task failures, but not on job failures. This is a nice to have feature that's requested before, see PIG-1891

Issue Links

• relates to: PIG-3597 Enable piggybank DBStorage to store to JDBC databases without Batch Statement support
• relates to: PIG-1891 Enable StoreFunc to make intelligent decision based on job success or failure