[APEXMALHAR-1614] AbstractFSWriter in append mode is not fault-tolerant

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Assignee: Timothy Farkas
• Reporter: Chandni Singh
• Created: 11/Dec/14 08:46
• Updated: 20/Jan/15 23:43
• Resolved: 20/Jan/15 23:43

Description

AbstractFSWriter in append mode is not fault tolerant and that feature should be removed.
Please Note : 'append' property of this operator only controls adding to a file which is existing ( not created by the same application).

Scenario:
-Checkpoint every 30th window.

• Application is launched. A file x (existing is modified). after modification size is 10mb. end-offsets will have 10mb.
• endWindow called. file is flushed.
• next window application crashes. Since end offsets was not check-pointed it will not know about this file.
• When the operator recovers, it will not know that it had modified file x and has to truncate it. This violates its fault-tolerant behavior.

We haven't seen any use of append feature. A file should be opened in append mode only if the operator identifies whether it has seen the file before or not.

Also if there is a need to write to existing files after the application is stopped and a new application is launched, then the HA feature of the engine takes care of this.
If the originalAppId is provided to the new app, the engine transfers the state and therefore writers would know what files they had seen before and will do the necessary.