I'm attaching a test case which illustrates the problem. When this problem occurs, the NN will fail to be able to read the edit log and will fail to start with an error like the following:
java.io.FileNotFoundException: File does not exist: /test-file
The sequence of events that I've identified that can cause this are the following:
- A file is opened for write and some data has been written/flushed to it, causing a block to be allocated.
- A snapshot is taken which includes the file.
- The file is deleted from the present file system, though the client has not yet closed the file. This will log an OP_DELETE to the edit log.
- Some error happens triggering pipeline recovery, which log an OP_UPDATE_BLOCKS to the edit log.
The reason it's possible for this to happen is basically because the updatePipeline RPC never checks if the file actually exists, but instead just finds the file INode based on the block ID being replaced in the pipeline. Later, when we're reading the OP_UPDATE_BLOCKS from the edit log, however, we try to find the file INode based on the path name of the file, which no longer exists because of the previous delete.
It's not entirely obvious to me what the right solution to this issue should be. It shouldn't be difficult to change the FSEditLogLoader to be able to read the OP_UPDATE_BLOCKS op if we just change it to look up the INode by block ID. On the other hand, however, I'm not entirely sure we should even be allowing this sequence of edit log ops in the first place. It doesn't seem unreasonable to me that we might check that the file actually exists in the present file system in the updatePipeline RPC call and throw an error if it doesn't, since continuing to write to a file that only exists in a snapshot doesn't make much sense. Along similar lines, it seems a little odd to me that an INode that only exists in the snapshot would continue to be considered under-construction, but perhaps that's not unreasonable in itself.
Would love to hear others' thoughts on this.