[MAPREDUCE-5511] Multifilewc and the mapred.* API: Is the use of getPos() valid?

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.0.0, 1.2.0
• Component/s: examples
• Assignee: Unassigned
• Reporter: jay vyas
• Created: 16/Sep/13 15:35
• Updated: 16/Sep/13 16:13

Description

The MultiFileWordCount class in the hadoop examples libraries uses a record reader which switches between files. This behaviour can cause the RawLocalFileSystem to break in a concurrent environment because of the way buffering works (in RawLocalFileSystem, switching between streams results in a temproraily "null" inner stream, and that inner stream is called by the getPos() implementation in the custom RecordReader for MultiFileWordCount).

There are basically 2 ways to handle this:

1) Wrap the getPos() implementation in the object returned by open() in the RawLocalFileSystem to cache the value of getPos() everytime it is called, so that calls to getPos() can return a valid long even if underlying stream is null. OR

2) Update the RecordReader in multifilewc to not rely on the inner input stream and cache the position / return 0 if the stream cannot return a valid value.

The final question here is: Is the RecordReader for MultiFileWordCount doing the right thing ? Or is it breaking the contract of getPos()... and really... what SHOULD getPos() return if the underlying stream has already been consumed?