[PIG-201] BufferedPositionedInputStream is not buffered

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Assignee: Mathieu Poumeyrol
• Reporter: Mathieu Poumeyrol
• Created: 10/Apr/08 09:43
• Updated: 25/Mar/10 00:12
• Resolved: 26/Jan/09 20:23

Description

BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.

I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.

FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...