[PIG-3251] Bzip2TextInputFormat requires double the memory of maximum record size

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Release Note: With hadoop 2.X or later, pig will use hadoop's bzip codec to handle bzip inputs. (To turn it off, set pig.bzip.use.hadoop.inputformat=false)
• Assignee: Koji Noguchi
• Reporter: Koji Noguchi
• Created: 18/Mar/13 22:45
• Updated: 08/Jun/16 20:48
• Resolved: 29/Oct/15 21:33

Description

While looking at user's OOM heap dump, noticed that pig's Bzip2TextInputFormat consumes memory at both

Bzip2TextInputFormat.buffer (ByteArrayOutputStream)
and actual Text that is returned as line.

For example, when having one record with 160MBytes, buffer was 268MBytes and Text was 160MBytes.

We can probably eliminate one of them.

Issue Links

• is duplicated by: PIG-4591 Drop use of the internal Bzip2TextInputFormat
• is related to: MAPREDUCE-5656 bzip2 codec can drop records when reading data in splits
• is related to: PIG-4533 Document error: Pig does support concatenated gz file
• relates to: PIG-3352 Bzip2TextInputFormat can duplicate records across splits