[HADOOP-9996] Improve TFile format to support any compression codecs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha1
• Component/s: io
• Labels: Rhino
• Assignee: Unassigned
• Reporter: Haifeng Chen
• Created: 26/Sep/13 06:10
• Updated: 12/May/16 18:27

Description

TFile is a container of key-value pairs. It supports block level compression by using compression codec. But one limitation of the current implementation is it supports only a few of fixed compression codecs. They are LZO, GZ or no compression. Some new compression codecs such as Snappy cannot be used because of this limitation.

We propose to extend the existing TFile compression feature to support any compression codecs. As TFile already used the named compression codecs and stored the name in the file meta data (for example, “lzo” was stored when LZO compression is used), we cannot change this for backward compatibility. To make it support any compression codec, we add a special name “codec” after which follows the real codec class name. For example, “codec: org.apache.hadoop.io.compress.SnappyCodec” is used and stored in the meta when SnappyCodec is used as the compression codec. We can still use the existing fixed names such as “lzo”, “gz” or “none” for specifying the TFile compression codec.

Issue Links

• is required by: PIG-3289 Encryption aware load and store functions
• is required by: HADOOP-9997 Improve TFile API to be able to pass the context for encryption codecs