[#HADOOP-3152] Make index interval configuable when using MapFileOutputFormat for map-reduce job

Hadoop Common

Make index interval configuable when using MapFileOutputFormat for map-reduce job

Created: 02/Apr/08 04:39 AM Updated: 21/May/08 08:06 PM

Return to search

Component/s:

Affects Version/s:

0.16.0

Fix Version/s:

0.17.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
mapfile.diff	2008-04-04 07:03 AM	Rong-En Fan	2 kB
mapfile.diff	2008-04-02 04:40 AM	Rong-En Fan	1 kB

Environment:

ubuntu, 64bit, jdk1.6

Hadoop Flags:	Reviewed
Release Note:	Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.
Resolution Date:	05/Apr/08 12:03 AM

Description

« Hide

Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
thread title "MapFile and MapFileOutputFormat". Currently, there is no way
to change the index interval for the output MapFile in a map-reduce job.
As suggested, adding a static method MapFile(Configuration, int) to set the
index interval and stores in Configuration, then MapFile.Writer constructor
reads the setting from configuration may be a good idea.

I also noticed that Hbase did similar things in ~~HBASE-364~~.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Rong-En Fan made changes - 02/Apr/08 04:40 AM

Field	Original Value	New Value
Attachment		mapfile.diff [ 12379100 ]

Doug Cutting made changes - 02/Apr/08 09:19 PM

Fix Version/s		0.17.0 [ 12312913 ]
Hadoop Flags		[Reviewed]
Assignee		Doug Cutting [ cutting ]
Status	Open [ 1 ]	Patch Available [ 10002 ]

Owen O'Malley made changes - 03/Apr/08 10:12 PM

Status

Patch Available [ 10002 ]

Open [ 1 ]

Rong-En Fan made changes - 04/Apr/08 07:03 AM

Attachment

mapfile.diff [ 12379358 ]

Doug Cutting made changes - 04/Apr/08 11:53 PM

Status

Open [ 1 ]

Patch Available [ 10002 ]

Doug Cutting made changes - 05/Apr/08 12:03 AM

Resolution		Fixed [ 1 ]
Release Note		Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.
Status	Patch Available [ 10002 ]	Resolved [ 5 ]

Doug Cutting made changes - 05/Apr/08 12:03 AM

Priority	Major [ 3 ]	Minor [ 4 ]
Issue Type	Bug [ 1 ]	New Feature [ 2 ]

Nigel Daley made changes - 21/May/08 08:06 PM

Status

Resolved [ 5 ]

Closed [ 6 ]