Hadoop Common
  1. Hadoop Common
  2. HADOOP-3152

Make index interval configuable when using MapFileOutputFormat for map-reduce job

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: io
    • Labels:
      None
    • Environment:

      ubuntu, 64bit, jdk1.6

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.

      Description

      Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
      thread title "MapFile and MapFileOutputFormat". Currently, there is no way
      to change the index interval for the output MapFile in a map-reduce job.
      As suggested, adding a static method MapFile(Configuration, int) to set the
      index interval and stores in Configuration, then MapFile.Writer constructor
      reads the setting from configuration may be a good idea.

      I also noticed that Hbase did similar things in HBASE-364.

      1. mapfile.diff
        2 kB
        Rong-En Fan
      2. mapfile.diff
        1 kB
        Rong-En Fan

        Activity

        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Doug Cutting made changes -
        Issue Type Bug [ 1 ] New Feature [ 2 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Doug Cutting made changes -
        Resolution Fixed [ 1 ]
        Release Note Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Doug Cutting made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Rong-En Fan made changes -
        Attachment mapfile.diff [ 12379358 ]
        Owen O'Malley made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Doug Cutting made changes -
        Assignee Doug Cutting [ cutting ]
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.17.0 [ 12312913 ]
        Rong-En Fan made changes -
        Field Original Value New Value
        Attachment mapfile.diff [ 12379100 ]
        Rong-En Fan created issue -

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Rong-En Fan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development