Hadoop Common
  1. Hadoop Common
  2. HADOOP-3152

Make index interval configuable when using MapFileOutputFormat for map-reduce job

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: io
    • Labels:
      None
    • Environment:

      ubuntu, 64bit, jdk1.6

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.

      Description

      Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
      thread title "MapFile and MapFileOutputFormat". Currently, there is no way
      to change the index interval for the output MapFile in a map-reduce job.
      As suggested, adding a static method MapFile(Configuration, int) to set the
      index interval and stores in Configuration, then MapFile.Writer constructor
      reads the setting from configuration may be a good idea.

      I also noticed that Hbase did similar things in HBASE-364.

      1. mapfile.diff
        1 kB
        Rong-En Fan
      2. mapfile.diff
        2 kB
        Rong-En Fan

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #451 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/451/ )
        Hide
        Doug Cutting added a comment -

        I committed this.

        Show
        Doug Cutting added a comment - I committed this.
        Hide
        Doug Cutting added a comment -

        Looks good to me!

        Show
        Doug Cutting added a comment - Looks good to me!
        Hide
        Rong-En Fan added a comment -

        New patch is attached with unit test. Please kindly review it. Thanks!

        Show
        Rong-En Fan added a comment - New patch is attached with unit test. Please kindly review it. Thanks!
        Hide
        Rong-En Fan added a comment -

        same patch, but includes unit test

        Show
        Rong-En Fan added a comment - same patch, but includes unit test
        Hide
        Owen O'Malley added a comment -

        Please do write one. Thanks!

        Show
        Owen O'Malley added a comment - Please do write one. Thanks!
        Hide
        Rong-En Fan added a comment -

        There is not existing test for index interval in TestMapFile.java unit test.
        So I did not add a new one for this. If this is necessary, then I will try to write
        one for it.

        Show
        Rong-En Fan added a comment - There is not existing test for index interval in TestMapFile.java unit test. So I did not add a new one for this. If this is necessary, then I will try to write one for it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12379100/mapfile.diff
        against trunk revision 643282.

        @author +1. The patch does not contain any @author tags.

        tests included -1. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12379100/mapfile.diff against trunk revision 643282. @author +1. The patch does not contain any @author tags. tests included -1. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/console This message is automatically generated.
        Hide
        Doug Cutting added a comment -

        +1 This looks good to me.

        Show
        Doug Cutting added a comment - +1 This looks good to me.
        Hide
        Rong-En Fan added a comment -

        Please kindly do code review for this patch. Thanks!

        Show
        Rong-En Fan added a comment - Please kindly do code review for this patch. Thanks!

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Rong-En Fan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development