Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1659

RaidNode should write temp files on /tmp and add random numbers to their names to avoid conflicts

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The RaidNode methods to raid files and recover them should write recovery and tmp files on /tmp instead of /raid.

      Besides that, filenames should have a random number appended to them to avoid conflicts. This makes the code safer and avoids errors when multiple recoveries run in parallel.

        Activity

        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Rodrigo!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Rodrigo!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442324/MAPREDUCE-1659.0.patch
        against trunk revision 936042.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442324/MAPREDUCE-1659.0.patch against trunk revision 936042. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/120/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        I agree. I will wait for HadoopQA tests before committing this one. Thanks.

        Show
        dhruba borthakur added a comment - I agree. I will wait for HadoopQA tests before committing this one. Thanks.
        Hide
        Rodrigo Schmidt added a comment -

        hadoop.tmp.dir seems to be the local tmp directory used by mapreduce. Maybe it's better to leave "/tmp" in that case. Anyway, there are config vars for these temp dirs at the raid level, so users can redirect the tmp files to different places if they wish to.

        Show
        Rodrigo Schmidt added a comment - hadoop.tmp.dir seems to be the local tmp directory used by mapreduce. Maybe it's better to leave "/tmp" in that case. Anyway, there are config vars for these temp dirs at the raid level, so users can redirect the tmp files to different places if they wish to.
        Hide
        dhruba borthakur added a comment -

        Code looks good. If you can use the tmp directory setup by hadoop.tmp.dir, that will be great.

        Show
        dhruba borthakur added a comment - Code looks good. If you can use the tmp directory setup by hadoop.tmp.dir, that will be great.
        Hide
        Rodrigo Schmidt added a comment -

        New patch

        Show
        Rodrigo Schmidt added a comment - New patch

          People

          • Assignee:
            Rodrigo Schmidt
            Reporter:
            Rodrigo Schmidt
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development