Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-879

TestTaskTrackerLocalization fails on MAC OS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Environment:

      Mac OS X 10.5.7

    • Hadoop Flags:
      Reviewed

      Description

      TestTaskTrackerLocalization failed on an 'ant test' run.

        Issue Links

          Activity

          Hide
          Devaraj Das added a comment -

          Attaching the logs of the failure

          Show
          Devaraj Das added a comment - Attaching the logs of the failure
          Hide
          Vinod Kumar Vavilapalli added a comment -

          TestTrackerDistributedCacheManagerWithLinuxTaskController added as part of MAPREDUCE-856 will also fail on Macs for the same reason.

          Show
          Vinod Kumar Vavilapalli added a comment - TestTrackerDistributedCacheManagerWithLinuxTaskController added as part of MAPREDUCE-856 will also fail on Macs for the same reason.
          Hide
          Mahadev konar added a comment -

          Isnt that too large a name ? Sorry I couldnt help but notice such a huge name which I thought was just how you had written it but then I checked MAPREDUCE-856 to see that such a file name actually exists.

          Show
          Mahadev konar added a comment - Isnt that too large a name ? Sorry I couldnt help but notice such a huge name which I thought was just how you had written it but then I checked MAPREDUCE-856 to see that such a file name actually exists.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Isnt that too large a name ?

          Well, at that time, I couldn't quickly think of a name which gives enough description about the test. Suggestions?

          Show
          Vinod Kumar Vavilapalli added a comment - Isnt that too large a name ? Well, at that time, I couldn't quickly think of a name which gives enough description about the test. Suggestions?
          Hide
          steve_l added a comment -

          underlying cause is that -c isn't an option for stat on OS/X; instead the arguments are taken as paths, which then fail

          stat . -c %A:%U:%G
          234881026 4327070 drwxr-xr-x 20 slo staff 0 680 "Dec  1 20:49:15 2009" "Dec  1 20:59:19 2009" "Dec  1 20:59:19 2009" "Aug 21 12:12:38 2009" 4096 0 0 .
          stat: -c: stat: No such file or directory
          stat: %A:%U:%G: stat: No such file or directory
          
          Show
          steve_l added a comment - underlying cause is that -c isn't an option for stat on OS/X; instead the arguments are taken as paths, which then fail stat . -c %A:%U:%G 234881026 4327070 drwxr-xr-x 20 slo staff 0 680 "Dec 1 20:49:15 2009" "Dec 1 20:59:19 2009" "Dec 1 20:59:19 2009" "Aug 21 12:12:38 2009" 4096 0 0 . stat: -c: stat: No such file or directory stat: %A:%U:%G: stat: No such file or directory
          Hide
          Sreekanth Ramakrishnan added a comment -

          Attaching a patch fixing the issue, instead of using stat the patch uses the following which outputs in same format as stat -c command used

          ls -l -d path | awk '{print $1":"$3":"$4}
          

          The long format of the file listing on mac and linux are same and we are pretty much safe on all OS's which share the same long format listing.

          Tested the the test case on linux and mac both passed successfully.

          Show
          Sreekanth Ramakrishnan added a comment - Attaching a patch fixing the issue, instead of using stat the patch uses the following which outputs in same format as stat -c command used ls -l -d path | awk '{print $1":"$3":"$4} The long format of the file listing on mac and linux are same and we are pretty much safe on all OS's which share the same long format listing. Tested the the test case on linux and mac both passed successfully.
          Hide
          Hemanth Yamijala added a comment -

          I am fine with this approach and the patch. I also tested the command on freebsd and it works fine. So, possibly all Posix or BSD compliant systems are covered. And given this is a test case, we possibly should keep things simple. Unless someone objects, I am OK committing this. Sreekanth, can you please run through Hudson ?

          Show
          Hemanth Yamijala added a comment - I am fine with this approach and the patch. I also tested the command on freebsd and it works fine. So, possibly all Posix or BSD compliant systems are covered. And given this is a test case, we possibly should keep things simple. Unless someone objects, I am OK committing this. Sreekanth, can you please run through Hudson ?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427324/mapreduce-879-1.patch
          against trunk revision 889085.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427324/mapreduce-879-1.patch against trunk revision 889085. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/315/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The test case failure of the testJobShell seems to be an issue of MAPREDUCE-1275
          The gridmix failure is handled in MAPREDUCE-1124

          Show
          Sreekanth Ramakrishnan added a comment - The test case failure of the testJobShell seems to be an issue of MAPREDUCE-1275 The gridmix failure is handled in MAPREDUCE-1124
          Hide
          Hemanth Yamijala added a comment -

          I just committed this to trunk and branch 0.21. Thanks, Sreekanth !

          Show
          Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.21. Thanks, Sreekanth !
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #171 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/171/)
          . Fix broken unit test TestTaskTrackerLocalization on MacOS. Contributed by Sreekanth Ramakrishnan.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #171 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/171/ ) . Fix broken unit test TestTaskTrackerLocalization on MacOS. Contributed by Sreekanth Ramakrishnan.
          Hide
          Allen Wittenauer added a comment -

          Sorry I missed this.

          Interpreting ls output is guaranteed to be not safe if the user name or group name is longer than 8 characters.

          Show
          Allen Wittenauer added a comment - Sorry I missed this. Interpreting ls output is guaranteed to be not safe if the user name or group name is longer than 8 characters.
          Hide
          Hemanth Yamijala added a comment -

          How does using FileSystem.listStatus sound ? It seems like that would give us all the info we need (including permissions for Local FS), right ?

          Show
          Hemanth Yamijala added a comment - How does using FileSystem.listStatus sound ? It seems like that would give us all the info we need (including permissions for Local FS), right ?
          Hide
          Allen Wittenauer added a comment -

          If I'm looking at the code correctly, it does the same thing.

             /// loads permissions, owner, and group from `ls -ld`
              private void loadPermissionInfo() {
                IOException e = null;
                try {
                  StringTokenizer t = new StringTokenizer(
                      execCommand(new File(getPath().toUri()),
                                  Shell.getGET_PERMISSION_COMMAND()));
                  //expected format
                  //-rw-------    1 username groupname ...
          

          The problem is that ls is one of the shell commands that output was never defined for users/groups over 8 characters because at the time, UNIX didn't support users/groups over 8 characters. Thus every OS does their own thing when faced with this situation. Some put a space such that awk works, some truncate to 8, some smash them all together. [In the case of Solaris, the official response is "Not Support" much to the angst of the SA community, but Sun is specifically trying to avoid undefined behavior like this one.]

          stat (as we've discovered) isn't portable either.

          Best bet is probably to shell out a shell script that does the proper thing for the OS it is running on until we get a decent JNI library in place.

          Show
          Allen Wittenauer added a comment - If I'm looking at the code correctly, it does the same thing. /// loads permissions, owner, and group from `ls -ld` private void loadPermissionInfo() { IOException e = null ; try { StringTokenizer t = new StringTokenizer( execCommand( new File(getPath().toUri()), Shell.getGET_PERMISSION_COMMAND())); //expected format //-rw------- 1 username groupname ... The problem is that ls is one of the shell commands that output was never defined for users/groups over 8 characters because at the time, UNIX didn't support users/groups over 8 characters. Thus every OS does their own thing when faced with this situation. Some put a space such that awk works, some truncate to 8, some smash them all together. [In the case of Solaris, the official response is "Not Support" much to the angst of the SA community, but Sun is specifically trying to avoid undefined behavior like this one.] stat (as we've discovered) isn't portable either. Best bet is probably to shell out a shell script that does the proper thing for the OS it is running on until we get a decent JNI library in place.
          Hide
          Ravi Phulari added a comment -

          I see this test failing on Hadoop 20.s , We need to backport this patch to 0.20 branch.

          Show
          Ravi Phulari added a comment - I see this test failing on Hadoop 20.s , We need to backport this patch to 0.20 branch.
          Hide
          Jakob Homan added a comment -

          re-opening to close with correct status.

          Show
          Jakob Homan added a comment - re-opening to close with correct status.
          Hide
          Jakob Homan added a comment -

          Re-resolving as fixed. Prior re-opening was in error.

          Show
          Jakob Homan added a comment - Re-resolving as fixed. Prior re-opening was in error.

            People

            • Assignee:
              Sreekanth Ramakrishnan
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development