Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1284

TestLocalizationWithLinuxTaskController fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixes a bug in linux task controller by making the paths array passed to fts_open() as null-terminated as per the man page.

      Description

      With current trunk, the testcase TestLocalizationWithLinuxTaskController fails with an exit code of 139 from task-controller when doing INITIALIZE_USER

      1. MR-1284.patch
        0.6 kB
        Ravi Gummadi

        Issue Links

          Activity

          Hide
          Ravi Gummadi added a comment -

          log messages show the following:
          -------------------------------------------------
          [junit] 2009-12-09 17:07:16,371 INFO tasktracker.Localizer (Localizer.java:initializeUserDirs(202)) - Initializing user ravi_tmp on this TT.
          [junit] 2009-12-09 17:07:16,390 WARN mapred.LinuxTaskController (LinuxTaskController.java:runCommand(192)) - Exit code from INITIALIZE_USER is : 139
          [junit] 2009-12-09 17:07:16,391 WARN mapred.LinuxTaskController (LinuxTaskController.java:runCommand(194)) - Exception thrown by INITIALIZE_USER : org.apache.hadoop.util.Shell$ExitCodeException:
          [junit] at org.apache.hadoop.util.Shell.runCommand(Shell.java:243)
          [junit] at org.apache.hadoop.util.Shell.run(Shell.java:170)
          [junit] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363)
          [junit] at org.apache.hadoop.mapred.LinuxTaskController.runCommand(LinuxTaskController.java:190)
          [junit] at org.apache.hadoop.mapred.LinuxTaskController.initializeUser(LinuxTaskController.java:425)
          [junit] at org.apache.hadoop.mapred.ClusterWithLinuxTaskController$MyLinuxTaskController.initializeUser(ClusterWithLinuxTaskController.java:68)
          [junit] at org.apache.hadoop.mapreduce.server.tasktracker.Localizer.initializeUserDirs(Localizer.java:271)
          [junit] at org.apache.hadoop.mapred.TestTaskTrackerLocalization.testTaskLocalization(TestTaskTrackerLocalization.java:456)
          [junit] at org.apache.hadoop.mapred.TestLocalizationWithLinuxTaskController.testTaskLocalization(TestLocalizationWithLinuxTaskController.java:223)
          -------------------------------------------------------

          fts_open() in secure_path() seem to be making task-controller exit.

          Show
          Ravi Gummadi added a comment - log messages show the following: ------------------------------------------------- [junit] 2009-12-09 17:07:16,371 INFO tasktracker.Localizer (Localizer.java:initializeUserDirs(202)) - Initializing user ravi_tmp on this TT. [junit] 2009-12-09 17:07:16,390 WARN mapred.LinuxTaskController (LinuxTaskController.java:runCommand(192)) - Exit code from INITIALIZE_USER is : 139 [junit] 2009-12-09 17:07:16,391 WARN mapred.LinuxTaskController (LinuxTaskController.java:runCommand(194)) - Exception thrown by INITIALIZE_USER : org.apache.hadoop.util.Shell$ExitCodeException: [junit] at org.apache.hadoop.util.Shell.runCommand(Shell.java:243) [junit] at org.apache.hadoop.util.Shell.run(Shell.java:170) [junit] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:363) [junit] at org.apache.hadoop.mapred.LinuxTaskController.runCommand(LinuxTaskController.java:190) [junit] at org.apache.hadoop.mapred.LinuxTaskController.initializeUser(LinuxTaskController.java:425) [junit] at org.apache.hadoop.mapred.ClusterWithLinuxTaskController$MyLinuxTaskController.initializeUser(ClusterWithLinuxTaskController.java:68) [junit] at org.apache.hadoop.mapreduce.server.tasktracker.Localizer.initializeUserDirs(Localizer.java:271) [junit] at org.apache.hadoop.mapred.TestTaskTrackerLocalization.testTaskLocalization(TestTaskTrackerLocalization.java:456) [junit] at org.apache.hadoop.mapred.TestLocalizationWithLinuxTaskController.testTaskLocalization(TestLocalizationWithLinuxTaskController.java:223) ------------------------------------------------------- fts_open() in secure_path() seem to be making task-controller exit.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The trunk test passes successfully for me. Some mistake in building the binary or using an old version of the binary? Can you recheck?

          Show
          Vinod Kumar Vavilapalli added a comment - The trunk test passes successfully for me. Some mistake in building the binary or using an old version of the binary? Can you recheck?
          Hide
          Ravi Gummadi added a comment -

          Attaching patch with the fix.

          Problem was that the paths array passed to fts_open() should be null-terminated as per the man page.

          Show
          Ravi Gummadi added a comment - Attaching patch with the fix. Problem was that the paths array passed to fts_open() should be null-terminated as per the man page.
          Hide
          Ravi Gummadi added a comment -

          The issue of "not having null-terminated array that is being passed to fts_open()" is caught now because of different code generated by gcc with the option -O2 added in task-controller's MakeFile.in in MAPREDUCE-1119.

          Show
          Ravi Gummadi added a comment - The issue of "not having null-terminated array that is being passed to fts_open()" is caught now because of different code generated by gcc with the option -O2 added in task-controller's MakeFile.in in MAPREDUCE-1119 .
          Hide
          Ravi Gummadi added a comment -

          Unit tests passed on my local machine. Only failures are TestGridmixSubmission(MAPREDUCE-1124) and TestTrackerDistributedCacheManagerWithLinuxTaskController(MAPREDUCE-1289).

          Show
          Ravi Gummadi added a comment - Unit tests passed on my local machine. Only failures are TestGridmixSubmission( MAPREDUCE-1124 ) and TestTrackerDistributedCacheManagerWithLinuxTaskController( MAPREDUCE-1289 ).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The reason I think why I didn't get this error is that I have more recent versions of the glibc(2.10-1) (which are more forgiving, perhaps).

          Show
          Vinod Kumar Vavilapalli added a comment - The reason I think why I didn't get this error is that I have more recent versions of the glibc(2.10-1) (which are more forgiving, perhaps).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the patch.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          submitting for hudson

          Show
          Amareshwari Sriramadasu added a comment - submitting for hudson
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427577/MR-1284.patch
          against trunk revision 891111.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427577/MR-1284.patch against trunk revision 891111. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/200/console This message is automatically generated.
          Hide
          Ravi Gummadi added a comment -

          Contrib tests failures shown are not related to this patch.

          Show
          Ravi Gummadi added a comment - Contrib tests failures shown are not related to this patch.
          Hide
          Hemanth Yamijala added a comment -

          +1 for the patch. Given the nature of the fix, and the fact that it fixes a broken test (which I verified by running manually), I think there is no need for additional tests. I will commit this.

          Show
          Hemanth Yamijala added a comment - +1 for the patch. Given the nature of the fix, and the fact that it fixes a broken test (which I verified by running manually), I think there is no need for additional tests. I will commit this.
          Hide
          Hemanth Yamijala added a comment -

          I committed this to trunk and branch 0.21. Thanks, Ravi !

          Show
          Hemanth Yamijala added a comment - I committed this to trunk and branch 0.21. Thanks, Ravi !
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )

            People

            • Assignee:
              Ravi Gummadi
              Reporter:
              Ravi Gummadi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development