Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/streaming, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestStreamingAsDifferentUser fails on trunk with following exception :

      Can not create a Path from a null string
      java.lang.IllegalArgumentException: Can not create a Path from a null string
      at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
      at org.apache.hadoop.fs.Path.<init>(Path.java:90)
      at org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
      at org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
      at org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)

      The corresponding line for the exception :

          Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
      
      1. 1322.patch
        3 kB
        Devaraj Das
      2. 1322-1.patch
        3 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/)
          . Defines default value for staging directory to be user based fixing a failing streaming test. Contributed by Devaraj Das and Amareshwari Sriramadasu.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/ ) . Defines default value for staging directory to be user based fixing a failing streaming test. Contributed by Devaraj Das and Amareshwari Sriramadasu.
          Hide
          Hemanth Yamijala added a comment -

          I just committed this to trunk. Thanks, Devaraj and Amareshwari !

          Show
          Hemanth Yamijala added a comment - I just committed this to trunk. Thanks, Devaraj and Amareshwari !
          Hide
          Hemanth Yamijala added a comment -

          +1. Looks fine to me. I will commit this patch.

          Show
          Hemanth Yamijala added a comment - +1. Looks fine to me. I will commit this patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          -1 core tests.

          Is due to MAPREDUCE-1412

          Show
          Amareshwari Sriramadasu added a comment - -1 core tests. Is due to MAPREDUCE-1412
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431757/1322-1.patch
          against trunk revision 904397.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431757/1322-1.patch against trunk revision 904397. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/416/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Updated to trunk

          Show
          Amareshwari Sriramadasu added a comment - Updated to trunk
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch does not apply anymore

          Show
          Amareshwari Sriramadasu added a comment - Patch does not apply anymore
          Hide
          Devaraj Das added a comment -

          Isnt MAPREDUCE-1139 the right fix?

          That fix would address this issue as well but I'd like the staging directory to be defined in mapred-default.xml anyway.

          Error Launching job : chmod: cannot access `/tmp/hadoop/mapred/staging/dummy44913967/.staging/job_local_0001': No such file or directory

          The attached patch should fix this problem (this happens due to the fact that the same /tmp/hadoop directory is being read/written by multiple users).

          Show
          Devaraj Das added a comment - Isnt MAPREDUCE-1139 the right fix? That fix would address this issue as well but I'd like the staging directory to be defined in mapred-default.xml anyway. Error Launching job : chmod: cannot access `/tmp/hadoop/mapred/staging/dummy44913967/.staging/job_local_0001': No such file or directory The attached patch should fix this problem (this happens due to the fact that the same /tmp/hadoop directory is being read/written by multiple users).
          Hide
          Amar Kamat added a comment -

          Isnt MAPREDUCE-1139 the right fix? Also I see the following error message due to staging area sharing in testcases

          09/12/22 19:18:10 ERROR streaming.StreamJob: 
          Error Launching job : chmod: cannot access `/tmp/hadoop/mapred/staging/dummy44913967/.staging/job_local_0001': No such file or directory
          

          Should we fix that here?

          Show
          Amar Kamat added a comment - Isnt MAPREDUCE-1139 the right fix? Also I see the following error message due to staging area sharing in testcases 09/12/22 19:18:10 ERROR streaming.StreamJob: Error Launching job : chmod: cannot access `/tmp/hadoop/mapred/staging/dummy44913967/.staging/job_local_0001': No such file or directory Should we fix that here?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12428712/1322.patch
          against trunk revision 893055.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428712/1322.patch against trunk revision 893055. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/235/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          I ran the failing test with the patch, it passed now.

          Show
          Amareshwari Sriramadasu added a comment - I ran the failing test with the patch, it passed now.
          Hide
          Devaraj Das added a comment -

          My initial thought on this one was that this config doesn't require to be tweaked at all (even by admins). It should typically point to "/user". But for testing purposes, i wanted to have it point to some temp dir. It didn't strike me that streaming tests don't pick the conf up from src/test.
          After thinking about it some more, i think it makes sense to indeed define the config in mapred-default.xml.
          Attaching a patch.

          Show
          Devaraj Das added a comment - My initial thought on this one was that this config doesn't require to be tweaked at all (even by admins). It should typically point to "/user". But for testing purposes, i wanted to have it point to some temp dir. It didn't strike me that streaming tests don't pick the conf up from src/test. After thinking about it some more, i think it makes sense to indeed define the config in mapred-default.xml. Attaching a patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          Looks like "mapreduce.jobtracker.staging.root.dir" is not added in mapred-default.xml, but is added to src/test/mapred-site.xml. Streaming tests do not pick mapred-site.xml from src/test/. Is there any reason for not adding the configuration in mapred-default.xml?

          Show
          Amareshwari Sriramadasu added a comment - Looks like "mapreduce.jobtracker.staging.root.dir" is not added in mapred-default.xml, but is added to src/test/mapred-site.xml. Streaming tests do not pick mapred-site.xml from src/test/. Is there any reason for not adding the configuration in mapred-default.xml?

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development