Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2, test
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed failing JUnit tests in Gridmix.
    • Tags:
      junit test
    • Target Version/s:

      Description

      When I run JUnit tests (e.g. TestDistCacheEmulation, TestSleepJob and TestCompressionEmulationUtils), I see job submission failing with the following error:

      java.lang.IllegalStateException: Variable substitution depth too large: 20 ${fs.default.name}
              at org.apache.hadoop.conf.Configuration.substituteVars(Configuration.java:551)
              at org.apache.hadoop.conf.Configuration.get(Configuration.java:569)
              at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1020)
              at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:564)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:353)
              at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159)
              at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156)
              at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176)
              at org.apache.hadoop.mapred.gridmix.Gridmix.launchGridmixJob(Gridmix.java:190)
              at org.apache.hadoop.mapred.gridmix.Gridmix.writeInputData(Gridmix.java:150)
              at org.apache.hadoop.mapred.gridmix.Gridmix.start(Gridmix.java:425)
              at org.apache.hadoop.mapred.gridmix.Gridmix.runJob(Gridmix.java:380)
              at org.apache.hadoop.mapred.gridmix.Gridmix.access$000(Gridmix.java:56)
              at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:313)
              at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:311)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
              at org.apache.hadoop.mapred.gridmix.Gridmix.run(Gridmix.java:311)
      
      1. 3462.trunk.patch
        4 kB
        Ravi Gummadi
      2. MAPREDUCE-3462.branch-0.23.patch
        3 kB
        Ravi Prakash

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Vinod Kumar Vavilapalli made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Hide
          Eli Collins added a comment -

          Also, I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix.

          I filed MAPREDUCE-3736 for the general fix. Ravi, if you figured anything out please chime in there.

          Show
          Eli Collins added a comment - Also, I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix. I filed MAPREDUCE-3736 for the general fix. Ravi, if you figured anything out please chime in there.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/948/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/948/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #150 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/150/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #150 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/150/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/128/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/128/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/915/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/915/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #344 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/344/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #344 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/344/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1509 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1509/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1509 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1509/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1560 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1560/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1560 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1560/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #322 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/322/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #322 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/322/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #333 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/333/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #333 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/333/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227052 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1488 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1488/)
          MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1488 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1488/ ) MAPREDUCE-3462 . Fix Gridmix JUnit testcase failures. (Ravi Prakash and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227051 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestSleepJob.java
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note Fixed failing JUnit tests in Gridmix.
          Fix Version/s 0.23.1 [ 12318883 ]
          Fix Version/s 0.24.0 [ 12317654 ]
          Resolution Fixed [ 1 ]
          Hide
          Amar Kamat added a comment -

          I just committed this to trunk and branch-0.23. Thanks Ravi Prakash and Ravi Gummadi!

          Show
          Amar Kamat added a comment - I just committed this to trunk and branch-0.23. Thanks Ravi Prakash and Ravi Gummadi!
          Hide
          Amar Kamat added a comment -

          Fixing contrib tests to respect src/test/mapred-site.xml can be addressed later. I will commit this patch for now.

          Show
          Amar Kamat added a comment - Fixing contrib tests to respect src/test/mapred-site.xml can be addressed later. I will commit this patch for now.
          Hide
          Ravi Prakash added a comment -

          Also, I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix. For now, couldn't these tests passing unblock you?

          Show
          Ravi Prakash added a comment - Also, I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix. For now, couldn't these tests passing unblock you?
          Hide
          Ravi Prakash added a comment -

          I'd be wary of adding it in src/main/..mapred-default.xml . I'd think src/test/../mapred-test.xml is where it should be added if at all. If that doesn't work currently, we should make it work. Just out of curiosity, do you know of any other tests which are failing due to this exception?

          Show
          Ravi Prakash added a comment - I'd be wary of adding it in src/main/..mapred-default.xml . I'd think src/test/../mapred-test.xml is where it should be added if at all. If that doesn't work currently, we should make it work. Just out of curiosity, do you know of any other tests which are failing due to this exception?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12508507/3462.trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1503//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1503//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508507/3462.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1503//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1503//console This message is automatically generated.
          Ravi Gummadi made changes -
          Attachment 3462.trunk.patch [ 12508507 ]
          Hide
          Ravi Gummadi added a comment -

          Patch for trunk with all gridmix testcases updated with this config setting.

          I guess it would be better to set it somewhere in tests-config.

          Show
          Ravi Gummadi added a comment - Patch for trunk with all gridmix testcases updated with this config setting. I guess it would be better to set it somewhere in tests-config.
          Hide
          Amar Kamat added a comment -

          I think setting mapreduce.job.hdfs-servers to an empty string in src/java/mapred-default.xml should take care of the failures. Thoughts?

          Show
          Amar Kamat added a comment - I think setting mapreduce.job.hdfs-servers to an empty string in src/java/mapred-default.xml should take care of the failures. Thoughts?
          Hide
          Amar Kamat added a comment -

          Tested the fix on TestCompressionEmulationUtils and the test passed. I was wondering if it makes sense to add this to mapred-site.xml either at the top level (i.e conf/mapred-site.xml) or just for tests (i.e src/test/mapred-site.xml). I tired setting this property in src/test/mapred-site.xml but the test still failed. Somehow, we should make sure that the contrib tests load the src/test/mapred-site.xml.

          Show
          Amar Kamat added a comment - Tested the fix on TestCompressionEmulationUtils and the test passed. I was wondering if it makes sense to add this to mapred-site.xml either at the top level (i.e conf/mapred-site.xml ) or just for tests (i.e src/test/mapred-site.xml ). I tired setting this property in src/test/mapred-site.xml but the test still failed. Somehow, we should make sure that the contrib tests load the src/test/mapred-site.xml .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12508428/MAPREDUCE-3462.branch-0.23.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1498//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1498//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508428/MAPREDUCE-3462.branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1498//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1498//console This message is automatically generated.
          Ravi Prakash made changes -
          Component/s mrv2 [ 12314301 ]
          Ravi Prakash made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ravi Prakash made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Affects Version/s 0.23.0 [ 12315570 ]
          Affects Version/s 0.24.0 [ 12317654 ]
          Target Version/s 0.24.0 [ 12317654 ] 0.23.1 [ 12318883 ]
          Description When I run JUnit tests (e.g. TestGridmixSubmission), I see job submission failing with the following error:
          {noformat}
          java.lang.IllegalStateException: Variable substitution depth too large: 20 ${fs.default.name}
                  at org.apache.hadoop.conf.Configuration.substituteVars(Configuration.java:551)
                  at org.apache.hadoop.conf.Configuration.get(Configuration.java:569)
                  at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1020)
                  at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:564)
                  at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:353)
                  at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159)
                  at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:396)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
                  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156)
                  at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.launchGridmixJob(Gridmix.java:190)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.writeInputData(Gridmix.java:150)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.start(Gridmix.java:425)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.runJob(Gridmix.java:380)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.access$000(Gridmix.java:56)
                  at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:313)
                  at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:311)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:396)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.run(Gridmix.java:311)
          {noformat}
          When I run JUnit tests (e.g. TestDistCacheEmulation, TestSleepJob and TestCompressionEmulationUtils), I see job submission failing with the following error:
          {noformat}
          java.lang.IllegalStateException: Variable substitution depth too large: 20 ${fs.default.name}
                  at org.apache.hadoop.conf.Configuration.substituteVars(Configuration.java:551)
                  at org.apache.hadoop.conf.Configuration.get(Configuration.java:569)
                  at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1020)
                  at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:564)
                  at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:353)
                  at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1159)
                  at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1156)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:396)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
                  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1156)
                  at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1176)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.launchGridmixJob(Gridmix.java:190)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.writeInputData(Gridmix.java:150)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.start(Gridmix.java:425)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.runJob(Gridmix.java:380)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.access$000(Gridmix.java:56)
                  at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:313)
                  at org.apache.hadoop.mapred.gridmix.Gridmix$1.run(Gridmix.java:311)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:396)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
                  at org.apache.hadoop.mapred.gridmix.Gridmix.run(Gridmix.java:311)
          {noformat}
          Ravi Prakash made changes -
          Attachment MAPREDUCE-3462.branch-0.23.patch [ 12508428 ]
          Hide
          Ravi Prakash added a comment -

          Inside of Configuration.getStrings(), getProps().getProperty("mapreduce.job.hdfs-servers") returns "$fs.default.name".
          System.getProperty(fs.default.name) is returning the string "$

          {fs.default.name}

          " which continually is tried to be expanded causing the maximum depth exception.

          This patch simply sets mapreduce.job.hdfs-servers to an empty string so that it doesn't return $fs.default.name (as defined in yarn-default.xml)

          Show
          Ravi Prakash added a comment - Inside of Configuration.getStrings(), getProps().getProperty("mapreduce.job.hdfs-servers") returns "$fs.default.name". System.getProperty(fs.default.name) is returning the string "$ {fs.default.name} " which continually is tried to be expanded causing the maximum depth exception. This patch simply sets mapreduce.job.hdfs-servers to an empty string so that it doesn't return $fs.default.name (as defined in yarn-default.xml)
          Ravi Prakash made changes -
          Assignee Ravi Prakash [ raviprak ]
          Amar Kamat made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Amar Kamat made changes -
          Field Original Value New Value
          Link This issue relates to MAPREDUCE-3168 [ MAPREDUCE-3168 ]
          Amar Kamat created issue -

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Amar Kamat
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development