Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2445

TestMiniMRWithDFSWithDistinctUsers is very broken

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: security, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This test has a number of issues:

      • it side steps the normal job submission API for no apparent reason, manually writing splits file and uploading submission files. (but forgets to upload the job jar, so the jobs all fail)
      • it doesn't call waitForCompletion, or check job status (so it doesn't notice that the jobs all fail)
      • it doesn't verify in any way that the job output is owned by the user who supposedly ran the job
      • it shuts down DFS before MR

      These all conspire to make it pass, but it isn't actually testing anything.

      1. mapreduce-2445.txt
        11 kB
        Todd Lipcon
      2. mapreduce-2445.txt
        8 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Here's a fixed up test case:

          • renamed "wc" and "pi" users to "alice" and "bob", since in fact both are running wordcount
          • set up a proper staging directory with sticky bit, etc, so job can be successfully submitted
          • sets up home directories for each user that are only writable by them (to make sure that the job is running as the right users)
          • avoids using internal job submission APIs, so job actually submits
          • checks return status of job to actually assert success
          • shuts down services in correct order
          Show
          Todd Lipcon added a comment - Here's a fixed up test case: renamed "wc" and "pi" users to "alice" and "bob", since in fact both are running wordcount set up a proper staging directory with sticky bit, etc, so job can be successfully submitted sets up home directories for each user that are only writable by them (to make sure that the job is running as the right users) avoids using internal job submission APIs, so job actually submits checks return status of job to actually assert success shuts down services in correct order
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477071/mapreduce-2445.txt
          against trunk revision 1094093.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          -1 contrib tests. The patch failed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/177//testReport/
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/177//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477071/mapreduce-2445.txt against trunk revision 1094093. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/177//testReport/ Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/177//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Oops, uploaded an incomplete patch. Take 2.

          Show
          Todd Lipcon added a comment - Oops, uploaded an incomplete patch. Take 2.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477074/mapreduce-2445.txt
          against trunk revision 1094093.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/178//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/178//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/178//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477074/mapreduce-2445.txt against trunk revision 1094093. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/178//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/178//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/178//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and 22, thanks Tom.

          Show
          Todd Lipcon added a comment - Committed to trunk and 22, thanks Tom.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #647 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/647/)
          MAPREDUCE-2445. Fix TestMiniMRWithDFSWithDistinctUsers to be a valid test. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #647 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/647/ ) MAPREDUCE-2445 . Fix TestMiniMRWithDFSWithDistinctUsers to be a valid test. Contributed by Todd Lipcon.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #44 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #44 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #669 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/669/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #669 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/669/ )

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development