Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5136

TestJobImpl->testJobNoTasks fails with IBM JAVA

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Environment:

      Linux + IBM JAVA 6

    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      I am not sure if this is a testcase or a design issue. During execution of TestJobImpl->testJobNoTasks() there is an assertion made based on the order of key/value pairs stored in adjacency list. However adjacency list was created by Configuration->getValByRegex() as a HashMap (order is not guaranteed):

      Testcase:
      JobSubmittedEventHandler jseHandler = new JobSubmittedEventHandler("testId",
      "testName", "testNodeName", "\"key2\"=\"value2\" \"key1\"=\"value1\" ");
      ....
      ....
      try {
      Assert.assertTrue(jseHandler.getAssertValue()); <===

      Configuration->getValByRegex():
      public Map<String,String> getValByRegex(String regex) {
      Pattern p = Pattern.compile(regex);
      Map<String,String> result = new HashMap<String,String>(); <=======

      as we all know, HashMap makes absolutely no guarantees about the iteration order. It can (and will) even change completely when new elements are added.

      Changing HashMap to LinkedHashMap fixes the ordering inconsistency, however with a small performance side effect.

        Activity

        Amir Sanjar created issue -
        Amir Sanjar made changes -
        Field Original Value New Value
        Attachment HADOOP-9465.patch [ 12577428 ]
        Hide
        Billie Rinaldi added a comment -

        I'm not sure why LinkedHashMap would necessarily solve this issue, since the order in which entries are put into the map is the order in which they are retrieved from the Properties Hashtable.

        Currently the string generated by JobImpl.getWorkflowAdjacencies is not intended to have any particular ordering. Here are some other possible ways to address the test issue:
        1) Make getWorkflowAdjacencies apply an ordering (e.g. putting the entries obtained by Configuration.getValByRegex into a sorted map instead of just using the HashMap it returns, presumably also with an effect on performance).
        2) Instead of testing an exact string with an assumed ordering, split the string on spaces and ensure each element appears as expected.
        3) Only test a single adjacency entry to avoid the ordering issue entirely.

        Show
        Billie Rinaldi added a comment - I'm not sure why LinkedHashMap would necessarily solve this issue, since the order in which entries are put into the map is the order in which they are retrieved from the Properties Hashtable. Currently the string generated by JobImpl.getWorkflowAdjacencies is not intended to have any particular ordering. Here are some other possible ways to address the test issue: 1) Make getWorkflowAdjacencies apply an ordering (e.g. putting the entries obtained by Configuration.getValByRegex into a sorted map instead of just using the HashMap it returns, presumably also with an effect on performance). 2) Instead of testing an exact string with an assumed ordering, split the string on spaces and ensure each element appears as expected. 3) Only test a single adjacency entry to avoid the ordering issue entirely.
        Hide
        Amir Sanjar added a comment -

        Billie, thanks for your reply. I will make appropriate changes, as you have suggested.

        Show
        Amir Sanjar added a comment - Billie, thanks for your reply. I will make appropriate changes, as you have suggested.
        Hide
        Amir Sanjar added a comment -

        I have made proper changes in TestJobImpl testcase. Is there away to transfer this JIRA to MAPREDUCE component?

        Show
        Amir Sanjar added a comment - I have made proper changes in TestJobImpl testcase. Is there away to transfer this JIRA to MAPREDUCE component?
        Sandy Ryza made changes -
        Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
        Key HADOOP-9465 MAPREDUCE-5136
        Affects Version/s 2.0.3-alpha [ 12323275 ]
        Affects Version/s 2.0.3-alpha [ 12323273 ]
        Component/s conf [ 12310711 ]
        Amir Sanjar made changes -
        Attachment HADOOP-9465.patch [ 12577428 ]
        Amir Sanjar made changes -
        Attachment MAPREDUCE-5136.patch [ 12577609 ]
        Hide
        Amir Sanjar added a comment -

        patch uploaded

        Show
        Amir Sanjar added a comment - patch uploaded
        Hide
        Jason Lowe added a comment -

        When the patch is ready for review, please move the JIRA into the Patch Available state via the "Submit Patch" button. That triggers the QA bot to comment on the patch and also flags the JIRA as being in a ready-to-review state for committers.

        Also if you need to move JIRAs to other projects in the future, you can do this via the "Move" option under the "More Actions" drop-down menu.

        Show
        Jason Lowe added a comment - When the patch is ready for review, please move the JIRA into the Patch Available state via the "Submit Patch" button. That triggers the QA bot to comment on the patch and also flags the JIRA as being in a ready-to-review state for committers. Also if you need to move JIRAs to other projects in the future, you can do this via the "Move" option under the "More Actions" drop-down menu.
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 2.0.5-beta [ 12324032 ]
        Assignee Amir Sanjar [ asanjar ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12577609/MAPREDUCE-5136.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3509//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3509//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12577609/MAPREDUCE-5136.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3509//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3509//console This message is automatically generated.
        Hide
        Billie Rinaldi added a comment -

        Also if you need to move JIRAs to other projects in the future, you can do this via the "Move" option under the "More Actions" drop-down menu.

        I think this is only possible if you are a contributor to the ticket's current project.

        Show
        Billie Rinaldi added a comment - Also if you need to move JIRAs to other projects in the future, you can do this via the "Move" option under the "More Actions" drop-down menu. I think this is only possible if you are a contributor to the ticket's current project.
        Hide
        Amir Sanjar added a comment -

        do we know into which build (release) this patch will be included? This patch is needed for IBM JAVA . thanks

        Show
        Amir Sanjar added a comment - do we know into which build (release) this patch will be included? This patch is needed for IBM JAVA . thanks
        Hide
        Billie Rinaldi added a comment -

        Amir, unfortunately I can't answer your question as I'm not a committer. But I wanted to ask, if you get a chance, could you try building a new checkout of branch-1 with IBM Java? It has a different test for workflow adjacencies that assumes if you iterate over a HashMap multiple times (without adding new elements) you will get the keys in the same order.

        Show
        Billie Rinaldi added a comment - Amir, unfortunately I can't answer your question as I'm not a committer. But I wanted to ask, if you get a chance, could you try building a new checkout of branch-1 with IBM Java? It has a different test for workflow adjacencies that assumes if you iterate over a HashMap multiple times (without adding new elements) you will get the keys in the same order.
        Hide
        Amir Sanjar added a comment -

        sure Billie, I will try that.

        Show
        Amir Sanjar added a comment - sure Billie, I will try that.
        Amir Sanjar made changes -
        Summary TestJobImpl->testJobNoTasks fails .. TestJobImpl->testJobNoTasks fails with IBM JAVA ..
        Hide
        Amir Sanjar added a comment -

        Billi, I assumed the intended testcase in branch_1 was TestJobHistory, was that correct? All JUnit tests, including validateJobHistoryFileContent() passed successfully with IBM JAVA.

        Show
        Amir Sanjar added a comment - Billi, I assumed the intended testcase in branch_1 was TestJobHistory, was that correct? All JUnit tests, including validateJobHistoryFileContent() passed successfully with IBM JAVA.
        Hide
        Billie Rinaldi added a comment -

        Yes, that's correct. Thanks for testing that!

        Show
        Billie Rinaldi added a comment - Yes, that's correct. Thanks for testing that!
        Jason Lowe made changes -
        Summary TestJobImpl->testJobNoTasks fails with IBM JAVA .. TestJobImpl->testJobNoTasks fails with IBM JAVA
        Hide
        Jason Lowe added a comment -

        +1, looks good to me. I'll commit this shortly.

        Show
        Jason Lowe added a comment - +1, looks good to me. I'll commit this shortly.
        Hide
        Jason Lowe added a comment -

        Thanks, Amir! I committed this to trunk and branch-2.

        Show
        Jason Lowe added a comment - Thanks, Amir! I committed this to trunk and branch-2.
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.0.5-beta [ 12324032 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3595 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3595/)
        MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3595 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3595/ ) MAPREDUCE-5136 . TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/181/)
        MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/181/ ) MAPREDUCE-5136 . TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1370 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1370/)
        MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061)

        Result = FAILURE
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1370 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1370/ ) MAPREDUCE-5136 . TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1397 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1397/)
        MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1397 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1397/ ) MAPREDUCE-5136 . TestJobImpl->testJobNoTasks fails with IBM JAVA. Contributed by Amir Sanjar (Revision 1467061) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467061 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Amir Sanjar
            Reporter:
            Amir Sanjar
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development