Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MapReduce part of HADOOP-7450

      1. MAPREDUCE-3311.patch
        0.4 kB
        Konstantin Boudnik
      2. MAPREDUCE-3311.patch
        7 kB
        Konstantin Boudnik
      3. MAPREDUCE-3311.patch
        7 kB
        Konstantin Boudnik
      4. MAPREDUCE-3311.patch
        7 kB
        Konstantin Boudnik
      5. MAPREDUCE-3311.patch
        8 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Gavin made changes -
          Link This issue depends upon HADOOP-7450 [ HADOOP-7450 ]
          Gavin made changes -
          Link This issue depends on HADOOP-7450 [ HADOOP-7450 ]
          Gavin made changes -
          Link This issue depends upon HDFS-2513 [ HDFS-2513 ]
          Gavin made changes -
          Link This issue depends on HDFS-2513 [ HDFS-2513 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #89 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/89/)
          MAPREDUCE-3311. Bump jetty to 6.1.26. Controbuted by Konstantin Boudnik

          cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202990
          Files :

          • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/mapreduce/ivy/libraries.properties
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/ivy.xml
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/dynamic-scheduler/ivy.xml
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/fairscheduler/ivy.xml
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/gridmix/ivy.xml
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/mumak/ivy.xml
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/raid/ivy.xml
          • /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/streaming/ivy.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #89 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/89/ ) MAPREDUCE-3311 . Bump jetty to 6.1.26. Controbuted by Konstantin Boudnik cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202990 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/ivy/libraries.properties /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/ivy.xml /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/dynamic-scheduler/ivy.xml /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/fairscheduler/ivy.xml /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/gridmix/ivy.xml /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/mumak/ivy.xml /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/raid/ivy.xml /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/streaming/ivy.xml
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          I have just committed the patch.

          Show
          Konstantin Boudnik added a comment - I have just committed the patch.
          Hide
          Konstantin Shvachko added a comment -

          I checked TestRaidNode is fixed by adding commons-lang to raid/ivy.xml
          It was failing in the beginning, but turned out it was due to the ulimit. Increased ulimit - it passes now.
          +1 on latest patch.

          Show
          Konstantin Shvachko added a comment - I checked TestRaidNode is fixed by adding commons-lang to raid/ivy.xml It was failing in the beginning, but turned out it was due to the ulimit. Increased ulimit - it passes now. +1 on latest patch.
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-3311.patch [ 12503938 ]
          Hide
          Konstantin Boudnik added a comment -

          I am sorry - put in wrong patch last time ;(

          Show
          Konstantin Boudnik added a comment - I am sorry - put in wrong patch last time ;(
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-3311.patch [ 12503840 ]
          Hide
          Konstantin Boudnik added a comment -

          Damn transient deps. are keeping biting...
          With this patch in place I see two out of three tests start passing. However, I can't tell if adding commons-lang fixes the raid test because it is failing on my laptop on vanilla MR as well.

          Please take a look and if raid got fixed with the latest patch then we can commit it.

          Show
          Konstantin Boudnik added a comment - Damn transient deps. are keeping biting... With this patch in place I see two out of three tests start passing. However, I can't tell if adding commons-lang fixes the raid test because it is failing on my laptop on vanilla MR as well. Please take a look and if raid got fixed with the latest patch then we can commit it.
          Hide
          Konstantin Shvachko added a comment -

          I checked fairscheduler. It also needs common-lang. At least this fixed TestFairSchedulerSystem for me. We should add common-lang on the higher level, may be directly into mapred/ivy.xml

          Show
          Konstantin Shvachko added a comment - I checked fairscheduler. It also needs common-lang. At least this fixed TestFairSchedulerSystem for me. We should add common-lang on the higher level, may be directly into mapred/ivy.xml
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-3311.patch [ 12503839 ]
          Hide
          Konstantin Boudnik added a comment -

          It seems like (after having proper artifacts for all deps from common and hdfs in place) that MAPREDUCE-3396 is caused by the changes in transient dependencies. Thus I am joining it to the current patch.

          Show
          Konstantin Boudnik added a comment - It seems like (after having proper artifacts for all deps from common and hdfs in place) that MAPREDUCE-3396 is caused by the changes in transient dependencies. Thus I am joining it to the current patch.
          Hide
          Konstantin Boudnik added a comment -

          s/not a blocker/not be a blocker/

          Show
          Konstantin Boudnik added a comment - s/not a blocker/not be a blocker/
          Hide
          Konstantin Boudnik added a comment -

          wrt MAPREDUCE-3396: looks like issue was caused by some crap laying around in my ivy cache after working on Jetty patches. I will verify it again, but after all it might not a blocker.

          Show
          Konstantin Boudnik added a comment - wrt MAPREDUCE-3396 : looks like issue was caused by some crap laying around in my ivy cache after working on Jetty patches. I will verify it again, but after all it might not a blocker.
          Hide
          Konstantin Shvachko added a comment -

          The following three tests failed. But they also are failing without this patch.

              [junit] Test org.apache.hadoop.streaming.TestUlimit FAILED
              [junit] Test org.apache.hadoop.mapred.TestFairSchedulerSystem FAILED
              [junit] Test org.apache.hadoop.raid.TestRaidNode FAILED
          

          So I am going to commit the Jetty upgrade patches.

          Show
          Konstantin Shvachko added a comment - The following three tests failed. But they also are failing without this patch. [junit] Test org.apache.hadoop.streaming.TestUlimit FAILED [junit] Test org.apache.hadoop.mapred.TestFairSchedulerSystem FAILED [junit] Test org.apache.hadoop.raid.TestRaidNode FAILED So I am going to commit the Jetty upgrade patches.
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-3311.patch [ 12503553 ]
          Hide
          Konstantin Boudnik added a comment -

          Fixing contrib issues.
          Please be aware that this patch requires the fix for MAPREDUCE-3396

          Show
          Konstantin Boudnik added a comment - Fixing contrib issues. Please be aware that this patch requires the fix for MAPREDUCE-3396
          Konstantin Boudnik made changes -
          Link This issue is blocked by MAPREDUCE-3396 [ MAPREDUCE-3396 ]
          Hide
          Konstantin Shvachko added a comment -

          TestMiniMRChildTask passed for me. Actually all MR tests passed. But test-contrib is failing with

          [ivy:resolve]           ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve]           ::          UNRESOLVED DEPENDENCIES         ::
          [ivy:resolve]           ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve]           :: org.mortbay.jetty#jsp-api-2.1;6.1.26: not found
          [ivy:resolve]           :: org.mortbay.jetty#jsp-2.1;6.1.26: not found
          [ivy:resolve]           ::::::::::::::::::::::::::::::::::::::::::::::
          
          Show
          Konstantin Shvachko added a comment - TestMiniMRChildTask passed for me. Actually all MR tests passed. But test-contrib is failing with [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: org.mortbay.jetty#jsp-api-2.1;6.1.26: not found [ivy:resolve] :: org.mortbay.jetty#jsp-2.1;6.1.26: not found [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-3311.patch [ 12501482 ]
          Hide
          Konstantin Boudnik added a comment -

          I have ran smoke tests with all dependencies build for jetty 6.1.26 and I see that
          org.apache.hadoop.mapred.TestMiniMRChildTask tests have failed.
          Failures do not seem to be patch related though.

          Also, I see messages like

          2011-10-29 16:03:44,381 ERROR mapred.TaskTracker (TaskTracker.java:offerService(1582)) - Caught exception: java.io.IOException: Jetty problem. Jetty d
          idn't bind to a valid port
                  at org.apache.hadoop.mapred.TaskTracker.checkJettyPort(TaskTracker.java:1387)
                  at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1560)
                  at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2418)
                  at org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:229)
                  at java.lang.Thread.run(Thread.java:680)
          

          but tests are passing ok after them.

          Show
          Konstantin Boudnik added a comment - I have ran smoke tests with all dependencies build for jetty 6.1.26 and I see that org.apache.hadoop.mapred.TestMiniMRChildTask tests have failed. Failures do not seem to be patch related though. Also, I see messages like 2011-10-29 16:03:44,381 ERROR mapred.TaskTracker (TaskTracker.java:offerService(1582)) - Caught exception: java.io.IOException: Jetty problem. Jetty d idn't bind to a valid port at org.apache.hadoop.mapred.TaskTracker.checkJettyPort(TaskTracker.java:1387) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1560) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2418) at org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:229) at java.lang.Thread.run(Thread.java:680) but tests are passing ok after them.
          Konstantin Boudnik made changes -
          Link This issue depends on HADOOP-7450 [ HADOOP-7450 ]
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Link This issue depends on HDFS-2513 [ HDFS-2513 ]
          Konstantin Boudnik created issue -

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development