Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Upgraded all core servers to use Jetty 6

      Description

      This is the third attempt at moving to jetty6. Apparently, the jetty-6.1.4 has fixed some of the issues we discovered in jetty during HADOOP-736 and HADOOP-1273. I'd like to keep this issue open for sometime so that we have enough time to test out things.

      1. jetty6.1.4.patch
        15 kB
        Devaraj Das
      2. hadoop-jetty6.1.4-lib.tar.gz
        8.24 MB
        Devaraj Das
      3. hadoop-1650-jetty6.1.5.patch
        15 kB
        Michael Bieniosek
      4. jetty6.1.6.patch
        16 kB
        Devaraj Das
      5. hadoop-jetty6.1.6-lib.tar.gz
        6.17 MB
        Devaraj Das
      6. jetty-hbase.patch
        28 kB
        stack
      7. jetty-hadoop-6.1.6.patch
        44 kB
        Devaraj Das
      8. 1650-20080120.patch
        43 kB
        Christophe Taton
      9. 1650-jetty6.1.9.patch
        16 kB
        Devaraj Das
      10. hadoop-jetty6.1.9-lib.tar.gz
        5.27 MB
        Devaraj Das
      11. 1650-jetty6.1.9.v1.patch
        16 kB
        Devaraj Das
      12. 1650-0.patch
        46 kB
        Chris Douglas
      13. 1650-1_core.patch
        49 kB
        Chris Douglas
      14. 1650-1_proxy_delta.patch
        16 kB
        Chris Douglas
      15. 1650-1_merge.patch
        62 kB
        Chris Douglas
      16. 1650-2.patch
        63 kB
        Chris Douglas
      17. shuffleData.zip
        312 kB
        Chris Douglas

        Issue Links

          Activity

          Hide
          Robert Chansler added a comment -

          Edit release note for publication.

          Show
          Robert Chansler added a comment - Edit release note for publication.
          Hide
          Chris Douglas added a comment -

          In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false.
          Any reason behind it?

          Deveraj ran into some GC issues in the last attempted migration (above) and some previous experiences with direct buffers left me wary. For the initial checkin, it made more sense to track down any stability problems we run into without the direct buffer noise. We can certainly revisit it when considering other optimizations and I certainly wouldn't be against making it configurable.

          Show
          Chris Douglas added a comment - In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false. Any reason behind it? Deveraj ran into some GC issues in the last attempted migration (above) and some previous experiences with direct buffers left me wary. For the initial checkin, it made more sense to track down any stability problems we run into without the direct buffer noise. We can certainly revisit it when considering other optimizations and I certainly wouldn't be against making it configurable.
          Hide
          David Yu added a comment -

          In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false.
          Any reason behind it?
          Generally its better to use direct buffers as they are cheaper to send.

          -dyu

          Show
          David Yu added a comment - In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false. Any reason behind it? Generally its better to use direct buffers as they are cheaper to send. -dyu
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #668 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/668/)
          . Upgrade to Jetty 6.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #668 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/668/ ) . Upgrade to Jetty 6.
          Hide
          Chris Douglas added a comment -

          I just committed this.

          Show
          Chris Douglas added a comment - I just committed this.
          Hide
          Chris Douglas added a comment -

          To test performance against Jetty5, I ran a 400 node shuffle test. 40 maps wrote 5GB each, feeding 400 reducers running on 385 nodes. Jetty6 finished in 684s, Jetty5 in 711; a negligible difference. In the attached graphs, each of the two runs is visible (two nodes in the cluster (tt1, tt2) that ran map jobs from both runs). The run on the left is Jetty6, the run on the right is Jetty5.

          Filename Tracking Notes
          *net Network Nearly identical; included to show when the map task ended and when the shuffle began
          *mem Memory Neither Jetty5 nor Jetty6 had substantial footprints during the shuffle.
          *conn TCP connections The rate of failed connections in Jetty5, contrasted with Jetty6, is encouraging
          *cpu CPU Comparable profiles from both Jetty5 and Jetty6 on both nodes.
          *load Load Slightly higher load running Jetty5. The spike on tt1 running Jetty5 is worth mentioning.
          Show
          Chris Douglas added a comment - To test performance against Jetty5, I ran a 400 node shuffle test. 40 maps wrote 5GB each, feeding 400 reducers running on 385 nodes. Jetty6 finished in 684s, Jetty5 in 711; a negligible difference. In the attached graphs, each of the two runs is visible (two nodes in the cluster (tt1, tt2) that ran map jobs from both runs). The run on the left is Jetty6, the run on the right is Jetty5. Filename Tracking Notes *net Network Nearly identical; included to show when the map task ended and when the shuffle began *mem Memory Neither Jetty5 nor Jetty6 had substantial footprints during the shuffle. *conn TCP connections The rate of failed connections in Jetty5, contrasted with Jetty6, is encouraging *cpu CPU Comparable profiles from both Jetty5 and Jetty6 on both nodes. *load Load Slightly higher load running Jetty5. The spike on tt1 running Jetty5 is worth mentioning.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good to me except the indentation in .eclipse.templates/.classpath.

          I think this needs some manual test for checking the web pages.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good to me except the indentation in .eclipse.templates/.classpath. I think this needs some manual test for checking the web pages.
          Hide
          Chris Douglas added a comment -

          Submitting this for review. Hudson will fail, since it doesn't have the Jetty6 jars.

          Show
          Chris Douglas added a comment - Submitting this for review. Hudson will fail, since it doesn't have the Jetty6 jars.
          Hide
          Chris Douglas added a comment -

          Updated patch. I tested this using gridmix2 (HADOOP-3770) and found the performance to be comparable to trunk with the same configuration (387 nodes, 39m0s; trunk is about 36m with 400 nodes) using Jetty 6.1.14. It passes all unit tests, but generated findbugs warnings for non-serializable/transient fields (SE_BAD_FIELD) in the jsps. I made an attempt to update the eclipse classpath; if someone could verify that this works for them, that would be helpful.

          Show
          Chris Douglas added a comment - Updated patch. I tested this using gridmix2 ( HADOOP-3770 ) and found the performance to be comparable to trunk with the same configuration (387 nodes, 39m0s; trunk is about 36m with 400 nodes) using Jetty 6.1.14. It passes all unit tests, but generated findbugs warnings for non-serializable/transient fields ( SE_BAD_FIELD ) in the jsps. I made an attempt to update the eclipse classpath; if someone could verify that this works for them, that would be helpful.
          Hide
          Kan Zhang added a comment -

          @Chris,
          > This may not be the appropriate solution for that component; it may require Jetty5.
          Based on our current design for HdfsProxy, the servlet container shouldn't matter as the servlet interface is well defined. Please go ahead and upgrade HdfsProxy to use Jetty6.

          Show
          Kan Zhang added a comment - @Chris, > This may not be the appropriate solution for that component; it may require Jetty5. Based on our current design for HdfsProxy, the servlet container shouldn't matter as the servlet interface is well defined. Please go ahead and upgrade HdfsProxy to use Jetty6.
          Hide
          Greg Wilkins added a comment -

          just a progress report on the work we are doing in jetty to support the hadoop use-case of sending large ranges from huge files. We've implemented a RandomAccessFileBuffer that allows NIO to efficiently send parts of files.
          We've updated our HttpGenerator to be able to use this.

          This is checked into jetty trunk and we are testing now.

          In a week or two, we hope to have some benchmarks and some instructions as to how to use this with hadoop.

          You can watch progress at http://jira.codehaus.org/browse/JETTY-748

          cheers

          Show
          Greg Wilkins added a comment - just a progress report on the work we are doing in jetty to support the hadoop use-case of sending large ranges from huge files. We've implemented a RandomAccessFileBuffer that allows NIO to efficiently send parts of files. We've updated our HttpGenerator to be able to use this. This is checked into jetty trunk and we are testing now. In a week or two, we hope to have some benchmarks and some instructions as to how to use this with hadoop. You can watch progress at http://jira.codehaus.org/browse/JETTY-748 cheers
          Hide
          Chris Douglas added a comment -

          Neither the TestServletFilter nor the TaskTracker changes are for inclusion; the former change was before HADOOP-4282 and the latter is just the remains of (discarded) scratch work. I just attached the patch for review of the Jetty6 approach and to resolve contrib conflicts. Once those are resolved I'll clean up the patch.

          Show
          Chris Douglas added a comment - Neither the TestServletFilter nor the TaskTracker changes are for inclusion; the former change was before HADOOP-4282 and the latter is just the remains of (discarded) scratch work. I just attached the patch for review of the Jetty6 approach and to resolve contrib conflicts. Once those are resolved I'll clean up the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          2 quick comments to the patch:

          • Why change TestServletFilter? Not sycn with trunk yet?
          • In TaskTracker.java, there are new imports to org.mortbay library. I think classes outside org.apache.hadoop.http should not refer to org.mortbay library. BTW, the new imports are not yet used. I guess you are in the middle of TaskTracker optimization.
          Show
          Tsz Wo Nicholas Sze added a comment - 2 quick comments to the patch: Why change TestServletFilter? Not sycn with trunk yet? In TaskTracker.java, there are new imports to org.mortbay library. I think classes outside org.apache.hadoop.http should not refer to org.mortbay library. BTW, the new imports are not yet used. I guess you are in the middle of TaskTracker optimization.
          Hide
          Chris Douglas added a comment -

          Merged patch with trunk.

          The HdfsProxy code was recently checked into contrib. The _merge and _delta patches subclass ProxyHttpServer from HttpServer to collapse some common functionality (i.e. upgrading the former to use Jetty6, though this doesn't update the packaging). This may not be the appropriate solution for that component; it may require Jetty5. I'll leave that up to someone with more insight into its particular use cases. Some of the changes to HttpServer from these patches would do well merged back into the original, but that can wait until contrib is sorted out. If we move to Jetty6, Chukwa should also use the jars out of lib instead of keeping its own copies.

          If mapred performance doesn't degrade using the basic upgrade- or if it only degrades slightly- we can move the TaskTracker optimization to a separate issue. We should also run gridmix to make sure we don't have any stability problems.

          Show
          Chris Douglas added a comment - Merged patch with trunk. The HdfsProxy code was recently checked into contrib. The _merge and _delta patches subclass ProxyHttpServer from HttpServer to collapse some common functionality (i.e. upgrading the former to use Jetty6, though this doesn't update the packaging). This may not be the appropriate solution for that component; it may require Jetty5. I'll leave that up to someone with more insight into its particular use cases. Some of the changes to HttpServer from these patches would do well merged back into the original, but that can wait until contrib is sorted out. If we move to Jetty6, Chukwa should also use the jars out of lib instead of keeping its own copies. If mapred performance doesn't degrade using the basic upgrade- or if it only degrades slightly- we can move the TaskTracker optimization to a separate issue. We should also run gridmix to make sure we don't have any stability problems.
          Hide
          Greg Wilkins added a comment -

          I'm currently working on a Jetty extension that will better suit the Hadoop use case, namely serving ranges of very large files. The extension will wrap a RandomAccessFile instance as a Jetty buffer and thus will be able to be passed to in Http Generator within Jetty and
          asynchronously flushed using FileChannel.transferTo(Channel)

          This is mostly written now and we will be benchmarking this shortly.

          Unfortunately, this will not make it into the Jetty 6.1.12 final release (out later today), but that does have some optimizations over 6.1.12.rc4

          Show
          Greg Wilkins added a comment - I'm currently working on a Jetty extension that will better suit the Hadoop use case, namely serving ranges of very large files. The extension will wrap a RandomAccessFile instance as a Jetty buffer and thus will be able to be passed to in Http Generator within Jetty and asynchronously flushed using FileChannel.transferTo(Channel) This is mostly written now and we will be benchmarking this shortly. Unfortunately, this will not make it into the Jetty 6.1.12 final release (out later today), but that does have some optimizations over 6.1.12.rc4
          Hide
          Chris Douglas added a comment -

          Merged patch with trunk, accommodating HADOOP-3854 and HADOOP-4284.

          I used jetty-6.1.12.rc4 for this patch. Applying this patch, removing:

          • lib/jetty-*
          • lib/servlet-api.jar
            and adding
          • lib/servlet-api-2.5-6.1.12.rc4.jar
          • lib/jetty-6.1.12.rc4.jar
          • lib/jsp-2.1
          • lib/jetty-util-6.1.12.rc4.jar

          Passes all unit tests on my machine. This is just a draft; it still needs to be benchmarked and the configuration of the Jetty components requires more thought.

          Show
          Chris Douglas added a comment - Merged patch with trunk, accommodating HADOOP-3854 and HADOOP-4284 . I used jetty-6.1.12.rc4 for this patch. Applying this patch, removing: lib/jetty-* lib/servlet-api.jar and adding lib/servlet-api-2.5-6.1.12.rc4.jar lib/jetty-6.1.12.rc4.jar lib/jsp-2.1 lib/jetty-util-6.1.12.rc4.jar Passes all unit tests on my machine. This is just a draft; it still needs to be benchmarked and the configuration of the Jetty components requires more thought.
          Hide
          Greg Wilkins added a comment -

          Note that in an email conversation many months ago with somebody, I realized that one of your problems with performance was almost certainly that the default threadpool size had been reduced from 255 to 50 threads in jetty. This was because we are mostly using the NIO connector. But if hadoop is still using the socket connector, then it will need a larger threadpool

          Show
          Greg Wilkins added a comment - Note that in an email conversation many months ago with somebody, I realized that one of your problems with performance was almost certainly that the default threadpool size had been reduced from 255 to 50 threads in jetty. This was because we are mostly using the NIO connector. But if hadoop is still using the socket connector, then it will need a larger threadpool
          Hide
          Greg Wilkins added a comment -

          Hi, I'm the lead developer on Jetty and would be happy to help out with optimizing jetty-6 for hadoop.
          I'm watching this issue, so just comment if you want my help.

          Show
          Greg Wilkins added a comment - Hi, I'm the lead developer on Jetty and would be happy to help out with optimizing jetty-6 for hadoop. I'm watching this issue, so just comment if you want my help.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Are we upgrading to Jetty 6 soon?

          Show
          Tsz Wo Nicholas Sze added a comment - Are we upgrading to Jetty 6 soon?
          Hide
          Devaraj Das added a comment -

          This is an updated patch w.r.t the trunk. Mukund, could you please run some big benchmarks with this patch & hadoop-jetty6.1.9-lib.tar.gz. Pls run the benchmarks at least 4-5 times. Thanks!

          Show
          Devaraj Das added a comment - This is an updated patch w.r.t the trunk. Mukund, could you please run some big benchmarks with this patch & hadoop-jetty6.1.9-lib.tar.gz. Pls run the benchmarks at least 4-5 times. Thanks!
          Hide
          Devaraj Das added a comment -

          Michael, the last time we benchmarked, the results were not satisfactory. Changing the jetty version is a big deal and we need to be absolutely sure about performance/reliability degradations since the current version in hadoop is pretty good in terms of reliability and reasonably ok in terms of performance...
          So please bear with us while we test/benchmark to make sure that jetty6 is at least as good as jetty5... This might take some time since it involves testing on large clusters and that requires some co-ordination with other folks ...

          Show
          Devaraj Das added a comment - Michael, the last time we benchmarked, the results were not satisfactory. Changing the jetty version is a big deal and we need to be absolutely sure about performance/reliability degradations since the current version in hadoop is pretty good in terms of reliability and reasonably ok in terms of performance... So please bear with us while we test/benchmark to make sure that jetty6 is at least as good as jetty5... This might take some time since it involves testing on large clusters and that requires some co-ordination with other folks ...
          Hide
          Devaraj Das added a comment -

          This is the lib dir for jetty-6.1.9 (note that the existing hadoop lib files/dirs - jetty-5.1.4*, jetty-ext, and, servlet-api should be removed).

          Show
          Devaraj Das added a comment - This is the lib dir for jetty-6.1.9 (note that the existing hadoop lib files/dirs - jetty-5.1.4*, jetty-ext, and, servlet-api should be removed).
          Hide
          Devaraj Das added a comment -

          Attached is a patch w.r.t the trunk. This is under test/benchmark.

          Show
          Devaraj Das added a comment - Attached is a patch w.r.t the trunk. This is under test/benchmark.
          Hide
          Michael Bieniosek added a comment -

          I'm running into compatibility problems with my other apps using jetty 6 because hadoop is still using jetty 5. Is there any chance this will be committed soon?

          Show
          Michael Bieniosek added a comment - I'm running into compatibility problems with my other apps using jetty 6 because hadoop is still using jetty 5. Is there any chance this will be committed soon?
          Hide
          Christophe Taton added a comment -

          The patch 1650-20080120 contains and updates with the current trunk the former patches jetty6.1.6.patch and jetty-hbase.patch.
          Junit tests passed on my machine.

          Show
          Christophe Taton added a comment - The patch 1650-20080120 contains and updates with the current trunk the former patches jetty6.1.6.patch and jetty-hbase.patch. Junit tests passed on my machine.
          Hide
          Mukund Madhugiri added a comment -

          I did a re-run on the 500 node cluster and don't see the NotReplicatedYetException and SocketTimeoutException. I do see the OutOfMemoryError when the sort job is running:

          2008-01-11 20:54:10,375 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200801112025_0002_m_005337_0: java.lang.OutOfMemoryError: Java heap space
          at java.util.Arrays.copyOf(Arrays.java:2786)
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
          at java.io.DataOutputStream.write(DataOutputStream.java:90)
          at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373)
          at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
          at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
          at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)

          Here is the sort data:

          500 nodes trunk trunk + patch
          randomWriter (mins) 27 25
          sort (mins) 86 107
          sortValidation (mins) 22 21
          Show
          Mukund Madhugiri added a comment - I did a re-run on the 500 node cluster and don't see the NotReplicatedYetException and SocketTimeoutException. I do see the OutOfMemoryError when the sort job is running: 2008-01-11 20:54:10,375 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200801112025_0002_m_005337_0: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373) at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043) Here is the sort data: 500 nodes trunk trunk + patch randomWriter (mins) 27 25 sort (mins) 86 107 sortValidation (mins) 22 21
          Hide
          Mukund Madhugiri added a comment -

          I ran the sort benchmark on 500 nodes and here is the data:

          1. Sort 500 using trunk:

          • Random writer: 19 mins
          • Sort: 79 mins
          • Sort validation: 19 mins

          2. Sort 500 using trunk + patch

          • Random Writer: 24 mins
          • Sort: 111 mins
          • Sort validation: 17 mins

          I see some errors/exceptions in the JT logs:

          1. See a lot of these during the random writer job:
          2007-12-28 22:31:02,456 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004256_0: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs
          .NotReplicatedYetException: Not replicated yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004256_0/part-04256
          2007-12-28 22:31:04,381 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004279_0: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs
          .NotReplicatedYetException: Not replicated yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004279_0/part-04279

          2. See a log of these during the random writer job:
          2007-12-28 22:31:57,715 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004485_0: java.net.SocketTimeoutException: timed out waiting for rpc r
          esponse
          2007-12-28 22:31:58,368 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004272_0: java.net.SocketTimeoutException: timed out waiting for rpc r
          esponse

          3. See a couple of these during the sort job:
          2007-12-28 22:37:43,783 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0002_m_002442_0: java.lang.OutOfMemoryError: Java heap space
          at java.util.Arrays.copyOf(Arrays.java:2786)
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
          at java.io.DataOutputStream.write(DataOutputStream.java:90)
          at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373)
          at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
          at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
          at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)

          Show
          Mukund Madhugiri added a comment - I ran the sort benchmark on 500 nodes and here is the data: 1. Sort 500 using trunk: Random writer: 19 mins Sort: 79 mins Sort validation: 19 mins 2. Sort 500 using trunk + patch Random Writer: 24 mins Sort: 111 mins Sort validation: 17 mins I see some errors/exceptions in the JT logs: 1. See a lot of these during the random writer job: 2007-12-28 22:31:02,456 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004256_0: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs .NotReplicatedYetException: Not replicated yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004256_0/part-04256 2007-12-28 22:31:04,381 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004279_0: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs .NotReplicatedYetException: Not replicated yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004279_0/part-04279 2. See a log of these during the random writer job: 2007-12-28 22:31:57,715 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004485_0: java.net.SocketTimeoutException: timed out waiting for rpc r esponse 2007-12-28 22:31:58,368 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004272_0: java.net.SocketTimeoutException: timed out waiting for rpc r esponse 3. See a couple of these during the sort job: 2007-12-28 22:37:43,783 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0002_m_002442_0: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373) at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
          Hide
          Devaraj Das added a comment -

          Thanks Stack for pointing out the "stacks" issue. This is the merged patch.

          Show
          Devaraj Das added a comment - Thanks Stack for pointing out the "stacks" issue. This is the merged patch.
          Hide
          stack added a comment -

          Here's a patch for hbase to make it work w/ jetty6. I tested that hbase servlets still work and that hbase tests pass.

          Deveraj, a minor comment: you might want to check out how the '/stacks' servlet is being added in HttpStatusServer. By my count, its being added two times at least and probably three.

          Good stuff.

          Show
          stack added a comment - Here's a patch for hbase to make it work w/ jetty6. I tested that hbase servlets still work and that hbase tests pass. Deveraj, a minor comment: you might want to check out how the '/stacks' servlet is being added in HttpStatusServer. By my count, its being added two times at least and probably three. Good stuff.
          Hide
          Devaraj Das added a comment -

          This tar ball has the jar files required for hadoop + jetty6.1.6. The files/dirs that should be removed from the existing lib directory are:
          1) jetty-5.1.4.jar
          2) jetty-5.1.4.LICENSE.txt
          3) jetty-ext
          4) servlet-api.jar

          I am in the process of benchmarking this on a big cluster ..

          Show
          Devaraj Das added a comment - This tar ball has the jar files required for hadoop + jetty6.1.6. The files/dirs that should be removed from the existing lib directory are: 1) jetty-5.1.4.jar 2) jetty-5.1.4.LICENSE.txt 3) jetty-ext 4) servlet-api.jar I am in the process of benchmarking this on a big cluster ..
          Hide
          Devaraj Das added a comment -

          Attached is a patch updated to the current trunk. Stack, could you pls submit one for the hbase part? I am going to attach the lib directory that should be used in hadoop for jetty6.1.6 shortly.

          Show
          Devaraj Das added a comment - Attached is a patch updated to the current trunk. Stack, could you pls submit one for the hbase part? I am going to attach the lib directory that should be used in hadoop for jetty6.1.6 shortly.
          Hide
          Mukund Madhugiri added a comment -

          The patch failed to apply to trunk. Can you please attach a new patch that can work with the current trunk?

          Show
          Mukund Madhugiri added a comment - The patch failed to apply to trunk. Can you please attach a new patch that can work with the current trunk?
          Hide
          Michael Bieniosek added a comment -

          Any news on this?

          Show
          Michael Bieniosek added a comment - Any news on this?
          Hide
          Devaraj Das added a comment -

          Cancelling PA for this one for now. Will PA once we test it on a big cluster and see that it works without issues.

          Show
          Devaraj Das added a comment - Cancelling PA for this one for now. Will PA once we test it on a big cluster and see that it works without issues.
          Hide
          Devaraj Das added a comment -

          Thanks Michael for the updated patch. Now we need to make sure that all hadoop MR benchmarks on big clusters run well before we commit this patch.

          Show
          Devaraj Das added a comment - Thanks Michael for the updated patch. Now we need to make sure that all hadoop MR benchmarks on big clusters run well before we commit this patch.
          Hide
          stack added a comment -

          +1 on updating hadoop to new jetty (I can take care of updating hbase).

          Show
          stack added a comment - +1 on updating hadoop to new jetty (I can take care of updating hbase).
          Hide
          Michael Bieniosek added a comment -

          Test failed because hudson used the old jetty jar

          Show
          Michael Bieniosek added a comment - Test failed because hudson used the old jetty jar
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12369707/hadoop-1650-jetty6.1.5.patch
          against trunk revision r595563.

          @author +1. The patch does not contain any @author tags.

          javadoc -1. The javadoc tool appears to have generated messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests -1. The patch failed contrib unit tests.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/testReport/
          Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12369707/hadoop-1650-jetty6.1.5.patch against trunk revision r595563. @author +1. The patch does not contain any @author tags. javadoc -1. The javadoc tool appears to have generated messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/console This message is automatically generated.
          Hide
          Michael Bieniosek added a comment -

          I didn't port the hbase code in my latest patch.

          Show
          Michael Bieniosek added a comment - I didn't port the hbase code in my latest patch.
          Hide
          Michael Bieniosek added a comment -

          Updated to trunk and jetty-6.1.5. Did some cleanup based on findbugs/javac warnings in previous patch.

          Show
          Michael Bieniosek added a comment - Updated to trunk and jetty-6.1.5. Did some cleanup based on findbugs/javac warnings in previous patch.
          Hide
          Devaraj Das added a comment -

          This patch is quite old and obviously needs to be updated. Let me see what needs to be done to have this patch working...

          Show
          Devaraj Das added a comment - This patch is quite old and obviously needs to be updated. Let me see what needs to be done to have this patch working...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12362491/hadoop-jetty6.1.4-lib.tar.gz
          against trunk revision r575578.

          @author +1. The patch does not contain any @author tags.

          patch -1. The patch command could not apply the patch.

          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/766/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12362491/hadoop-jetty6.1.4-lib.tar.gz against trunk revision r575578. @author +1. The patch does not contain any @author tags. patch -1. The patch command could not apply the patch. Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/766/console This message is automatically generated.
          Hide
          James Kennedy added a comment -

          I believe that this patch is now out of date with current build. I tried to apply it and got errors when attempting to run job tracker though I know it worked for me a month ago.

          Submitting patch to proove that it is out of sync.

          It would be great for me if this could get updated and committed soon. Thanks.

          Show
          James Kennedy added a comment - I believe that this patch is now out of date with current build. I tried to apply it and got errors when attempting to run job tracker though I know it worked for me a month ago. Submitting patch to proove that it is out of sync. It would be great for me if this could get updated and committed soon. Thanks.
          Hide
          Devaraj Das added a comment -

          My testing shows that jetty6 is more friendly towards jdk1.6. Pls use jdk1.6 for running hadoop with jetty6.

          Show
          Devaraj Das added a comment - My testing shows that jetty6 is more friendly towards jdk1.6. Pls use jdk1.6 for running hadoop with jetty6.
          Hide
          Devaraj Das added a comment -

          By the way, in the patch, the default connector is set to SelectChannelConnector. One could also use the SocketConnector or the BlockingChannelConnector by setting up the hadoop config item jetty.connector appropriately (the value should be one of org.mortbay.jetty.bio.SocketConnector, org.mortbay.jetty.nio.BlockingChannelConnector, org.mortbay.jetty.nio.SelectChannelConnector). http://docs.codehaus.org/display/JETTY/Architecture has some insights into each of the connectors. Also, the patch switches Direct ByteBuffers off (due to the associated GC issues).

          Show
          Devaraj Das added a comment - By the way, in the patch, the default connector is set to SelectChannelConnector. One could also use the SocketConnector or the BlockingChannelConnector by setting up the hadoop config item jetty.connector appropriately (the value should be one of org.mortbay.jetty.bio.SocketConnector, org.mortbay.jetty.nio.BlockingChannelConnector, org.mortbay.jetty.nio.SelectChannelConnector). http://docs.codehaus.org/display/JETTY/Architecture has some insights into each of the connectors. Also, the patch switches Direct ByteBuffers off (due to the associated GC issues).
          Hide
          Devaraj Das added a comment -

          This tar ball contains the jar files for jetty-6.1.4. They should be put in the lib directory of hadoop. Also, pls remove the files/dirs corresponding to jetty5 - servlet-api.jar, jetty-5*, jetty-ext, from the lib directory.

          Show
          Devaraj Das added a comment - This tar ball contains the jar files for jetty-6.1.4. They should be put in the lib directory of hadoop. Also, pls remove the files/dirs corresponding to jetty5 - servlet-api.jar, jetty-5*, jetty-ext, from the lib directory.
          Hide
          Devaraj Das added a comment -

          This is a patch against the trunk (revision 559315).

          Show
          Devaraj Das added a comment - This is a patch against the trunk (revision 559315).

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Devaraj Das
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development