This is a patch against the trunk (revision 559315).

This tar ball contains the jar files for jetty-6.1.4. They should be put in the lib directory of hadoop. Also, pls remove the files/dirs corresponding to jetty5 - servlet-api.jar, jetty-5*, jetty-ext, from the lib directory.

By the way, in the patch, the default connector is set to SelectChannelConnector. One could also use the SocketConnector or the BlockingChannelConnector by setting up the hadoop config item jetty.connector appropriately (the value should be one of org.mortbay.jetty.bio.SocketConnector, org.mortbay.jetty.nio.BlockingChannelConnector, org.mortbay.jetty.nio.SelectChannelConnector). http://docs.codehaus.org/display/JETTY/Architecture has some insights into each of the connectors. Also, the patch switches Direct ByteBuffers off (due to the associated GC issues).

My testing shows that jetty6 is more friendly towards jdk1.6. Pls use jdk1.6 for running hadoop with jetty6.

I believe that this patch is now out of date with current build. I tried to apply it and got errors when attempting to run job tracker though I know it worked for me a month ago.

Submitting patch to proove that it is out of sync.

It would be great for me if this could get updated and committed soon. Thanks.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12362491/hadoop-jetty6.1.4-lib.tar.gz
against trunk revision r575578.

@author +1. The patch does not contain any @author tags.

patch -1. The patch command could not apply the patch.

Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/766/console

This message is automatically generated.

This patch is quite old and obviously needs to be updated. Let me see what needs to be done to have this patch working...

Updated to trunk and jetty-6.1.5. Did some cleanup based on findbugs/javac warnings in previous patch.

I didn't port the hbase code in my latest patch.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12369707/hadoop-1650-jetty6.1.5.patch
against trunk revision r595563.

@author +1. The patch does not contain any @author tags.

javadoc -1. The javadoc tool appears to have generated messages.

javac +1. The applied patch does not generate any new compiler warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests -1. The patch failed core unit tests.

contrib tests -1. The patch failed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1109/console

This message is automatically generated.

Test failed because hudson used the old jetty jar

+1 on updating hadoop to new jetty (I can take care of updating hbase).

Thanks Michael for the updated patch. Now we need to make sure that all hadoop MR benchmarks on big clusters run well before we commit this patch.

Cancelling PA for this one for now. Will PA once we test it on a big cluster and see that it works without issues.

Any news on this?

The patch failed to apply to trunk. Can you please attach a new patch that can work with the current trunk?

Attached is a patch updated to the current trunk. Stack, could you pls submit one for the hbase part? I am going to attach the lib directory that should be used in hadoop for jetty6.1.6 shortly.

This tar ball has the jar files required for hadoop + jetty6.1.6. The files/dirs that should be removed from the existing lib directory are:
1) jetty-5.1.4.jar
2) jetty-5.1.4.LICENSE.txt
3) jetty-ext
4) servlet-api.jar

I am in the process of benchmarking this on a big cluster ..

Here's a patch for hbase to make it work w/ jetty6. I tested that hbase servlets still work and that hbase tests pass.

Deveraj, a minor comment: you might want to check out how the '/stacks' servlet is being added in HttpStatusServer. By my count, its being added two times at least and probably three.

Good stuff.

Thanks Stack for pointing out the "stacks" issue. This is the merged patch.

I ran the sort benchmark on 500 nodes and here is the data:

1. Sort 500 using trunk:

• Random writer: 19 mins
• Sort: 79 mins
• Sort validation: 19 mins

2. Sort 500 using trunk + patch

• Random Writer: 24 mins
• Sort: 111 mins
• Sort validation: 17 mins

I see some errors/exceptions in the JT logs:

1. See a lot of these during the random writer job:
2007-12-28 22:31:02,456 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004256_0: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs
.NotReplicatedYetException: Not replicated yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004256_0/part-04256
2007-12-28 22:31:04,381 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004279_0: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs
.NotReplicatedYetException: Not replicated yet:/user/hadoopqa/sortBenchmark500/input/_task_200712282210_0001_m_004279_0/part-04279

2. See a log of these during the random writer job:
2007-12-28 22:31:57,715 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004485_0: java.net.SocketTimeoutException: timed out waiting for rpc r
esponse
2007-12-28 22:31:58,368 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0001_m_004272_0: java.net.SocketTimeoutException: timed out waiting for rpc r
esponse

3. See a couple of these during the sort job:
2007-12-28 22:37:43,783 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200712282210_0002_m_002442_0: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373)
at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)

I did a re-run on the 500 node cluster and don't see the NotReplicatedYetException and SocketTimeoutException. I do see the OutOfMemoryError when the sort job is running:

2008-01-11 20:54:10,375 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200801112025_0002_m_005337_0: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.BytesWritable.write(BytesWritable.java:137)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:373)
at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)

Here is the sort data:

The patch 1650-20080120 contains and updates with the current trunk the former patches jetty6.1.6.patch and jetty-hbase.patch.
Junit tests passed on my machine.

I'm running into compatibility problems with my other apps using jetty 6 because hadoop is still using jetty 5. Is there any chance this will be committed soon?

Attached is a patch w.r.t the trunk. This is under test/benchmark.

This is the lib dir for jetty-6.1.9 (note that the existing hadoop lib files/dirs - jetty-5.1.4*, jetty-ext, and, servlet-api should be removed).

Michael, the last time we benchmarked, the results were not satisfactory. Changing the jetty version is a big deal and we need to be absolutely sure about performance/reliability degradations since the current version in hadoop is pretty good in terms of reliability and reasonably ok in terms of performance...
So please bear with us while we test/benchmark to make sure that jetty6 is at least as good as jetty5... This might take some time since it involves testing on large clusters and that requires some co-ordination with other folks ...

This is an updated patch w.r.t the trunk. Mukund, could you please run some big benchmarks with this patch & hadoop-jetty6.1.9-lib.tar.gz. Pls run the benchmarks at least 4-5 times. Thanks!

Are we upgrading to Jetty 6 soon?

Hi, I'm the lead developer on Jetty and would be happy to help out with optimizing jetty-6 for hadoop.
I'm watching this issue, so just comment if you want my help.

Note that in an email conversation many months ago with somebody, I realized that one of your problems with performance was almost certainly that the default threadpool size had been reduced from 255 to 50 threads in jetty. This was because we are mostly using the NIO connector. But if hadoop is still using the socket connector, then it will need a larger threadpool

Merged patch with trunk, accommodating HADOOP-3854 and HADOOP-4284.

I used jetty-6.1.12.rc4 for this patch. Applying this patch, removing:

• lib/jetty-*
• lib/servlet-api.jar
  and adding
• lib/servlet-api-2.5-6.1.12.rc4.jar
• lib/jetty-6.1.12.rc4.jar
• lib/jsp-2.1
• lib/jetty-util-6.1.12.rc4.jar

Passes all unit tests on my machine. This is just a draft; it still needs to be benchmarked and the configuration of the Jetty components requires more thought.

I'm currently working on a Jetty extension that will better suit the Hadoop use case, namely serving ranges of very large files. The extension will wrap a RandomAccessFile instance as a Jetty buffer and thus will be able to be passed to in Http Generator within Jetty and
asynchronously flushed using FileChannel.transferTo(Channel)

This is mostly written now and we will be benchmarking this shortly.

Unfortunately, this will not make it into the Jetty 6.1.12 final release (out later today), but that does have some optimizations over 6.1.12.rc4

Merged patch with trunk.

The HdfsProxy code was recently checked into contrib. The _merge and _delta patches subclass ProxyHttpServer from HttpServer to collapse some common functionality (i.e. upgrading the former to use Jetty6, though this doesn't update the packaging). This may not be the appropriate solution for that component; it may require Jetty5. I'll leave that up to someone with more insight into its particular use cases. Some of the changes to HttpServer from these patches would do well merged back into the original, but that can wait until contrib is sorted out. If we move to Jetty6, Chukwa should also use the jars out of lib instead of keeping its own copies.

If mapred performance doesn't degrade using the basic upgrade- or if it only degrades slightly- we can move the TaskTracker optimization to a separate issue. We should also run gridmix to make sure we don't have any stability problems.

2 quick comments to the patch:

• Why change TestServletFilter? Not sycn with trunk yet?
• In TaskTracker.java, there are new imports to org.mortbay library. I think classes outside org.apache.hadoop.http should not refer to org.mortbay library. BTW, the new imports are not yet used. I guess you are in the middle of TaskTracker optimization.

Neither the TestServletFilter nor the TaskTracker changes are for inclusion; the former change was before HADOOP-4282 and the latter is just the remains of (discarded) scratch work. I just attached the patch for review of the Jetty6 approach and to resolve contrib conflicts. Once those are resolved I'll clean up the patch.

just a progress report on the work we are doing in jetty to support the hadoop use-case of sending large ranges from huge files. We've implemented a RandomAccessFileBuffer that allows NIO to efficiently send parts of files.
We've updated our HttpGenerator to be able to use this.

This is checked into jetty trunk and we are testing now.

In a week or two, we hope to have some benchmarks and some instructions as to how to use this with hadoop.

You can watch progress at http://jira.codehaus.org/browse/JETTY-748

cheers

@Chris,
> This may not be the appropriate solution for that component; it may require Jetty5.
Based on our current design for HdfsProxy, the servlet container shouldn't matter as the servlet interface is well defined. Please go ahead and upgrade HdfsProxy to use Jetty6.

Updated patch. I tested this using gridmix2 (HADOOP-3770) and found the performance to be comparable to trunk with the same configuration (387 nodes, 39m0s; trunk is about 36m with 400 nodes) using Jetty 6.1.14. It passes all unit tests, but generated findbugs warnings for non-serializable/transient fields (SE_BAD_FIELD) in the jsps. I made an attempt to update the eclipse classpath; if someone could verify that this works for them, that would be helpful.

Submitting this for review. Hudson will fail, since it doesn't have the Jetty6 jars.

+1 patch looks good to me except the indentation in .eclipse.templates/.classpath.

I think this needs some manual test for checking the web pages.

To test performance against Jetty5, I ran a 400 node shuffle test. 40 maps wrote 5GB each, feeding 400 reducers running on 385 nodes. Jetty6 finished in 684s, Jetty5 in 711; a negligible difference. In the attached graphs, each of the two runs is visible (two nodes in the cluster (tt1, tt2) that ran map jobs from both runs). The run on the left is Jetty6, the run on the right is Jetty5.

I just committed this.

Integrated in Hadoop-trunk #668 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/668/)
. Upgrade to Jetty 6.

In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false.
Any reason behind it?
Generally its better to use direct buffers as they are cheaper to send.

-dyu

In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false.
Any reason behind it?

Deveraj ran into some GC issues in the last attempted migration (above) and some previous experiences with direct buffers left me wary. For the initial checkin, it made more sense to track down any stability problems we run into without the direct buffer noise. We can certainly revisit it when considering other optimizations and I certainly wouldn't be against making it configurable.

Edit release note for publication.