Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2980

Fetch failures and other related issues in Jetty 6.1.26

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.205.0, 0.23.0
    • Fix Version/s: None
    • Component/s: tasktracker
    • Labels:
      None

      Description

      Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related issues, including:

      • Much higher incidence of fetch failures
      • A few strange file-descriptor related bugs (eg MAPREDUCE-2389)
      • A few unexplained issues where long "fsck"s on the NameNode drop out halfway through with a ClosedChannelException

      Stress tests with 10000Map x 10000Reduce sleep jobs reliably reproduce fetch failures at a rate of about 1 per million on a 25 node test cluster. These problems are all new since the upgrade from 6.1.14.

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          I've been working with the Jetty folks on this, and they pointed me at an experimental branch (6.1.22-z6) which has some hacks in the NIO parts that seem to prevent the issue. I have verified that the 10,000 map by 10,000 reduce job completes with no fetch failures on the test cluster. In fact, no fetch failures after 20+ runs of this job.

          They're thinking about merging this branch and calling it 6.1.27. At that point we could upgrade Hadoop to use 6.1.27. I'd also like to consider an alternate release (6.1.26.hadoop.1) which is a build I've prepared by simply patching 6.1.26 with only the NIO changes, since the planned 6.1.27 contains a number of other unrelated changes. It may make sense to include this custom patch build in the maintenance release series (20x) if we are concerned by any of the other Jetty changes not having had enough time to bake.

          Show
          Todd Lipcon added a comment - I've been working with the Jetty folks on this, and they pointed me at an experimental branch (6.1.22-z6) which has some hacks in the NIO parts that seem to prevent the issue. I have verified that the 10,000 map by 10,000 reduce job completes with no fetch failures on the test cluster. In fact, no fetch failures after 20+ runs of this job. They're thinking about merging this branch and calling it 6.1.27. At that point we could upgrade Hadoop to use 6.1.27. I'd also like to consider an alternate release (6.1.26.hadoop.1) which is a build I've prepared by simply patching 6.1.26 with only the NIO changes, since the planned 6.1.27 contains a number of other unrelated changes. It may make sense to include this custom patch build in the maintenance release series (20x) if we are concerned by any of the other Jetty changes not having had enough time to bake.
          Hide
          Todd Lipcon added a comment -

          FWIW, I ran 150 sleep jobs over the weekend, each with 10,000 map by 10,000 reduce. Didn't see any fetch failures, and none of the TTs got stuck in the "spinning" state.

          The branch I tested is here: https://github.com/toddlipcon/jetty-hadoop-fix

          Show
          Todd Lipcon added a comment - FWIW, I ran 150 sleep jobs over the weekend, each with 10,000 map by 10,000 reduce. Didn't see any fetch failures, and none of the TTs got stuck in the "spinning" state. The branch I tested is here: https://github.com/toddlipcon/jetty-hadoop-fix
          Hide
          Arun C Murthy added a comment -

          Todd, thanks for digging into this!

          I'm happy to upgrade to jetty-6.1.27, but don't see much of a point with running our custom builds. Also, with MAPREDUCE-2524 this becomes less critical - thus, I think we should just wait for jetty-1.6.27.

          Show
          Arun C Murthy added a comment - Todd, thanks for digging into this! I'm happy to upgrade to jetty-6.1.27, but don't see much of a point with running our custom builds. Also, with MAPREDUCE-2524 this becomes less critical - thus, I think we should just wait for jetty-1.6.27.
          Hide
          Todd Lipcon added a comment -

          I agree it's less critical for the shuffle, but we're also seeing an issue where the NN drops an HTTP connection in the middle of long fsck response, in particular when it's under other load (eg big checkpoints). It's really spurious and hard to reproduce, but we have some inkling that it's related to this issue.

          I've been bugging the jetty folks about timeline for a 6.1.27 release, but it may be a couple months off. I figured that 6.1.26".1" would be an interim solution for 205 and/or 206 until a 6.1.27 release is ready and can be QAed. Are you -1 or just not wild about it? FWIW the patch is not a custom change, bur rather just the NIO-related changes that will be integrated for 6.1.27.

          Show
          Todd Lipcon added a comment - I agree it's less critical for the shuffle, but we're also seeing an issue where the NN drops an HTTP connection in the middle of long fsck response, in particular when it's under other load (eg big checkpoints). It's really spurious and hard to reproduce, but we have some inkling that it's related to this issue. I've been bugging the jetty folks about timeline for a 6.1.27 release, but it may be a couple months off. I figured that 6.1.26".1" would be an interim solution for 205 and/or 206 until a 6.1.27 release is ready and can be QAed. Are you -1 or just not wild about it? FWIW the patch is not a custom change, bur rather just the NIO-related changes that will be integrated for 6.1.27.
          Hide
          Rob Weltman added a comment -

          Arun - we have customers asking for this. Is it OK to apply the NIO changes to get around the connection dropping until jetty 1.6.27 is out?

          Show
          Rob Weltman added a comment - Arun - we have customers asking for this. Is it OK to apply the NIO changes to get around the connection dropping until jetty 1.6.27 is out?
          Hide
          Rajiv Chittajallu added a comment -

          This is an long standing issue that is affecting production installation. Without significant operations monitoring, I wouldn't call 0.20.2xx a stable release for production.

          Show
          Rajiv Chittajallu added a comment - This is an long standing issue that is affecting production installation. Without significant operations monitoring, I wouldn't call 0.20.2xx a stable release for production.
          Hide
          Todd Lipcon added a comment -

          Shall I ping the dev list to see what the opinions are? I agree that (a) it's unfortunate to ship a patched Jetty, and (b) that the current fetch failure rate is unacceptable in 20x. Unfortunately to fix (b) we need to do (a)... as of yet, no word on Jetty 6.1.27 release timeline either. It seems to be low on their priority list (sort of like if someone asked us to release Hadoop 0.18.4 or something!)

          Show
          Todd Lipcon added a comment - Shall I ping the dev list to see what the opinions are? I agree that (a) it's unfortunate to ship a patched Jetty, and (b) that the current fetch failure rate is unacceptable in 20x. Unfortunately to fix (b) we need to do (a)... as of yet, no word on Jetty 6.1.27 release timeline either. It seems to be low on their priority list (sort of like if someone asked us to release Hadoop 0.18.4 or something!)
          Hide
          Todd Lipcon added a comment -

          Sent a note to mapreduce-dev@ to solicit more opinions on this issue. We have rolled out this patch to some customers and into QA for our next CDH release.

          Show
          Todd Lipcon added a comment - Sent a note to mapreduce-dev@ to solicit more opinions on this issue. We have rolled out this patch to some customers and into QA for our next CDH release.
          Hide
          Thomas Graves added a comment -

          I am definitely interested in a fix for this. Todd, any update on when the Jetty 6.1.27 release might take place?

          My personal opinion would be to use the custom patch build if 6.1.27 isn't going to be released for some time.

          Show
          Thomas Graves added a comment - I am definitely interested in a fix for this. Todd, any update on when the Jetty 6.1.27 release might take place? My personal opinion would be to use the custom patch build if 6.1.27 isn't going to be released for some time.
          Hide
          Matt Foley added a comment -

          Should we instead be looking at upgrading to Jetty 7.x? It doesn't look like the Jetty dev community is supporting 6.x in any significant way any more, see for example
          JETTY-1374 Hadoop's TaskTracker OOM after upgrade to jetty6

          Show
          Matt Foley added a comment - Should we instead be looking at upgrading to Jetty 7.x? It doesn't look like the Jetty dev community is supporting 6.x in any significant way any more, see for example JETTY-1374 Hadoop's TaskTracker OOM after upgrade to jetty6
          Hide
          Todd Lipcon added a comment -

          Upgrading to jetty 7 in the 20x series seems like a bit too much risk for a maintenance series. Doing it trunk is probably a good idea, though it will present compatibility issues with other projects like HBase that want to work against Hadoop 0.20 and Hadoop 0.23 both.

          I haven't heard any more from the Jetty folks about 6.1.27 - I'll try to ping Greg this afternoon.

          Show
          Todd Lipcon added a comment - Upgrading to jetty 7 in the 20x series seems like a bit too much risk for a maintenance series. Doing it trunk is probably a good idea, though it will present compatibility issues with other projects like HBase that want to work against Hadoop 0.20 and Hadoop 0.23 both. I haven't heard any more from the Jetty folks about 6.1.27 - I'll try to ping Greg this afternoon.
          Hide
          Matt Foley added a comment -

          I'm afraid I don't see a qualitative difference between the "risk" of transitioning from 6.1.14 -> 6.1.26 vs the risk of transitioning from 6.1.26 -> 7.x. In fact, if we transition to a version of 7.x that is currently viewed as being stable (what a concept!) then it might be less risky than transitioning to 6.1.27, which does not have any claim of being stable.

          Show
          Matt Foley added a comment - I'm afraid I don't see a qualitative difference between the "risk" of transitioning from 6.1.14 -> 6.1.26 vs the risk of transitioning from 6.1.26 -> 7.x. In fact, if we transition to a version of 7.x that is currently viewed as being stable (what a concept!) then it might be less risky than transitioning to 6.1.27, which does not have any claim of being stable.
          Hide
          Todd Lipcon added a comment -

          Jetty 7.x is API-incompatible as I understand it. So the effect would ripple down to all dependent projects that also use Jetty in any places. There are many of those (HBase is the first that comes to mind).

          Show
          Todd Lipcon added a comment - Jetty 7.x is API-incompatible as I understand it. So the effect would ripple down to all dependent projects that also use Jetty in any places. There are many of those (HBase is the first that comes to mind).
          Hide
          Evert Lammerts added a comment -

          So this is not fixed in 1.0.0 either, as far as I can see...

          Show
          Evert Lammerts added a comment - So this is not fixed in 1.0.0 either, as far as I can see...
          Hide
          Evert Lammerts added a comment -

          Is it possible that this causes SocketTimeOut exceptions in HDFS as well? I'm getting excluded datanodes when copying large single files ( > 1TB). As far as I can see it's not due to xceivers or any physical bound (RAM / core / IO loads are fine), and I don't see anything in the NN / DN logs. I've worked around it by increasing dfs.socket.timeout to 10 minutes, but the network patterns I see in Ganglia are worrying - every 10 to 30 minutes a complete drop of activity for some minutes. It might of course be a problem with our switches or our bonded interfaces as well...

          Show
          Evert Lammerts added a comment - Is it possible that this causes SocketTimeOut exceptions in HDFS as well? I'm getting excluded datanodes when copying large single files ( > 1TB). As far as I can see it's not due to xceivers or any physical bound (RAM / core / IO loads are fine), and I don't see anything in the NN / DN logs. I've worked around it by increasing dfs.socket.timeout to 10 minutes, but the network patterns I see in Ganglia are worrying - every 10 to 30 minutes a complete drop of activity for some minutes. It might of course be a problem with our switches or our bonded interfaces as well...
          Hide
          Kihwal Lee added a comment -

          Todd, any update on the use of jetty 6.1.27?
          It seems far less riskier than moving to jetty 7.

          We also have users complaining about this.
          The new health_check helps a bit but is not ideal.

          Show
          Kihwal Lee added a comment - Todd, any update on the use of jetty 6.1.27? It seems far less riskier than moving to jetty 7. We also have users complaining about this. The new health_check helps a bit but is not ideal.
          Hide
          Todd Lipcon added a comment -

          Hey Kihwal. 6.1.27 still hasn't been released. We've been shipping a patched version of 6.1.26 with some fixes provided by Greg Wilkins - the tag is here: https://github.com/toddlipcon/jetty-hadoop-fix/tree/6.1.26.cloudera.1

          The problems aren't 100% gone with this build but they seem to be improved – at least nothing's been escalated to me in a few months, so I'm assuming it's a good sign! The other patch we've recently added is MAPREDUCE-3184, which is similar to the health check script approach - it just suicides the TT if it detects the problem.

          Show
          Todd Lipcon added a comment - Hey Kihwal. 6.1.27 still hasn't been released. We've been shipping a patched version of 6.1.26 with some fixes provided by Greg Wilkins - the tag is here: https://github.com/toddlipcon/jetty-hadoop-fix/tree/6.1.26.cloudera.1 The problems aren't 100% gone with this build but they seem to be improved – at least nothing's been escalated to me in a few months, so I'm assuming it's a good sign! The other patch we've recently added is MAPREDUCE-3184 , which is similar to the health check script approach - it just suicides the TT if it detects the problem.
          Hide
          Kang Xiao added a comment -

          hi Todd, is jetty 6.1.27 released now? Or which version you are using at present? Whe downgrade to jetty 6.1.14 but it seems that it cause tasktracker memory problem. org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint use to much memory in tasktracker.

          Show
          Kang Xiao added a comment - hi Todd, is jetty 6.1.27 released now? Or which version you are using at present? Whe downgrade to jetty 6.1.14 but it seems that it cause tasktracker memory problem. org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint use to much memory in tasktracker.
          Hide
          Todd Lipcon added a comment -

          Still no 6.1.27. We've been shipping the version I linked to from github above: https://github.com/toddlipcon/jetty-hadoop-fix/tree/6.1.26.cloudera.1

          That, combined with MAPREDUCE-3184 has made the problem quite livable.

          We also found that the upgrade from 6.1.26 to the github branch improved performance noticeably for shuffle-intensive jobs.

          Show
          Todd Lipcon added a comment - Still no 6.1.27. We've been shipping the version I linked to from github above: https://github.com/toddlipcon/jetty-hadoop-fix/tree/6.1.26.cloudera.1 That, combined with MAPREDUCE-3184 has made the problem quite livable. We also found that the upgrade from 6.1.26 to the github branch improved performance noticeably for shuffle-intensive jobs.
          Hide
          Konstantin Boudnik added a comment -

          6.1.27 hasn't been released since the last comment on the issue back in
          June'12. So, I think it is time to move on to Jetty 7 or even 8 - as has been
          proposed elsewhere - along with upgrade to servlet 3.0 API.

          Show
          Konstantin Boudnik added a comment - 6.1.27 hasn't been released since the last comment on the issue back in June'12. So, I think it is time to move on to Jetty 7 or even 8 - as has been proposed elsewhere - along with upgrade to servlet 3.0 API.
          Hide
          Harsh J added a comment -

          This wouldn't specifically affect branch-2, since MR no longer relies on Jetty servlets for shuffling there (given the new Netty based, more performant Shuffle handler that replaced it a while ago). Please correct me if I'm wrong.

          That said, the Jetty fixes may still be required for the other parts that still use it for various other data transfer purposes, or for folks who need to use it via their own plugins.

          Show
          Harsh J added a comment - This wouldn't specifically affect branch-2, since MR no longer relies on Jetty servlets for shuffling there (given the new Netty based, more performant Shuffle handler that replaced it a while ago). Please correct me if I'm wrong. That said, the Jetty fixes may still be required for the other parts that still use it for various other data transfer purposes, or for folks who need to use it via their own plugins.
          Hide
          Fengdong Yu added a comment -

          why don't upgrade to the latest version of Jetty? it's incompatible?

          Show
          Fengdong Yu added a comment - why don't upgrade to the latest version of Jetty? it's incompatible?

            People

            • Assignee:
              Unassigned
              Reporter:
              Todd Lipcon
            • Votes:
              2 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

              • Created:
                Updated:

                Development