[YARN-2879] Compatibility validation between YARN 2.2/2.4 and 2.6

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Assignee: Zhijie Shen
• Reporter: Zhijie Shen
• Created: 19/Nov/14 19:32
• Updated: 01/May/15 19:08
• Resolved: 01/May/15 19:08

Description

Recently, I did some simple backward compatibility experiments. Bascially, I've taken the following 2 steps:

1. Deploy the application (MR and DistributedShell) that is compiled against old YARN API (2.2/2.4) on new YARN cluster (2.6). The application is submitted via new Hadoop (2.6) client.

2. Deploy the application (MR and DistributedShell) that is compiled against old YARN API (2.2/2.4) on new YARN cluster (2.6). The application is submitted via old Hadoop (2.2/2.4) client that comes with the app.

I've tried these 2 steps on both insecure and secure cluster. Here's a short summary:

		DS 2.2	DS 2.4	MR 2.2 + Shuffle and RT 2.2	MR 2.2 + Shuffle and RT 2.6	MR 2.4 + Shuffle and RT 2.4	MR 2.4 + Shuffle and RT 2.6
Insecure	New Client	OK	OK	Client Incompatible	Client Incompatible	OK	OK
Insecure	Old Client	OK	OK	AM Incompatible	Client Incompatible	OK	OK
Secure	New Client	OK	OK	Client Incompatible	Client Incompatible	OK	OK
Secure	Old Client	OK	OK	AM Incompatible	Client Incompatible	OK	OK

Note that I've tried to run NM with both old and new version of shuffle handler plus the runtime libs.

In general, the compatibility looks good overall. There're a few issues that are related to MR, but they seem to be not the YARN issue. I'll post the individual problem in the follow-up comments.

Issue Links

• is related to: MAPREDUCE-6167 Prior 2.4 MR has compatibility issue because o.a.h.http.HttpConfig.setPolicy is removed
• is related to: MAPREDUCE-6168 Old MR client is still broken when receiving new counters from MR job