note that the opposite will probably apply: with the update, things written for Jackson 2.2.3 will inevitably break.

someone should do some full bigtop test runs with jackson bumped up just to see what happens

Steve, may be we should consider to mark this JIRA as incompatible?

oh yes

We need to go to a later version of jackson so S3A works on Java 8u60+; HADOOP-13050 ... because it needs an artifact com.fasterxml.jackson.dataformat:jackson-dataformat-cbor in sync with jackson2, and it's 2.3.3 or later only.

Attaching a patch to upgrade to the latest version (2.7.4). Let's see what happens.

This the HADOOP-13050 patch, which increments Jackson to 2.5.5; as jenkins doesn't actually test the AWS codebase, the aws JAR update doesn't get tested.

For the jackson update, we need to test across all modules

funny. race condition in coding. I went with approximately the same version that the aws JAR uses and its databinding JAR. I'll see what your one gets up to and remove the attachment I'd put in.

If we went with AWS lib 1.11.2, it'd be built against 2.6.6. I'd like them to at least be in sync

Agreed. Which AWS lib version do you want to choose, 1.10.77 or 1.11.2?

Anyway, I'm thinking we should upgrade Jackson to 2.5.5 or 2.6.6. I'll remove my patch.

-1 overall

This message was automatically generated.

javac warnings are related to deprecation in the s3a side of things; ignore here

Can we upgrade to jackson v2.7.6 or v2.8.0 - these versions coming soon, fix an XML Entity Expansion vulnerability?
Would it be possible to remove the dependency on jackson 1.9.13 too - this code base is no longer maintained and has the same XML Entity Expansion vulnerability?

Can we upgrade to jackson v2.7.6 or v2.8.0 - these versions coming soon, fix an XML Entity Expansion vulnerability?

Is there any detailed information that there is a vulnerability and it will be fixed in 2.7.6/2.8.0? I can only found CVE-2016-3720 and it is fixed in 2.7.4.

Would it be possible to remove the dependency on jackson 1.9.13 too

I'm thinking this can be done in a separate jira.

https://github.com/FasterXML/jackson-databind/issues/1279 covers the latest XEE bug

Thank you for the information! However, I'm thinking we need to upgrade jackson to 2.5.5/2.6.6 because aws-sdk 1.10/1.11 depends on the version. If we upgrade jackson to 2.7.6 or 2.8, we need to run aws-sdk with the version of jackson. It may work, but I'm not sure.

-1 overall

This message was automatically generated.

I'd like to help with this if I can. FWIW, I ran through an upgrade to 2.8.4 (latest). Required some very minor code changes (patch attached), but all the tests are looking good (including hadoop-aws) and manual testing didn't show any issues.

I do think it's time to upgrade ... but at the same time, if we push ahead of what everything else is using, there are going to be problems. Has anyone raised the upgrade with projects downstream (HBase etc?). What versions are they using?

This upgrade broke rumen? So it's probably going to break other code down below. Is there a version of Jackson which (a) fixes the XEE bug and (b) remains compatible at compile/run time with existing work. We know 2.6.6 does (b)

I think only 2.7.6 and 2.8.x have the XEE fix.

2.7.8 seems a happy balance. Very recent release, is apparently not vulnerable to that XEE bug, and changes to hadoop-rumen are not required to build or pass tests successfully.

sounds good; I'm wondering if we should raise it as a branch-2 change too, given the security implications, and now compatibility @ compile/link

thanks for doing this.

+1 for 2.7.8 in trunk and branch-2. Thanks PJ Fanning, Sean Mackrory, and Steve Loughran.

+1 for trunk, not rushing to put it in branch-2 yet because team HBase will be unhappy....we need to make sure that everyone is ready.

We do not declare jackson-databind as a dependency of hadoop-common; it is declared a dependency of hadoop-aws in the 2.7.x branch: (HADOOP-13692), and ends up in tools/lib in a hadoop distro. So we are explicitly and implicitly exporting it; not something we can ignore. We

Is there a standalone patch which increments jackson to 2.7.8?

Just attached one - didn't see any required changes other than the POM.

One lingering concern I have is that while I understand the need for more coordination in branch-2, the minor changes required if we did move to Jackson 2.8.x in Hadoop 3.0 seem smaller to me than the danger in being locked into Jackson 2.7.x for the life of Hadoop 3.x, where incompatible changes soon become even worse. Granted, there's also no guarantee that Jackson 2.9.x won't also contain incompatible changes, so I could just be fighting a losing battle, but wanted to raise the concern.

-1 overall

This message was automatically generated.

We're working on a shaded hadoop client at HADOOP-11804, after which classpath updates will have far more limited effect on clients.

that's great. I tihnk we are going to have to move up to Jackson 2.7+ in branch 2, with hadoop trunk -> 2.8+ and the PAI changes.

+1.

I've applied this to branch-2 and trunk. I don't want to go near 2.7 & 2.8 : maybe add a warning in the notes of the security risk & recommend an upgrade?

patch applied to 2.9+

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10868 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10868/)
HADOOP-12705 Upgrade Jackson 2.2.3 to 2.7.8 (stevel: rev 3eb7b686879d26fa2505b23e5e80b2f2a0ac436f)

• (edit) hadoop-project/pom.xml

Please remember to set the appropriate 3.x fix version when committing to trunk, thanks!