[BEAM-9239] Dependency conflict with Spark using aws io

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Affects Version/s: 2.17.0
• Fix Version/s: 2.29.0
• Component/s: io-java-aws, runner-spark
• Assignee: Unassigned
• Reporter: David McIntosh
• Created: 31/Jan/20 23:46
• Updated: 19/Mar/21 20:55
• Resolved: 19/Mar/21 20:54

Description

Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws io is also used:

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
	at org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
	at org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
	at com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
	at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
	at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
	at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
	at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
	at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
	at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
	at org.apache.beam.runners.core.construction.SerializablePipelineOptions.<init>(SerializablePipelineOptions.java:43)
	at org.apache.beam.runners.spark.translation.EvaluationContext.<init>(EvaluationContext.java:71)
	at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
	at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)

The cause seems to be that the Spark driver environment uses an older version of Jackson. I tried to update jackson on the Spark cluster but that led to several other errors.

The change that started causing this was:
https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116

After reverting that change I was able to successfully run my job.

Issue Links

• is superceded by: BEAM-11958 Don't use new Jackson APIs to avoid classpath issues when parsing AWS configuration