Details
-
Bug
-
Status: Triage Needed
-
P2
-
Resolution: Fixed
-
2.18.0
Description
From Aaron's comment in https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476 .
My org must use Avro 1.9.x (due to some Avro schema resolution issues resolved in 1.9.x) so downgrading Avro is not possible for us.
Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are not available in 1.9.x.
The Java class is org.apache.avro.data.TimeConversions.TimestampConversion in Avro 1.8.
It's renamed to org.apache.avro.data.JodaTimeConversions in Avro 1.9.
Beam Java SDK cannot upgrade Avro to 1.9
Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
Illustration of the dependency
Short-term Solution
As illustrated above, as long as Beam Java SDK uses only the intersection of Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide flexibility in runtime Avro versions (as it did until Beam 2.16).
Difference of the TimeConversion Classes
Avro 1.9's TimestampConversion overrides getRecommendedSchema method. Details below:
Avro 1.8's TimeConversions.TimestampConversion:
public static class TimestampConversion extends Conversion<DateTime> { @Override public Class<DateTime> getConvertedType() { return DateTime.class; } @Override public String getLogicalTypeName() { return "timestamp-millis"; } @Override public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType type) { return new DateTime(millisFromEpoch, DateTimeZone.UTC); } @Override public Long toLong(DateTime timestamp, Schema schema, LogicalType type) { return timestamp.getMillis(); } }
Avro 1.9's JodaTimeConversions.TimestampConversion:
public static class TimestampConversion extends Conversion<DateTime> { @Override public Class<DateTime> getConvertedType() { return DateTime.class; } @Override public String getLogicalTypeName() { return "timestamp-millis"; } @Override public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType type) { return new DateTime(millisFromEpoch, DateTimeZone.UTC); } @Override public Long toLong(DateTime timestamp, Schema schema, LogicalType type) { return timestamp.getMillis(); } @Override public Schema getRecommendedSchema() { return LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG)); } }
Attachments
Attachments
Issue Links
- is related to
-
BEAM-8388 Update Avro to 1.9.1 from 1.8.2
-
- In Progress
-
- links to
