[FLINK-8186] AvroInputFormat regression: fails to deserialize GenericRecords on standalone cluster with hadoop27 compat

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Assignee: Aljoscha Krettek
• Reporter: Sebastian Klemke
• Created: 01/Dec/17 13:20
• Updated: 06/Dec/17 11:01
• Resolved: 05/Dec/17 15:53

Description

The following job runs fine on a Flink 1.3.2 cluster, but fails on a Flink 1.4.0 RC2 standalone cluster, "hadoop27" flavour:

public class GenericRecordCount {
    public static void main(String[] args) throws Exception {
        String input = ParameterTool.fromArgs(args).getRequired("input");

        ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

        long count = env.readFile(new AvroInputFormat<>(new Path(input), GenericRecord.class), input)
                .count();

        System.out.printf("Counted %d records\n", count);
    }
}

Runs fine in LocalExecutionEnvironment and also on no-hadoop flavour standalone cluster, though. Exception thrown in Flink 1.4.0 hadoop27:

12/01/2017 13:22:09     DataSource (at readFile(ExecutionEnvironment.java:514) (org.apache.flink.formats.avro.AvroInputFormat))(4/4) switched to FAILED
java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.avro.generic.GenericRecord.<init>()
        at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353)
        at org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369)
        at org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901)
        at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
        at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
        at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
        at org.apache.flink.formats.avro.AvroInputFormat.nextRecord(AvroInputFormat.java:165)
        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:167)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException: org.apache.avro.generic.GenericRecord.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
        ... 11 more

Issue Links

• links to: GitHub Pull Request #5120