Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8186

AvroInputFormat regression: fails to deserialize GenericRecords on standalone cluster with hadoop27 compat

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      The following job runs fine on a Flink 1.3.2 cluster, but fails on a Flink 1.4.0 RC2 standalone cluster, "hadoop27" flavour:

      public class GenericRecordCount {
          public static void main(String[] args) throws Exception {
              String input = ParameterTool.fromArgs(args).getRequired("input");
      
              ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
      
              long count = env.readFile(new AvroInputFormat<>(new Path(input), GenericRecord.class), input)
                      .count();
      
              System.out.printf("Counted %d records\n", count);
          }
      }
      

      Runs fine in LocalExecutionEnvironment and also on no-hadoop flavour standalone cluster, though. Exception thrown in Flink 1.4.0 hadoop27:

      12/01/2017 13:22:09     DataSource (at readFile(ExecutionEnvironment.java:514) (org.apache.flink.formats.avro.AvroInputFormat))(4/4) switched to FAILED
      java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.avro.generic.GenericRecord.<init>()
              at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353)
              at org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369)
              at org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901)
              at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
              at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
              at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
              at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
              at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
              at org.apache.flink.formats.avro.AvroInputFormat.nextRecord(AvroInputFormat.java:165)
              at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:167)
              at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.NoSuchMethodException: org.apache.avro.generic.GenericRecord.<init>()
              at java.lang.Class.getConstructor0(Class.java:3082)
              at java.lang.Class.getDeclaredConstructor(Class.java:2178)
              at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
              ... 11 more
      

        Attachments

        1. pom.xml
          19 kB
          Sebastian Klemke
        2. GenericRecordCount.java
          0.7 kB
          Sebastian Klemke

          Issue Links

            Activity

              People

              • Assignee:
                aljoscha Aljoscha Krettek
                Reporter:
                packet Sebastian Klemke
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: