Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-154

SimpleJsonConverter raises ClassCastException when long type used in schema

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      I trying to run the given example source/converter named SimpleJsonConverter to run with our own json/schema.

      Part of schema looks like this:

      {name: tag, type: [long, null]}

      And an example row from the source data:
      tag:

      {long:1}

      ,

      When i run the job it raises a cast exception, i figured out because gson treats each number as double precision (double); when the target object does not specify the expected type, and since that is a generic Map<String, Object> it becomes a Double. See SimpleJsonConverter::convertRecord.

      The converter should be generic, i.e. supports any schema given; i'm not sure how to tell gson to apply the right parsing based on the schema?

      Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.Long
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:79)
      at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
      at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
      at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
      at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)

      Github Url : https://github.com/linkedin/gobblin/issues/433
      Github Reporter : Paul424
      Github Created At : 2015-11-05T08:17:12Z
      Github Updated At : 2017-01-12T04:29:55Z

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            abti Abhishek Tiwari

            Dates

              Created:
              Updated:

              Slack

                Issue deployment