Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2229

Sqoop2: Kite connector should handle doubles

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.99.6
    • Component/s: sqoop2-kite-connector
    • Labels:
      None

      Description

      Caused by: org.apache.avro.UnresolvedUnionException: Not in union ["double","null"]: 0.0
      	at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:561)
      	at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:144)
      	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
      	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
      	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:106)
      	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
      	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
      	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
      	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
      	... 12 more
      2015-03-18 13:53:13,158 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0018:Error occurs during loader run
      	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:261)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:724)
      Caused by: org.kitesdk.data.DatasetRecordException: Failed to append record
      	at org.kitesdk.data.spi.filesystem.AvroAppender.append(AvroAppender.java:74)
      	at org.kitesdk.data.spi.filesystem.FileSystemWriter.write(FileSystemWriter.java:158)
      	at org.apache.sqoop.connector.kite.KiteDatasetExecutor.writeRecord(KiteDatasetExecutor.java:85)
      	at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:72)
      	at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
      	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
      	... 6 more
      Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["double","null"]: 0.0
      	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263)
      	at org.kitesdk.data.spi.filesystem.AvroAppender.append(AvroAppender.java:72)
      	... 11 more
      Caused by: org.apache.avro.UnresolvedUnionException: Not in union ["double","null"]: 0.0
      	at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:561)
      	at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:144)
      	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
      	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
      	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:106)
      	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
      	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
      	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
      	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
      	... 12 more
      

      Right now Sqoop just makes every thing a double, when there are floats and doubles in Kite. Sqoop should choose the data type correctly based on the byte size in Floating Point.

        Attachments

        1. SQOOP-2229.001.patch
          5 kB
          Abraham Elmahrek
        2. SQOOP-2229.002.patch
          5 kB
          Abraham Elmahrek

          Issue Links

            Activity

              People

              • Assignee:
                abec Abraham Elmahrek
                Reporter:
                abec Abraham Elmahrek
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: