Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-3252

Wrong AVRO schema on import of DATE column with oraoop Connector

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.4.6
    • Fix Version/s: None
    • Component/s: codegen, connectors/oracle
    • Labels:
      None
    • Environment:

      Hortonworks HDP 2.6.0.1

      Description

      I have found a bug, when importing a table from an oracle database using the oracle connector for hadoop.

      The sqoop command looks like this:

      sqoop import -Dmapreduce.job.user.classpath.first=true -Doraoop.timestamp.string=true --connect jdbc:oracle:thin:@oracle_server --username USER --password xxx --as-avrodatafile --direct --table TEST_DATE --target-dir /user/admin/test_date
      

      The generated AVRO schema for the date field is [null, long]. This is why an error is thrown when sqoop tries to write the date string to that field. The workaround for this problem is to add a mapping for this column:

      --map-column-java DATE_COL=String
      

      But since I want to build a general solution adding a mapping for each DATE column is not an option.

      The type of the generated AVRO schema for DATE Columns should be [null, string] when using the Oracle connector for Hadoop and when oraoop.timestamp.string=true.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joha0123 Johannes Mayer
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified