Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1369

Avro export ignores --columns option

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.7
    • Component/s: None
    • Labels:
      None

      Description

      In JdbcExportJob AVRO_COLUMN_TYPES_MAP is being set with the full schema of the output table. This causes the AvroExportMapper to fail with unknown fields if --columns was used to restrict the columns to export (it then tries to set a value on the generated class which doesn't exist).

      There are multiple ways I can see to solve this.

      • Filter the columnTypes passed on to the Mapper in JdbcExportJob.configureInputFormat
      • Pass the --columns value on to the AvroExportMapper and let it ignore things that are not in there
      • Let AvroExportMapper not fail when it can't set a field.

      I might be able to provide a patch and I'd go with the simplest (the first one probably) if there are no objections.

        Attachments

        1. SQOOP-1369.patch
          3 kB
          Paul Mazak
        2. SQOOP-1369_5.patch
          15 kB
          Paul Mazak
        3. SQOOP-1369_4.patch
          12 kB
          Paul Mazak
        4. SQOOP-1369_3.patch
          12 kB
          Paul Mazak
        5. SQOOP-1369_2.patch
          7 kB
          Paul Mazak

          Activity

            People

            • Assignee:
              pmazak Paul Mazak
              Reporter:
              larsfrancke Lars Francke
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: