Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-1986 Sqoop2: Schema matching improvements
  3. SQOOP-2008

Matching rules between from / to schema -revisted ( Use the avro model)

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • None
    • None

    Description

      Is it a norm to fill unmatched columns with nulls as we do ?

        protected void tryFillNullInArrayForUnexpectedColumn(Column column,
            Object[] array, int index) throws SqoopException {
          if (!column.isNullable()) {
            throw new SqoopException(SchemaError.SCHEMA_0004, "Target column " +
                column + " didn't match with any source column and cannot be null.");
          }
      
          LOG.warn("Column " + column +
              " has no matching source column. Will be ignored.");
          array[index] = null;
        }
      
      

      I looked at kites schema resolution is neat
      http://kitesdk.org/docs/0.17.1/Schema-Evolution.html
      similar concepts can be applied in sqoop and it relies on of how avro does reader/ writer schema resolution

      Attachments

        Activity

          People

            Unassigned Unassigned
            vybs Veena Basavaraj
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment