Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1986 Sqoop2: Schema matching improvements
  3. SQOOP-2008

Matching rules between from / to schema -revisted ( Use the avro model)

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Is it a norm to fill unmatched columns with nulls as we do ?

        protected void tryFillNullInArrayForUnexpectedColumn(Column column,
            Object[] array, int index) throws SqoopException {
          if (!column.isNullable()) {
            throw new SqoopException(SchemaError.SCHEMA_0004, "Target column " +
                column + " didn't match with any source column and cannot be null.");
          }
      
          LOG.warn("Column " + column +
              " has no matching source column. Will be ignored.");
          array[index] = null;
        }
      
      

      I looked at kites schema resolution is neat
      http://kitesdk.org/docs/0.17.1/Schema-Evolution.html
      similar concepts can be applied in sqoop and it relies on of how avro does reader/ writer schema resolution

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vybs Veena Basavaraj
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: