Avro
  1. Avro
  2. AVRO-1452

Problem when using AvroMultipleOutputs with multiple schemas

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.6
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Environment:

      Any Platform

    • Tags:
      AvroMultipleOutputs

      Description

      When using multiple named outputs with different Key/Value Schemas, the last provided schema overrides any previous schema definitions after first write attempt. This happens due to issue with the following code in AvroMultipleOutputs.java:509
      /begin/
      Job job = new Job(context.getConfiguration());
      ...
      setSchema(job, keySchema, valSchema);
      taskContext = createTaskAttemptContext(
      job.getConfiguration(), context.getTaskAttemptID());
      /end/
      Every time this code runs, actual configuration instance passed to createTaskAttemptContext remains the same, because Job constructor creates new configuration copy only if it is not instanceof JobConf. This way we have properties "avro.schema.output.XXX" overwrote each time new TaskAttemptContext is initialised and also mistakenly shared Configuration instance for all TaskAttemptContextes

      Proposed fix:
      a) use "Job getInstance(Configuration conf)" or
      b) call "new Job(new Configuration(context.getConfiguration))"

        Activity

        Hide
        Vladislav Spivak added a comment -

        Will try to arrange some standalone sample ASAP

        Show
        Vladislav Spivak added a comment - Will try to arrange some standalone sample ASAP
        Hide
        Doug Cutting added a comment -

        Can you please provide a complete test that demonstrates this problem? Thanks!

        Show
        Doug Cutting added a comment - Can you please provide a complete test that demonstrates this problem? Thanks!

          People

          • Assignee:
            Unassigned
            Reporter:
            Vladislav Spivak
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development