[PARQUET-379] PrimitiveType.union erases original type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.6.0, 1.7.0, 1.8.0
• Fix Version/s: 1.9.0
• Component/s: parquet-mr
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 23/Sep/15 20:44
• Updated: 23/Jun/24 03:27
• Resolved: 06/Feb/16 19:58

Description

The following ScalaTest test case

  test("merge primitive types") {
    val expected =
      Types.buildMessage()
        .addField(
          Types
            .required(INT32)
            .as(DECIMAL)
            .precision(7)
            .scale(2)
            .named("f"))
        .named("root")

    assert(expected.union(expected) === expected)
  }

produces the following assertion error

message root {
  required int32 f;
}
 did not equal message root {
  required int32 f (DECIMAL(9,0));
}

This is because PrimitiveType.union doesn't handle original type properly. An open question is that, can two primitive types with the same primitive type name but different original types be unioned?

Issue Links

• blocks: PARQUET-392 Release Parquet-mr 1.9.0
• is related to: PARQUET-385 PrimitiveType.union accepts fixed_len_byte_array fields with different lengths when strict mode is on
• is related to: PARQUET-753 GroupType.union() doesn't merge the original type
• links to: https://github.com/apache/parquet-mr/pull/315