[SPARK-30509] Deprecation log warning is not printed in Avro schema inferring

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 14/Jan/20 10:15
• Updated: 14/Jan/20 19:48
• Resolved: 14/Jan/20 19:48

Description

The bug can be reproduced by the test:

  test("log a warning of ignoreExtension deprecation") {
    val logAppender = new LogAppender
    withTempPath { dir =>
      Seq(("a", 1, 2), ("b", 1, 2), ("c", 2, 1), ("d", 2, 1))
        .toDF("value", "p1", "p2")
        .repartition(2)
        .write
        .format("avro")
        .option("header", true)
        .save(dir.getCanonicalPath)
      withLogAppender(logAppender) {
        spark
          .read
          .format("avro")
          .option(AvroOptions.ignoreExtensionKey, false)
          .option("header", true)
          .load(dir.getCanonicalPath)
          .count()
      }
      val deprecatedEvents = logAppender.loggingEvents
        .filter(_.getRenderedMessage.contains(
          s"Option ${AvroOptions.ignoreExtensionKey} is deprecated"))
      assert(deprecatedEvents.size === 1)
    }
  }

Issue Links

• links to: GitHub Pull Request #27200