[SPARK-38266] UnresolvedException: Invalid call to dataType on unresolved object caused by GetDateFieldOperations

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1, 3.3.0
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 21/Feb/22 02:39
• Updated: 22/Feb/22 07:53
• Resolved: 21/Feb/22 02:41

Description

test("GetDateFieldOperations should skip unresolved nodes") {
  withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
    val df = Seq("1644821603").map(i => (i.toInt, i)).toDF("tsInt", "tsStr")
    val df1 = df.select(df("tsStr").cast("timestamp")).as("df1")
    val df2 = df.select(df("tsStr").cast("timestamp")).as("df2")
    df1.join(df2, $"df1.tsStr" === $"df2.tsStr", "left_outer")
    val df3 = df1.join(df2, $"df1.tsStr" === $"df2.tsStr", "left_outer")
      .select($"df1.tsStr".as("timeStr")).as("df3")
    // This throws "UnresolvedException: Invalid call to
    // dataType on unresolved object" instead of "AnalysisException: Column 'df1.timeStr' does not exist."
    df3.join(df1, year($"df1.timeStr") === year($"df3.tsStr"))
  }
} 

Issue Links

• is caused by: SPARK-35937 Extracting date field from timestamp should work in ANSI mode
• links to: [Github] Pull Request #35568 (Ngone51)
• links to: [Github] Pull Request #35601 (Ngone51)
• links to: [Github] Pull Request #35601 (Ngone51)