[SPARK-19796] taskScheduler fails serializing long statements received by thrift server

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Imran Rashid
• Reporter: Giambattista
• Created: 02/Mar/17 10:25
• Updated: 06/Mar/17 20:18
• Resolved: 06/Mar/17 20:06

Description

This problem was observed after the changes made for SPARK-17931.

In my use-case I'm sending very long insert statements to Spark thrift server and they are failing at TaskDescription.scala:89 because writeUTF fails if requested to write strings longer than 64Kb (see https://www.drillio.com/en/2009/java-encoded-string-too-long-64kb-limit/ for a description of the issue).

As suggested by Imran Rashid I tracked down the offending key: it is "spark.job.description" and it contains the complete SQL statement.

The problem can be reproduced by creating a table like:
create table test (a int) using parquet

and by sending an insert statement like:
scala> val r = 1 to 128000
scala> println("insert into table test values (" + r.mkString("),(") + ")")

Issue Links

• relates to: SPARK-17931 taskScheduler has some unneeded serialization
• links to: [Github] Pull Request #17140 (squito)