[SPARK-20156] Java String toLowerCase "Turkish locale bug" causes Spark problems

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Shell

Environment

Ubunutu 16.04
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)

• Assignee: Sean R. Owen
• Reporter: Serkan Taş
• Created: 30/Mar/17 11:30
• Updated: 08/Nov/18 13:04
• Resolved: 10/Apr/17 22:12

Description

If the regional setting of the operation system is Turkish, the famous java locale problem occurs (https://jira.atlassian.com/browse/CONF-5931 or https://issues.apache.org/jira/browse/AVRO-1493).
e.g :

"SERDEINFO" lowers to "serdeınfo"
"uniquetable" uppers to "UNİQUETABLE"

work around :
add -Duser.country=US -Duser.language=en to the end of the line
SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dscala.usejavacp=true"

in spark-shell.sh

Issue Links

• is duplicated by: SPARK-20361 JVM locale affects SQL type names
• links to: [Github] Pull Request #17527 (srowen)
• links to: [Github] Pull Request #17655 (gatorsmile)
• links to: [Github] Pull Request #22975 (zhengruifeng)