The df.write.mode("overwrite").insertInto("table") has inconsistent behaviour between Scala and Python. In Python, insertInto ignores "mode" parameter and appends by default. Only when changing syntax to df.write.insertInto("table", overwrite=True) we get expected behaviour.
This is a native Spark syntax, expected to be the same between languages... Also, in other write methods, like saveAsTable or write.parquet "mode" seem to be respected.
Reproduce, Python, ignore "overwrite":
Reproduce, Scala, works as expected:
Tested on Spark 2.2.1 (EMR) and 2.4.0 (Databricks)