It appears that df.distinct.count can return incorrect values after
SPARK-23713. It's possible that other operations are affected as well; distinct just happens to be the one that we noticed. I believe that this issue was introduced by SPARK-23713 because I can't reproduce it until that commit, and I've been able to reproduce it after that commit as well as with tags/v2.4.0-rc1.
Below are example spark-shell sessions to illustrate the problem. Unfortunately the data used in these examples can't be uploaded to this Jira ticket. I'll try to create test data which also reproduces the issue, and will upload that if I'm able to do so.
Example from Spark 2.3.1, which behaves correctly:
Example from Spark 2.4.0-rc1, which returns different output: