The rule PullOutPythonUDFInJoinCondition was implemented in https://github.com/apache/spark/commit/2a8cbfddba2a59d144b32910c68c22d0199093fe
As far as I understand, this rule was intended to prevent the use of Python UDFs in join condition if they take arguments from both sides of the join, and this doesn't make sense in combination with the join type.
The rule PullOutPythonUDFInJoinCondition seems to make an assumption, that if a given UDF is only using columns from a single side of the join, it will be already pushed down under the join before this rule is executed.
However, this is not always the case. Here's a simple example that fails, even though it looks like it should run just fine (and it does in earlier versions of Spark):