Test files can be created as so:
Output with V1 Parquet reader:
With V1 Parquet reader, the exchange for lookup is reused (see last line).
Output with V2 Parquet reader (spark.sql.sources.useV1SourceList=""):
With the V2 Parquet reader, the exchange for lookup is not reused (see last 4 lines).
You can see the same issue with the Orc reader (and I assume any other datasource that extends Filescan).
The issue appears to be this check in FileScan#equals:
partitionFilters and dataFilters are not normalized, so their exprIds don't match. Thus FileScan objects don't match, even if they are the same.
As a side note, FileScan#equals has a dangling boolean expression:
The result of that expression is not actually used anywhere. We might want to include it in the final decision, even though that's not the issue here.