ARROW-8647, see comment at https://github.com/apache/arrow/pull/7536#issuecomment-653124260
When using dictionary type for the partition fields, this now creates partition expressions that also use a dictionary type. Which means that doing something like dataset.to_table(filter=ds.field("part") == "A") to filter on the partition field with a plain string expression doesn't work, limiting the usability of this option (and even with the new Python scalar stuff, it would not be easy to construct the correct expression):
It might be an option to keep the partition_expression use the dictionary value type instead of dictionary type? Or alternatively, as fsaintjacques proposed, ensure that any comparison involving the dict type should also work with the "effective" logical type (the value type of the dict).