Details
Description
As of Python 3 _metaclass_ is no longer supported https://www.python.org/dev/peps/pep-3115/.
However, we have multiple classes which where never migrated to Python 3 compatible syntax:
- A number of ML Params} with _metaclass_ = ABCMeta
- Some of the SQL types with _metaclass_ = DataTypeSingleton
As a result some functionalities are broken in Python 3. For example
>>> from pyspark.sql.types import BooleanType >>> BooleanType() is BooleanType() False
or
>>> import inspect >>> from pyspark.ml import Estimator >>> inspect.isabstract(Estimator) False
where in both cases we expect to see True.
Attachments
Issue Links
- is related to
-
SPARK-32138 Drop Python 2, 3.4 and 3.5 in codes and documentation
-
- Resolved
-
- links to
