Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32814

Metaclasses are broken for a few classes in Python 3

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0, 3.0.0, 3.1.0
    • 3.1.0
    • ML, PySpark, SQL
    • None

    Description

      As of Python 3 _metaclass_ is no longer supported https://www.python.org/dev/peps/pep-3115/.

      However, we have multiple classes which where never migrated to Python 3 compatible syntax:

      • A number of ML Params} with _metaclass_ = ABCMeta
      • Some of the SQL types with _metaclass_ = DataTypeSingleton

      As a result some functionalities are broken in Python 3. For example

      >>> from pyspark.sql.types import BooleanType                                                                                                                                                                      
      >>> BooleanType() is BooleanType()                                                                                                                                                                                 
      False
      

      or

      >>> import inspect                                                                                                                                                                                                 
      >>> from pyspark.ml import Estimator                                                                                                                                                                               
      >>> inspect.isabstract(Estimator)                                                                                                                                                                                  
      False
      

      where in both cases we expect to see True.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zero323 Maciej Szymkiewicz
            zero323 Maciej Szymkiewicz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment