Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22369

PySpark: Document methods of spark.catalog interface

    Details

    • Type: Documentation
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      The following methods from the spark.catalog interface are not documented.

      $ pyspark
      >>> dir(spark.catalog)
      ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_jcatalog', '_jsparkSession', '_reset', '_sparkSession', 'cacheTable', 'clearCache', 'createExternalTable', 'createTable', 'currentDatabase', 'dropGlobalTempView', 'dropTempView', 'isCached', 'listColumns', 'listDatabases', 'listFunctions', 'listTables', 'recoverPartitions', 'refreshByPath', 'refreshTable', 'registerFunction', 'setCurrentDatabase', 'uncacheTable']
      

      As a user I would like to have these methods documented on
      http://spark.apache.org/docs/latest/api/python/pyspark.sql.html . Old methods of the SQLContext (e.g. pyspark.sql.SQLContext.cacheTable() vs. pyspark.sql.SparkSession.catalog.cacheTable() or pyspark.sql.HiveContext.refreshTable() vs. pyspark.sql.SparkSession.catalog.refreshTable() ) should point to the new method.

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              asmaier Andreas Maier
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: