Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41279 Feature parity: DataFrame API in Spark Connect
  3. SPARK-41888

Support StreamingQueryListener for DataFrame.observe

    XMLWordPrintableJSON

Details

    Description

      **********************************************************************
      1334
      File "/__w/spark/spark/python/pyspark/sql/connect/dataframe.py", line 619, in pyspark.sql.connect.dataframe.DataFrame.observe
      1335
      Failed example:
      1336
          observation.get
      1337
      Exception raised:
      1338
          Traceback (most recent call last):
      1339
            File "/usr/lib/python3.9/doctest.py", line 1336, in __run
      1340
              exec(compile(example.source, filename, "single",
      1341
            File "<doctest pyspark.sql.connect.dataframe.DataFrame.observe[6]>", line 1, in <module>
      1342
              observation.get
      1343
            File "/__w/spark/spark/python/pyspark/sql/utils.py", line 378, in wrapped
      1344
              raise NotImplementedError()
      1345
          NotImplementedError
      1346
      **********************************************************************
      1347
      File "/__w/spark/spark/python/pyspark/sql/connect/dataframe.py", line 642, in pyspark.sql.connect.dataframe.DataFrame.observe
      1348
      Failed example:
      1349
          spark.streams.addListener(MyErrorListener())
      1350
      Exception raised:
      1351
          Traceback (most recent call last):
      1352
            File "/usr/lib/python3.9/doctest.py", line 1336, in __run
      1353
              exec(compile(example.source, filename, "single",
      1354
            File "<doctest pyspark.sql.connect.dataframe.DataFrame.observe[9]>", line 1, in <module>
      1355
              spark.streams.addListener(MyErrorListener())
      1356
          AttributeError: 'SparkSession' object has no attribute 'streams'
      1357
      **********************************************************************
      

      Attachments

        Issue Links

          Activity

            People

              beliefer Jiaan Geng
              beliefer Jiaan Geng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: