Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36310

Fix hasnan() window function in IndexOpsMixin

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • PySpark
    • None

    Description

       

      File "/__w/spark/spark/python/pyspark/pandas/groupby.py", line 1497, in pyspark.pandas.groupby.GroupBy.rank
      Failed example:
          df.groupby("a").rank().sort_index()
      Exception raised:
      ...
      pyspark.sql.utils.AnalysisException: It is not allowed to use a window function inside an aggregate function. Please use the inner window function in a sub-query.
      

      As shown above, hasnans() used in "rank" causes "It is not allowed to use a window function inside an aggregate function" exception.

      We shall adjust that.

       

      Attachments

        Issue Links

          Activity

            People

              XinrongM Xinrong Meng
              XinrongM Xinrong Meng
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: