Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39227

Reach parity with pandas boolean cast

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Pandas API on Spark, PySpark
    • None

    Description

      There are pandas APIs that need boolean casts: all, any.

      Currently, pandas-on-Spark has different behaviors on special inputs against these APIs, for example, empty string, list, etc, as mentioned https://github.com/apache/spark/pull/36547#issuecomment-1129228724 by zero323.

      We shall match pandas behavior on boolean cast.

      Meanwhile, Series/Frame that contains empty strings, lists should be considered as test input to increase test coverage.

      Attachments

        Activity

          People

            Unassigned Unassigned
            XinrongM Xinrong Meng
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: