Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36161

dropDuplicates does not type check argument

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.2
    • 3.2.0
    • PySpark
    • None

    Description

      When giving a single string to dropDuplicates method in pyspark a cryptic error is returned. Rather than returning a cryptic error, handle gracefully or return an exception. 

       

      Proposal: Model after dropna behavior. If single string then use it to create a list, otherwise if argument is not list/tuple return an exception

      Attachments

        Activity

          People

            sammyjmoseley Samuel Moseley
            sammyjmoseley Samuel Moseley
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: