Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11812

pyspark reduceByKeyAndWindow does not handle unspecified invFunc (invFunc=None)

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.2
    • 1.3.2, 1.4.2, 1.5.3, 1.6.0
    • PySpark
    • None

    Description

      invFunc is optional and can be None. Instead of invFunc (the parameter) invReduceFunc (a local function) was checked for trueness (that is, not None, in this context). A local function is never None,
      thus the case of invFunc=none (a common one when inverse reduction is not defined) is treated incorrectly, resulting in loss of data.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            david.tolpin@gmail.com David Tolpin Assign to me
            david.tolpin@gmail.com David Tolpin
            dont_ping_this_account dont_ping_this_account
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment