Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18038

Move output partitioning definition from UnaryNodeExec to its children

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.0.1
    • 2.1.0
    • SQL
    • None

    Description

      This was a suggestion by Reynold Xin over one of the dev list discussion : http://apache-spark-developers-list.1001551.n3.nabble.com/Project-not-preserving-child-partitioning-td19417.html

      I think this is very risky because preserving output partitioning should not be a property of UnaryNodeExec (e.g. exchange).
      It would be better (safer) to move the output partitioning definition into each of the operator and remove it from UnaryExecNode.
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tejasp Tejas Patil Assign to me
            tejasp Tejas Patil
            Votes:
            0 Vote for this issue
            Watchers:
            3 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment