Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35805 API auditing in Pandas API on Spark
  3. SPARK-35808

Always enable the `pandas_metadata` in DataFrame.parquet

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • 3.2.0
    • None
    • PySpark
    • None

    Description

      We have a argument named `pandas_metadata` inĀ ps.read_parquet, but seems we can just always enable so that it always respect the pandas metadata when reading the Parquet file written by pandas.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            itholic Haejoon Lee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment