Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29875

Avoid to use deprecated pyarrow.open_stream API in Spark 2.4.x

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.4
    • Fix Version/s: 2.4.5
    • Component/s: PySpark, SQL
    • Labels:
      None

      Description

      In Spark 2.4.x, if we use PyArrow higher then 0.12.0, it shows a bunch of warnings as below:

      from pyspark.sql.functions import pandas_udf, PandasUDFType
      
      @pandas_udf("integer", PandasUDFType.SCALAR)  # doctest: +SKIP
      def add_one(x):
          return x + 1
      
      spark.range(100).select(add_one("id")).collect()
      
      UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning: pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
        warnings.warn("pyarrow.open_stream is deprecated, please use "
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              hyukjin.kwon Hyukjin Kwon

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment