Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41281 Feature parity: SparkSession API in Spark Connect
  3. SPARK-42023

createDataFrame should corse types of string false to bool false

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Connect
    • None

    Description

      
      pyspark/sql/tests/test_types.py:249 (TypesParityTests.test_infer_schema_upcast_boolean_to_string)
      self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_upcast_boolean_to_string>
      
          def test_infer_schema_upcast_boolean_to_string(self):
      >       df = self.spark.createDataFrame([[True, 1], ["false", 1]], schema=["a", "b"])
      
      ../test_types.py:251: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      ../../connect/session.py:282: in createDataFrame
          _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
      pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
          ???
      pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
          ???
      pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
          ???
      pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
          ???
      pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
          ???
      pyarrow/array.pxi:320: in pyarrow.lib.array
          ???
      pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
          ???
      pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
          ???
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      >   ???
      E   pyarrow.lib.ArrowInvalid: Could not convert 'false' with type str: tried to convert to boolean
      
      pyarrow/error.pxi:100: ArrowInvalid
      
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gurwls223 Hyukjin Kwon
            gurwls223 Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment