XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Connect
    • None

    Description

      df = self.spark.createDataFrame(
          [
              (
                  [1, 2, 3],
                  2,
                  2,
              ),
              (
                  [4, 5],
                  2,
                  2,
              ),
          ],
          ["x", "index", "len"],
      )
      
      expected = [Row(sliced=[2, 3]), Row(sliced=[5])]
      self.assertTrue(
          all(
              [
                  df.select(slice(df.x, 2, 2).alias("sliced")).collect() == expected,
                  df.select(slice(df.x, lit(2), lit(2)).alias("sliced")).collect() == expected,
                  df.select(slice("x", "index", "len").alias("sliced")).collect() == expected,
              ]
          )
      )
      
      self.assertEqual(
          df.select(slice(df.x, size(df.x) - 1, lit(1)).alias("sliced")).collect(),
          [Row(sliced=[2]), Row(sliced=[4])],
      )
      self.assertEqual(
          df.select(slice(df.x, lit(1), size(df.x) - 1).alias("sliced")).collect(),
          [Row(sliced=[1, 2]), Row(sliced=[4])],
      )
       Traceback (most recent call last):
        File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 596, in test_slice
          df.select(slice("x", "index", "len").alias("sliced")).collect() == expected,
        File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/utils.py", line 332, in wrapped
          return getattr(functions, f.__name__)(*args, **kwargs)
        File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 1525, in slice
          raise TypeError(f"start should be a Column or int, but got {type(start).__name__}")
      TypeError: start should be a Column or int, but got str

      Attachments

        Activity

          People

            gurwls223 Hyukjin Kwon
            techaddict Sandeep Singh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: