XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.5.0
    • Connect
    • None

    Description

      >>> import pandas as pd
      >>>
      >>> map_data = [{"a": 1}, {"b": 2, "c": 3}, {}, None, {"d": None}]
      >>> pdf = pd.DataFrame({"id": [0, 1, 2, 3, 4], "m": map_data})
      >>> schema = "id long, m map<string, long>"
      >>> spark.createDataFrame(pdf, schema=schema)
      Traceback (most recent call last):
      ...
      pyspark.errors.exceptions.connect.AnalysisException: [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `col_1` is of type "STRUCT<col_0: BIGINT, col_1: BIGINT, col_2: BIGINT, col_3: VOID>" while it's required to be "MAP<STRING, BIGINT>".
      

      Attachments

        Activity

          People

            ueshin Takuya Ueshin
            ueshin Takuya Ueshin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: