Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-2110

option of TempCsv should be removed since the default delimiter may conflicts with field value

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.3.0
    • data-load
    • None

    Description

      Currently in carbondata, an option named ‘tempCSV’ is available during loading dataframe.

       

      After enabling this option, Carbondata will write the dataframe to a standard csv file at first and then load the data files.

       

      The delimiters of the standard csv file, such as field delimiter / escape char/ quote char/ multi-line/ line separator and so on may conflict with the actual field value. For example, if a field contains ',', then it will cause problem in further data loading if we save the tempCSV using ',' as field separator.

       

      Since we are not sure about the content of dataframe, I think it's better to deprecate this option. To make forward compatible, user can still use this option but will get warning about it.

      Attachments

        Activity

          People

            xuchuanyin Chuanyin Xu
            xuchuanyin Chuanyin Xu
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 7h
                7h