[CARBONDATA-3565] Binary to string issue when loading dataframe data in NewRddIterator

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.0
• Component/s: spark-integration
• Assignee: Unassigned
• Reporter: ChenKai
• Created: 02/Nov/19 16:00
• Updated: 08/Apr/20 07:14
• Resolved: 08/Apr/20 07:14

Description

• issue
  Spark DataFrame(SQL) load complex binary data to a hive table, the data will be broken when reading out. I see in RddIterator, the data will be converted to a string, and then be converted back.

• test case
  Binary data can be DataOutputStream#writeDouble and so on.

• discussion
  I think CarbonScalaUtil#getString operation can be removed now. I dig deep into the code in 2016, the code was used in kettle CsvInput (commit: 0018756d). But the code has been removed now, I think this converting operation is a little redundant. (UPDATE: The follow-up code GenericParser will use this string-convert logic, should consider here.)

Issue Links

• links to: GitHub Pull Request #3430