[CASSANDRA-19223] [Analytics] Column type mapping error for timestamp type during bulk writes - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Normal
Resolution: Fixed
Fix Version/s: NA
Component/s: Analytics Library
Labels:
None

Bug Category:
Correctness - Consistency
Severity:
Normal
Complexity:
Normal
Discovered By:
User Report
Platform:

All
Impacts:

None
Since Version:

NA
Source Control Link:

https://github.com/apache/cassandra-analytics/commit/e82fceaecfe5ea04ac3ddff92be5a6a41456333c
Test and Documentation Plan:

Hide

Added new integration test to validate the fix

Show
Added new integration test to validate the fix

Description

When doing bulk reads with the analytics library, a user can specify the last modified column as an option. Bulk reader will add a column with the last modified column to the data frame. If a user wants to use the bulk-read data frame to persist data, and using the WriterOptions.TIMESTAMP feature from the last modified column from the bulk-read data frame, the bulk write will fail with a data type mapping error.

Caused by: java.lang.RuntimeException: Unsupported conversion for LONG from java.sql.Timestamp
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$LongConverter.convertInternal(SqlToCqlTypeConverter.java:245)
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$LongConverter.convertInternal(SqlToCqlTypeConverter.java:231)
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$Converter.convert(SqlToCqlTypeConverter.java:203)
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$NullableConverter.convert(SqlToCqlTypeConverter.java:212)
	at org.apache.cassandra.spark.bulkwriter.TableSchema.normalize(TableSchema.java:91)
	at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)