[CASSANDRA-15990] Running CQL command with non-ASCII values raises UnicodeDecodeError

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta2, 4.0
• Component/s: Tool/cqlsh
• Bug Category: Degradation
• Severity: Normal
• Complexity: Low Hanging Fruit
• Discovered By: User Report
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/8abbfc1fc5244e5686de0d04a1f71a004204030f

Test and Documentation Plan

Trunk patch , new basic dtests added

• Assignee: Ekaterina Dimitrova
• Reporter: Joseph Chu

Authors

Ekaterina Dimitrova

Reviewers

Brandon Williams, David Capwell

• Created: 28/Jul/20 05:03
• Updated: 03/Jan/21 17:02
• Resolved: 19/Aug/20 15:36

Description

There are INSERT statements that contains non-ASCII values that have run fine in Cassandra 3.11, but now raises a UnicodeDecodeError when I try executing them in 4.0-alpha4 and 4.0-beta1. 

Example input and output:

echo $LANG
en_US.UTF-8
$ cqlsh --debug
Using CQL driver: <module 'cassandra' from '/usr/share/cassandra/bin/../lib/cassandra-driver-internal-only-3.23.0.post0-1a184b99.zip/cassandra-driver-3.23.0.post0-1a184b99/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using ssl: False
Connected to Cassandra Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 4.0-beta1 | CQL spec 3.4.5 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE KEYSPACE killr_video WITH replication = {'class': 'NetworkTopologyStrategy', 'DC-Houston': 1};
cqlsh> USE killr_video;
cqlsh:killr_video> CREATE TABLE movies_by_genre ( genre TEXT, title TEXT, year INT, duration INT, avg_rating FLOAT, country TEXT, PRIMARY KEY ((genre), title, year));
cqlsh:killr_video> INSERT INTO movies_by_genre (genre, title, year, duration, avg_rating, country)
 ... VALUES ('Action', 'The Extraordinary Adventures of Adèle Blanc-Sec', 2010, 107, 6.30, 'France');
Traceback (most recent call last):
 File "/usr/share/cassandra/bin/cqlsh.py", line 937, in onecmd
 self.handle_statement(st, statementtext)
 File "/usr/share/cassandra/bin/cqlsh.py", line 962, in handle_statement
 readline.add_history(new_hist)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 134: ordinal not in range(128)