[CALCITE-2616] Can't create Unicode literal by RelBuilder

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.17.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Anton Haidai
• Created: 09/Oct/18 15:35
• Updated: 29/Apr/23 01:06

Description

Test in RelBuilderTest to reproduce the issue:

@Test public void testScanWithFilterByUnicodeValue() {
  final RelBuilder builder = RelBuilder.create(config().build());
  RelNode root =
      builder.scan("EMP")
          .filter(
              builder.call(SqlStdOperatorTable.EQUALS,
                  builder.field("ENAME"),
                  builder.literal("Петро ピーター")
              )
          )
          .build();
}

 Result:

org.apache.calcite.runtime.CalciteException: Failed to encode 'Петро ピーター' in character set 'ISO-8859-1'

Possible workaround: create saffron.properties with the following property

saffron.default.charset=UTF-16LE

But UTF-8 will not work as a value of this property, see SqlUtil.translateCharacterSetName

Related code:

• SqlUtil.translateCharacterSetName(charsetName)
• RelDataTypeFactoryImpl.getDefaultCharset()
• SaffronProperties

 Could it be considered to switch defaults from "ISO-8859-1" to "UTF-8" in SaffronProperties?

 

Issue Links

• is related to: CALCITE-3308 RelBuilder.literal with a string argument should create a character literal with appropriate character set
• relates to: CALCITE-2619 Reduce string literal creation cost by removing unnecessary charset check