[LIVY-754] precision and scale are not encoded in decimal type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.8.0
• Component/s: Thriftserver
• Assignee: Wing Yew Poon
• Reporter: Wing Yew Poon
• Created: 02/Apr/20 01:27
• Updated: 20/May/20 17:39
• Resolved: 20/May/20 17:39

Description

The Livy Thrift server support for decimal type in 0.7 is inadequate.
Before LIVY-699, decimal is mapped to the catch-all string type. With LIVY-699, decimal is mapped to a decimal type that is inadequate in that it does not encode the precision and scale. The type in Livy is represented by a BasicDataType case class which contains a String field, name; and a DataType (an enum) field, dataType. In the case of decimal, the dataType is DataType.DECIMAL. The precision and scale of the decimal is not encoded.
When the DataType is converted to a TTypeDesc for sending a Thrift response to a client request for result set metadata, the TTypeDesc contains a TPrimitiveTypeEntry(TTypeId.DECIMAL_TYPE) without TTypeQualifiers (which are needed to capture the precision and scale). This results in problems for clients. E.g., if we connect to the Thrift server in beeline and do a select from a table with column of decimal type, we get

java.lang.NullPointerException
	at org.apache.hive.jdbc.JdbcColumn.columnPrecision(JdbcColumn.java:310)
	at org.apache.hive.jdbc.JdbcColumn.columnDisplaySize(JdbcColumn.java:262)
	at org.apache.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(HiveResultSetMetaData.java:63)
	at org.apache.hive.beeline.IncrementalRows.<init>(IncrementalRows.java:57)
	at org.apache.hive.beeline.IncrementalRowsWithNormalization.<init>(IncrementalRowsWithNormalization.java:47)
	at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2322)
	at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1026)
	at org.apache.hive.beeline.Commands.execute(Commands.java:1215)
	at org.apache.hive.beeline.Commands.sql(Commands.java:1144)
	at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
	at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
	at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1134)
	at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
	at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
	at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)

Note: You have to use "--verbose" with beeline to see the stack trace for the NPE.

Issue Links

• links to: GitHub Pull Request #288