Description
The CLOB JDO type introduced with HIVE-12274 does not work correctly with PostgreSQL. The value is written out-of-band and the LOB handle is written,as an INT, into the table. SELECTs return the INT value, which should had been read via the lo_get PG built-in, and then cast into string.
Furthermore, the behavior is different between fields upgraded from earlier metastore versions (they retain their string storage) vs. values inserted after the upgrade (inserted as LOB roots).
Teh code in MetasoreDirectSql.getPartitionsFromPartitionIds/extractSqlClob expects the underlying JDO/Datanucleus to map the column to a Clob but that does not happen, the value is a Java String containing the int which is the LOB root saved by PG.
This manifests at runtime with errors like:
hive> select * from srcpart; Failed with exception java.io.IOException:java.lang.IllegalArgumentException: Error: type expected at the position 0 of '24030:24031' but '24030' is found.
the 24030:24031 should be 'string:string'.
repro:
CREATE TABLE srcpart (key STRING COMMENT 'default', value STRING COMMENT 'default') PARTITIONED BY (ds STRING, hr STRING) STORED AS TEXTFILE; LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt" OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="11"); select * from srcpart;
I did not see the issue being hit by non-partitioned/textfile tables, but that is just the luck of the path taken by the code. Inspection of my PG metastore shows all the CLOB fields suffering from this issue.
Attachments
Attachments
Issue Links
- is broken by
-
HIVE-12274 Increase width of columns used for general configuration in the metastore.
- Closed