Details
-
Bug
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
2.1.0
-
None
-
None
Description
Scenario-1: Create external table pointing to external storage handler hbase/kafka
In this case there is no entity created for the external hbase path, though there is atlas entity for hbase table.Also no lineage present on atlas for this table to external path.
Query for Hive Table with external HBase storage handler:
create table default.tbl29_hbs(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( "hbase.columns.mapping" = ":key,cf:name", "hbase.table.name" = "default:hbs_tbl29" );
Below are the Alter table scenarios which are not propagated to Atlas
Scenario-2: SET SERDE
CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int, d int, e int, f int) ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe' Assert Error: alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest) Time elapsed: 61 sec <<< FAILURE! java.lang.AssertionError: The following asserts failed: serdeInfo serializationLib mismatch expected [org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found [org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe] at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38) at org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
Scenario-3: SET FILEFORMAT TEXTFILE
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e int, f int) ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile ALTER TABLE db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE Assert Error: alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest) Time elapsed: 60.788 sec <<< FAILURE! java.lang.AssertionError: The following asserts failed: outputFormat mismatch expected [org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found [org.apache.hadoop.mapred.SequenceFileInputFormat] at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38) at org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
Scenario-4: SET FILEFORMAT SEQUENCEFILE
CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d int, e int, f int) ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile ALTER TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE Assert Error: alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest) Time elapsed: 60.796 sec <<< FAILURE! java.lang.AssertionError: The following asserts failed: outputFormat mismatch expected [org.apache.hadoop.mapred.SequenceFileOutputFormat] but found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat] at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38) at org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
Scenario-5: SET FILEFORMAT PARQUET
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d int, e int, f int) ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile ALTER TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET Assert Error: alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest) Time elapsed: 60.803 sec <<< FAILURE! java.lang.AssertionError: The following asserts failed: serdeInfo serializationLib mismatch expected [org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found [org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat mismatch expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found [org.apache.hadoop.mapred.SequenceFileInputFormat] at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38) at org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
Scenario-6: ALTER TABLE using CLUSTERED BY
create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID Int,FirstName String,Designation String, Salary Int,Department String) clustered by (department) into 3 buckets stored as orc TBLPROPERTIES ('transactional'='true') ALTER TABLE db2k9eooxraj5.cluster_by_k9eooxraj5 CLUSTERED BY (viewTime, userid) SORTED BY (page_url) INTO 48 BUCKETS Assert Error: alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest) Time elapsed: 61.25 sec <<< FAILURE! java.lang.AssertionError: The following asserts failed: bucketCols mismatch: lists don't have the same size expected [2] but found [0], numBuckets mismatch expected [48] but found [-1] at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38) at org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)