[KYLIN-1508] NPE error ERROR - Build Cube - Step 3 if LOOKUP table is Hive View - : java.lang.NullPointerException at org.apache.kylin.source.hive.HiveTable.getSignature

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: all, v1.2, v1.3.0, v1.5.0
• Component/s: Job Engine
• Environment: CentOS 6.7 , Hortonworks Hadoop 2.2.4.2 - sandbox
• Flags: Important
• Assignee: Dong Li
• Reporter: Richard Calaba
• Created: 18/Mar/16 17:15
• Updated: 05/Apr/16 08:00
• Resolved: 05/Apr/16 08:00

Description

Affected releases: The error occurs in 1.2 release (downloaded as binary release from Kylin Home page). The same error occurs also in latest snapshot 2.1 (compiled today 3/12/16 from Kylin Git using bracnh 2.x-staging). Tests were done on Hortonworks HDP 2.2.4 sandbox. Thus I assume that all current branches are affected.

Error Summary: Whereas it is possible to successfully build cube which FACT table referring Hive View – it seems not to be possible to build a cube which has LOOKUP table referring Hive View. It seems to be bug in metadata processing in Step 3 of the Build process. Build process fails with exception:

(release 1.2)
java.io.IOException: java.lang.NullPointerException
at org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:72)
at org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:62)
at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:85)
at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:205)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:60)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
at org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.kylin.common.util.HadoopUtil.fixWindowsPath(HadoopUtil.java:76)
at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:68)
at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:63)
at org.apache.kylin.dict.lookup.FileTable.getSizeAndLastModified(FileTable.java:85)
at org.apache.kylin.dict.lookup.HiveTable.getSignature(HiveTable.java:57)
... 16 more

Error in latest branch - (snapshoot-2.1) :

java.io.IOException: java.lang.NullPointerException
at org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71)
at org.apache.kylin.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:64)
at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89)
at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Importance of resolution of this bug: I believe that having the ability to use Hive Views is essential and should work for both FACT tables/views and LOOKUP/DIMENSION tables/views. 1st it helps to do more efficient OLAP staging by not forcing to materialize temporary views used just for OLAP modelling. 2ndly and most importantly having views with left outer and inner joins can further overcome the current Kylin limitation of not beeing to able to provide snowflake-like models.

Further info - reproduce scenario: I have created 2 Hive Tables (FACT and LOOKUP) using CREATE TABLE AS SELECT … statement in Hive. Then I have created 2 Hive Views using CREATE VIEW AS SELECT … supplying exactly same SQL statements (using left outer and inner joins). All 4 tables/views are correctly producing data in Hive. If I build simple Cube in Kylin using 2 tables – everything works fine. If I use Hive view for FACT table and regular Hive Table for LOOKUP – it also works. The error occurs if I try to use Hiev View as LOOKUP table and define dimension from it.

Just for completeness - below is full Hive QL for creating the table/views. It operates on tables which are created by internal tests while building Kylin 1.2. But I believe the exact view definition should not matter. I did couple of additional tests to create very simple views by doing only "SELECT * FROM one_table" and the error was still the same.

Hive Query Editor : Create Fact as View

CREATE VIEW view_fact_tab_enhanced
AS SELECT *
FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM
ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD

Hive Query Editor : Create Fact as Table

CREATE TABLE table_fact_tab_enhanced
AS SELECT *
FROM DEFAULT.TEST_KYLIN_FACT AS TEST_KYLIN_FACT LEFT OUTER JOIN EDW.TEST_SELLER_TYPE_DIM AS TEST_SELLER_TYPE_DIM
ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD

Hive Query Editor : Create LOOKUP as View

CREATE VIEW view_dim_tab_self_join AS
SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id
FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN DEFAULT.TEST_CATEGORY_GROUPINGS AS B
ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY A.leaf_categ_id, B.leaf_categ_name, A.site_id

Hive Query Editor : Create LOOKUP as Table

CREATE TABLE table_dim_tab_self_join AS
SELECT distinct A.leaf_categ_id, B.leaf_categ_name, A.site_id
FROM DEFAULT.TEST_CATEGORY_GROUPINGS AS A LEFT OUTER JOIN DEFAULT.TEST_CATEGORY_GROUPINGS AS B
ON A.leaf_categ_id = B.leaf_categ_id AND A.site_id = B.site_id ORDER BY A.leaf_categ_id, B.leaf_categ_name, A.site_id

Additional curiosity – seems release 1.2 is handling better the final size of the generated Cube – assuming I didn’t make any mistake in cube definitions and used same in both 1.2-release and 2.1-SNAPSHOT, I am getting smaller cube size in release 1.2 than in 2.1-SNAPSHOT codeline - in both cases the number of rows is exactly same.

Issue Links

• duplicates: KYLIN-1077 Support Hive View as Lookup Table
• relates to: KYLIN-1077 Support Hive View as Lookup Table