[KYLIN-5107] when excuting the cube job at "Convert Cuboid Data to HFile" step：Error running child

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: v3.1.2
• Component/s: Job Engine
• Environment: hadoop3.2.1 ; habse2.3.4
• Assignee: Unassigned
• Reporter: mzz
• Created: 21/Oct/21 05:28
• Updated: 26/Oct/21 01:11

Description

As title, when excuting the cube job at "Convert Cuboid Data to HFile" step,throw an exception:

2021-10-21 10:15:38,928 ERROR [main] org.apache.kylin.engine.mr.KylinReducer: 
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.util.FSUtils.setStoragePolicy(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;Ljava/lang/String;)V
	at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureStoragePolicy(HFileOutputFormat3.java:468)
	at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:287)
	at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:243)
	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:559)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
	at org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:141)
	at org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:135)
	at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
2021-10-21 10:15:38,929 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do cleanup, available memory: 2773m
2021-10-21 10:15:38,929 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Total rows: 1
2021-10-21 10:15:38,929 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.hadoop.hbase.util.FSUtils.setStoragePolicy(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;Ljava/lang/String;)V
	at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureStoragePolicy(HFileOutputFormat3.java:468)
	at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:287)
	at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:243)
	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:559)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
	at org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:141)
	at org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:135)
	at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

I try move all of hbase libs to

{KYLIN_HOME}

/lib ,but this problem not solved.