[HUDI-376] AWS Glue dependency issue for EMR 5.28.0

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.5.1
• Component/s: Usability
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Xing Pan
• Created: 02/Dec/19 10:13
• Updated: 25/May/21 19:31
• Resolved: 08/Jan/20 00:07

Description

Hi hudi team, it's really encouraging that Hudi is finally officially supported application on AWS EMR. Great job!

I found a ClassNotFound exception when using:

/usr/lib/hudi/bin/run_sync_tool.sh

in emr master.

And I think is due to demand of aws glue data sdk dependency. (I used aws glue as hive meta data)

So I added a line to run_sync_tool.sh to get a quick fix for this:

HIVE_JARS=$HIVE_JARS:/usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar:/usr/share/aws/emr/emr-metrics-collector/lib/aws-java-sdk-glue-1.11.475.jar

not sure if any more jars needed, but these two jar fixed my problem.

 

I think it would be great if take glue in consideration for emr scripts.

Issue Links

• links to: GitHub Pull Request #1189