[HUDI-2525] Test prometheus metrics with hudi (both spark ds and deltastreamer)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Labels: sev:critical user-support-issues
• Assignee: Sagar Sumit
• Reporter: sivabalan narayanan
• Created: 04/Oct/21 15:32
• Updated: 14/Oct/21 11:53
• Resolved: 14/Oct/21 11:53

Description

Test prometheus metrics with hudi (both spark ds and deltastreamer)

 

exception w/ deltastreamer

Exception in thread "main" java.lang.NoSuchMethodError: 'void io.prometheus.client.dropwizard.DropwizardExports.<init>(org.apache.hudi.com.codahale.metrics.MetricRegistry)'	at org.apache.hudi.metrics.prometheus.PrometheusReporter.<init>(PrometheusReporter.java:49)	at org.apache.hudi.metrics.MetricsReporterFactory.createReporter(MetricsReporterFactory.java:75)	at org.apache.hudi.metrics.Metrics.<init>(Metrics.java:50)	at org.apache.hudi.metrics.Metrics.init(Metrics.java:96)	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics.<init>(HoodieDeltaStreamerMetrics.java:44)	at org.apache.hudi.utilities.deltastreamer.DeltaSync.<init>(DeltaSync.java:224)	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.<init>(HoodieDeltaStreamer.java:606)	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:143)	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:107)	at org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.<init>(HoodieDeltaStreamerWrapper.java:39)	at org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter.<init>(HoodieTestSuiteWriter.java:88)	at org.apache.hudi.integ.testsuite.dag.WriterContext.initContext(WriterContext.java:70)	at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.runTestSuite(HoodieTestSuiteJob.java:188)	at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:170)	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)	at java.base/java.lang.reflect.Method.invoke(Unknown Source)	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)21/10/01 17:06:01 INFO ShutdownHookManager: Shutdown hook called