[METRON-1986] Batch Profiler Fails to Resolve Stats Stellar Functions

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.7.1
• Assignee: Nick Allen
• Reporter: Nick Allen
• Created: 31/Jan/19 20:44
• Updated: 23/Apr/19 19:51
• Resolved: 19/Feb/19 21:15

Description

The Batch Profiler does not seem to see the STATS_* functions. When I attempt to run a profile using any of those functions, it fails. It is able to 'see' a limited subset of functions, but not all those that I would expect.

[metron@nat-r7-dwys-metron-10 metron]$ cat config/zookeeper/batch-profiler.json
{
  "profiles": [
    {
      "profile": "profile-using-stats",
      "foreach": "ip_src_addr",
      "update": { "s": "STATS_ADD(s, length)" },
      "result": "STATS_SUM(s)"
    }
  ],
  "timestampField":"timestamp"
}

The error indicating that the `STATS_ADD` function cannot be resolved.

19/01/28 16:17:38 ERROR DefaultProfileBuilder: Unable to apply message to profile: Bad 'update' expression: error='Unable to parse: STATS_ADD(s, length) due to: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER with relevant variables s=missing,length=missing', expr='STATS_ADD(s, length)', profile='get_mean_length', entity='192.168.66.1', variables-available='[java.lang.Object@19fdd93a, java.lang.Object@2b9545d2, java.lang.Object@6eb0a63a, java.lang.Object@62b7a53a, java.lang.Object@5c5c9b0d, java.lang.Object@6aada2b8, java.lang.Object@8decf48, java.lang.Object@d88dc3, java.lang.Object@66b5c5ae, java.lang.Object@5a8bc995, java.lang.Object@3db163af, java.lang.Object@27909f0f, java.lang.Object@5e1b4144, java.lang.Object@74478bd8, java.lang.Object@6881fbe0, java.lang.Object@126bb4ce, java.lang.Object@5053a836, java.lang.Object@512dfe04, java.lang.Object@51cb7ee2, java.lang.Object@5a3af410, java.lang.Object@15546cbe, java.lang.Object@5a58ec73, java.lang.Object@25d929f5, java.lang.Object@7b93967f, java.lang.Object@7eba9414, java.lang.Object@5191c77, java.lang.Object@6220b9cf, java.lang.Object@135e806c, java.lang.Object@7cf1a78, java.lang.Object@36a4d7de, java.lang.Object@d20a5dd, java.lang.Object@664a5335]' org.apache.metron.stellar.dsl.ParseException: Bad 'update' expression: error='Unable to parse: STATS_ADD(s, length) due to: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER with relevant variables s=missing,length=missing', expr='STATS_ADD(s, length)', profile='get_mean_length', entity='192.168.66.1', variables-available='[java.lang.Object@19fdd93a, java.lang.Object@2b9545d2, java.lang.Object@6eb0a63a, java.lang.Object@62b7a53a, java.lang.Object@5c5c9b0d, java.lang.Object@6aada2b8, java.lang.Object@8decf48, java.lang.Object@d88dc3, java.lang.Object@66b5c5ae, java.lang.Object@5a8bc995, java.lang.Object@3db163af, java.lang.Object@27909f0f, java.lang.Object@5e1b4144, java.lang.Object@74478bd8, java.lang.Object@6881fbe0, java.lang.Object@126bb4ce, java.lang.Object@5053a836, java.lang.Object@512dfe04, java.lang.Object@51cb7ee2, java.lang.Object@5a3af410, java.lang.Object@15546cbe, java.lang.Object@5a58ec73, java.lang.Object@25d929f5, java.lang.Object@7b93967f, java.lang.Object@7eba9414, java.lang.Object@5191c77, java.lang.Object@6220b9cf, java.lang.Object@135e806c, java.lang.Object@7cf1a78, java.lang.Object@36a4d7de, java.lang.Object@d20a5dd, java.lang.Object@664a5335]' at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:303) at org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:141) at org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:168) at org.apache.metron.profiler.spark.function.ProfileBuilderFunction.call(ProfileBuilderFunction.java:88) at org.apache.metron.profiler.spark.function.ProfileBuilderFunction.call(ProfileBuilderFunction.java:49) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$mapGroups$1.apply(KeyValueGroupedDataset.scala:219) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$mapGroups$1.apply(KeyValueGroupedDataset.scala:219) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$1.apply(KeyValueGroupedDataset.scala:196) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$1.apply(KeyValueGroupedDataset.scala:196) at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$10$$anonfun$apply$4.apply(objects.scala:337) at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$10$$anonfun$apply$4.apply(objects.scala:336) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$2$$anon$1.hasNext(InMemoryRelation.scala:149) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1092) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335) at org.apache.spark.rdd.RDD.iterator(RDD.scala:286) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to parse: STATS_ADD(s, length) due to: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER with relevant variables s=missing,length=missing at org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:173) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154) at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160) at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95) at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291) ... 43 more Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:696) at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:659) at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151) ... 46 more Caused by: java.lang.IllegalStateException: Unknown function: `STATS_ADD` at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:150) at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:48) at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:691) ... 49 more 

This is the case even though the Profiler jar contains the STATS functions like `STATS_ADD`.

[metron@nat-r7-dwys-metron-10 metron]$ jar -tvf $PROFILER_JAR | grep "StellarStatisticsFunctions\$Add" 2492 Wed Jan 09 23:13:10 UTC 2019 org/apache/metron/statistics/StellarStatisticsFunctions$Add.class }}

 

Issue Links

• links to: GitHub Pull Request #1328