[METRON-2284] Metron Profiler for Spark doesn't work as expected

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Affects Version/s: 0.7.1
• Fix Version/s: Next + 1
• Assignee: Nick Allen
• Reporter: Maxim Dashenko
• Created: 10/Oct/19 17:06
• Updated: 12/Nov/19 20:03
• Resolved: 12/Nov/19 20:03

Description

Used command:

/usr/hdp/current/spark2-client/bin/spark-submit --class org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file /usr/hcp/current/metron/config/batch-profiler.properties ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config /usr/hcp/current/metron/config/batch-profiler.properties --profiles ~/profiler.json

 cat /usr/hcp/current/metron/config/batch-profiler.properties

profiler.batch.input.path=/tmp/test_data.logs
profiler.batch.input.format=json
profiler.period.duration=15
profiler.period.duration.units=MINUTES

 
cat ~/profiler.json

{
 "profiles":[
   {
     "profile":"batchtest5",
     "onlyif":"source.type == 'testsource' and devicehostname == 'windows9.something.com'",
     "foreach":"devicehostname",
     "init":{
       "val":"SET_INIT()"
     },
     "update":{
       "val":"SET_ADD(val, IS_EMPTY(devicehostname))"
     },
    "result":{
       "profile":"val"
    }
   }
 ],
 "timestampField":"timestamp"
}

 cat test_data.logs

{"devicehostname": "windows9.something.com", "timestamp": 1567241981000, "source.type": "testsource"}

Stellar statement

PROFILE_GET('batchtest5', 'windows9.something.com', PROFILE_FIXED(100, 'DAYS'))

Returns:

[[true]]

Expected result:

[[false]]

Issue Links

• links to: GitHub Pull Request #1556