[METRON-1120] Profile's 'groupBy' Expression Has No Reference to Time

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Affects Version/s: 0.4.0
• Fix Version/s: 0.4.1
• Assignee: Nick Allen
• Reporter: Nick Allen
• Created: 18/Aug/17 13:16
• Updated: 25/Aug/17 15:09
• Resolved: 25/Aug/17 15:09

Description

It is often the case that patterns and behaviors will differ based on calendar effects like day of week. For example, activity on a weekday can be very different from a weekend. The Profiler's "Group By" functionality is one way to account for calendar effects.

This profile definition operates over any incoming telemetry that has an `ip_src_addr` and a `timestamp` field. It produces a profile that segments the data by day of week. It does by using a 'groupBy' expression to extract the day of week from the telemetry's `timestamp` field.

{
  "profiles": [
    {
      "profile": "calender-effects",
      "onlyif":  "exists(ip_src_addr) and exists(timestamp)",
      "foreach": "ip_src_addr",
      "init":    { "count": 0 },
      "update":  { "count": "count + 1" },
      "result":  "count",
      "groupBy": ["DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT'))"]
    }
  ]
}

When retrieving profile data using the Profiler Client API, I only want to retrieve data from the same day of week to account for any calendar effects. The following example retrieves profile data only for Thursdays over the past 60 days.

>>> thursday := 5
>>> PROFILE_GET("calendar-effects", "10.0.0.1", PROFILE_FIXED(60, "DAYS"), [thursday])

The Problem

The `groupBy` expression only has access to the Profile's `result` value. It does not have any way to reference the current tick time in the Profiler. Here is an example showing the problem.

Define the profile and a message.

[Stellar]>>> conf
{
  "profiles": [
    {
      "profile": "calender-effects",
      "onlyif":  "exists(ip_src_addr) and exists(timestamp)",
      "foreach": "ip_src_addr",
      "init":    { "count": "0" },
      "update":  { "count": "count + 1" },
      "result":  "count",
      "groupBy": ["DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT'))"]
    }
  ]
}
[Stellar]>>> msg
{
     "ip_src_addr": "10.0.0.1",
     "protocol": "HTTPS",
     "length": "10",
     "bytes_in": 234,
     "timestamp": "2017-08-17 09:00:00"
}

Initialize the Profiler and apply the message a few times.

[Stellar]>>> p := PROFILER_INIT(conf)
[Stellar]>>> PROFILER_APPLY(msg, p)
org.apache.metron.profiler.StandAloneProfiler@9472c85
[Stellar]>>> PROFILER_APPLY(msg, p)
org.apache.metron.profiler.StandAloneProfiler@9472c85
[Stellar]>>> PROFILER_APPLY(msg, p)
org.apache.metron.profiler.StandAloneProfiler@9472c85

Flush the profile, which will trigger execution of the `groupBy` expression.

[Stellar]>>> PROFILER_FLUSH(p)
[!] Bad 'groupBy' expression: Unexpected type: expected=Object, actual=null, expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT')), profile=calender-effects, entity=10.0.0.1
org.apache.metron.stellar.dsl.ParseException: Bad 'groupBy' expression: Unexpected type: expected=Object, actual=null, expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT')), profile=calender-effects, entity=10.0.0.1
	at org.apache.metron.profiler.DefaultProfileBuilder.execute(DefaultProfileBuilder.java:257)
	at org.apache.metron.profiler.DefaultProfileBuilder.flush(DefaultProfileBuilder.java:159)
	at org.apache.metron.profiler.DefaultMessageDistributor.lambda$flush$0(DefaultMessageDistributor.java:101)
	at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
	at org.apache.metron.profiler.DefaultMessageDistributor.flush(DefaultMessageDistributor.java:99)
	at org.apache.metron.profiler.StandAloneProfiler.flush(StandAloneProfiler.java:82)
	at org.apache.metron.profiler.client.stellar.ProfilerFunctions$ProfilerFlush.apply(ProfilerFunctions.java:191)
	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:556)
	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160)
	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
	at org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:287)
	at org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:270)
	at org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:409)
	at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Unexpected type: expected=Object, actual=null, expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'GMT'))
	at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:128)
	at org.apache.metron.profiler.DefaultProfileBuilder.lambda$execute$3(DefaultProfileBuilder.java:253)
	at java.util.ArrayList.forEach(ArrayList.java:1249)
	at org.apache.metron.profiler.DefaultProfileBuilder.execute(DefaultProfileBuilder.java:253)
	... 16 more

Issue Links

• links to: GitHub Pull Request #708