[EAGLE-635] Refactor policy parser and validator for richer plan details and better performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v0.5.0
• Fix Version/s: v0.5.0
• Assignee: Hao Chen
• Reporter: Hao Chen
• Created: 18/Oct/16 18:06
• Updated: 11/Nov/16 06:09
• Resolved: 11/Nov/16 06:08

Description

Changes

• Refactor policy parser and validator for richer plan details and better performance
• Decouple PolicyExecutionPlan and PolicyValidation

API

• Parse API

POST /metadata/policies/parse 
Accept-Type: text

from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX#window.timeBatch(2 min) select cmd, user, count() as total_count group by cmd,user insert into HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX_OUT"

• Validation API

  POST /metadata/policies/validate 
  Accept-Type: application/json
  
  {
     "name": "hdfsPolicy",
     "description": "hdfsPolicy",
     "inputStreams": [
        "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
     ],
     "outputStreams": [
        "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX_OUT"
     ],
     "definition": {
        "type": "siddhi",
        "value": "from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX#window.timeBatch(2 min) select cmd, user, count() as total_count group by cmd,user insert into HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX_OUT "
     },
     "partitionSpec": [
        {
           "streamId": "hdfs_audit_log_enriched_stream",
           "type": "GROUPBY",
           "columns" : [
              "cmd"
           ]
        }
     ],
     "parallelismHint": 2
  }
  
  

Use Cases

• parse: could continuously call `parse` API aside during user keeps typing to verify the syntax and automatically generate input/output/partition , as it won't call back-end db, so will be very fast.
• validate: when user finishes defining policy, the api will validate the metadata end2end

Issue Links

• links to: GitHub Pull Request #529