[S2GRAPH-240] Add management api for Experiment and Bucket.

• Type: Improvement
• Status: To Do
• Priority: Major
• Resolution: Unresolved
• Component/s: s2core
• Assignee: Unassigned
• Reporter: Do Yung Yoon
• Due: 21/Sep/18
• Created: 06/Sep/18 03:11
• Updated: 19/Sep/18 07:05

Description

Background

Experiment and Bucket exist to support A/B bucket test.

Here is how they can be used for bucket test.

1. Experiment: Experiment represents a single API endpoint for experimenting multiple buckets.

2. Create Bucket: Bucket contains single s2graph query template in JSON. it can be thought of as the stored procedure.

A single experiment can have one to many buckets.

Without specifying the bucket name, experiment follows traffic ratio to select which bucket to select the given user's POST request, which contains variables need to be interpolated into stored query template in the selected bucket.

Example.

Here is one example configuration for a single experiment.

Experiment 1:
	Bucket A(1~99): 99% of requests will be lead to this bucket.
	Bucket B(99~100): 1% of requests will be lead to this bucket.

Bucket A: following is example of s2graph query to recommend moivies for the user.

{
  "select": [
    "movie_id"
  ],
  "srcVertices": [
    {
      "serviceName": "s2graph",
      "columnName": "user_id",
      "id": "[[USER_ID]]"
    }
  ],
  "steps": [
  	{
  		"step": [
	      {
	        "label": "user_watched_movies",
	        "limit": 10
	      }
	    ]
  	}, 
    {
    	"step": [
    		{
	        "label": "similar_movie_model_x",
	        "limit": 10
	      }
    	]
    }
  ]
}

Note that to test two logic, similar_movie_model_x and similar_movie_model_y, we need to create two buckets, each per model.

Also, the changing part of this query is only `[[USER_ID]]`, so register template as a bucket and client send only changing variables.

curl -XPOST -H 'Content-Type:application/json' localhost:9000/graphs/experiment/access_token_for_this_experiment/experiment_name/id -d '
{
	"[[USER_ID]]": "steamshon"
}
'

Given user request, Experiment builds complete s2graph query mixing payload and registered template which yield "similar movies to user watched".

This experiment includes `S2-Impression-Id`(bucket name) which indicate the bucket name that built result. Once the user makes the conversion, the client send feedback edge as follow.

{
	"timestamp": "now",
	"from": "user_id", 
	"to": "S2-Impression-Id value",
	"label": "movie_recommend_feedback",
	"props": {
		"conversion_type": "imp/click/etc"
	}
}

Once this data is available, we can decide which bucket is best based on real user feedback.

Suggestion

Since it is a handy feature, there is no management(CRUD) API on Bucket and Experiment.

It would be great if we can polish this feature more.

1. support GraphQL in Bucket: would be better if we can use GraphQL query in Bucket.
2. add management API.