[BAHIR-130] Support Cloudant Lite Plan

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Later
• Affects Version/s: Spark-2.0.0, Spark-2.0.1, Spark-2.0.2, Spark-2.1.0, Spark-2.1.1, Spark-2.2.0
• Fix Version/s: Spark-2.1.1, Spark-2.2.1
• Component/s: Spark SQL Data Sources
• Environment: ApacheSpark, any
• Assignee: Romeo Kienzer
• Reporter: Romeo Kienzer
• Created: 23/Aug/17 12:07
• Updated: 16/Oct/17 18:47
• Resolved: 16/Oct/17 18:47

Description

Cloudant has a plan called "Lite" supporting only five requests per second. So you end up with the following exception:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 0.0 failed 10 times, most recent failure: Lost task 4.9 in stage 0.0 (TID 42, yp-spark-dal09-env5-0040): java.lang.RuntimeException: Database harlemshake2 request error:

{"error":"too_many_requests","reason":"You've exceeded your current limit of 5 requests per second for query class. Please try later.","class":"query","rate":5}

at org.apache.bahir.cloudant.common.JsonStoreDataAccess.getQueryResult(JsonStoreDataAccess.scala:158)
at org.apache.bahir.cloudant.common.JsonStoreDataAccess.getIterator(JsonStoreDataAccess.scala:72)

Suggestion: Change JsonStoreDataAccess.scala in a way that when a 403 HTTP status code is returned the response is parsed in order to obtain the rate limit and then throttle the query down to that limit. In addition issue a WARNING in the log

Issue Links

• links to: GitHub Pull Request #49