[BEAM-4186] Need to be able to set QuerySplitter in DatastoreIO.v1()

• Type: New Feature
• Status: Resolved
• Priority: P3
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: io-java-gcp

Docs Text

I would like to use Dataflow to be able to split queries of the form

SELECT * FROM ledger  WHERE type = 'purchase'
     AND eventTime >= DATETIME("2017-01-01T00:00:00Z")
     AND eventTime < DATETIME("2018-01-01T00:00:00Z")

the default query splitter (QuerySplitterImpl) refuses to split queries of this sort.  However it is obvious that this this query could be split by eventTime.  400 shards would break this up into pieces slightly shorter than a day.  50,000 shards would give you chunks of 11 minutes.

I have already written such a query splitter.  However I need to be able to tell DatastoreV1 to replace the default query splitter with mine in order to get this to work.

• Assignee: Frank Yellin
• Reporter: Frank Yellin
• Created: 29/Apr/18 23:25
• Updated: 24/Jul/20 20:17
• Resolved: 05/May/18 17:24

Description

I want to add a method

      withQuerySplitter(QuerySplitter querySplitter)

to DatastoreV1.Reader.  The implementation is fairly straightforward, except for enforcing the requirement that the query splitter must be Serializable for this to work.

 

 

Issue Links

• links to: GitHub Pull Request #5246