[SOLR-11299] Time partitioned collections (umbrella issue)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SolrCloud
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 30/Aug/17 18:32
• Updated: 08/Jun/19 15:20

Description

Solr ought to have the ability to manage large-scale time-series data (think logs or sensor data / IOT) itself without a lot of manual/external work. The most naive and painless approach today is to create a collection with a high numShards with hash routing but this isn't as good as partitioning the underlying indexes by time for these reasons:

• Easy to scale up/down horizontally as data/requirements change. (No need to over-provision, use shard splitting, or re-index with different config)
• Faster queries:
  • can search fewer shards, reducing overall load
  • realtime search is more tractable (since most shards are stable – good caches)
  • "recent" shards (that might be queried more) can be allocated to faster hardware
  • aged out data is simply removed, not marked as deleted. Deleted docs still have search overhead.
• Outages of a shard result in a degraded but sometimes a useful system nonetheless (compare to random subset missing)

Ideally you could set this up once and then simply work with a collection (potentially actually an alias) in a normal way (search or update), letting Solr handle the addition of new partitions, removing of old ones, and appropriate routing of requests depending on their nature.

This issue is an umbrella issue for the particular tasks that will make it all happen – either subtasks or issue linking.

Issue Links

• duplicates: SOLR-11536 Solr Index Partition by Timestamp(day)
• incorporates: SOLR-11444 Improve Aliases.java and comma delimited collection list handling
• is related to: SOLR-9690 Date/Time DocRouter
• is related to: SOLR-12295 Time Routed Aliases: Delete obsolete collections Async
• is related to: SOLR-9562 Minimize queried collections for time series alias
• relates to: SOLR-12308 LISTALIASES should return up to date response

Sub-Tasks

• Collection Alias metadata for time partitioned collections: Resolved
• Add URP to route time partitioned collections: Closed
• Expose Alias Properties CRUD in REST API: Closed
• create next time collection based on a fixed time gap: Closed
• TimePartitionedUpdateProcessor.lookupShardLeaderOfCollection should route to the ideal shard: Closed
• API to create a Time Routed Alias and first collection: Closed
• API command to delete oldest collections in a time routed alias: Open
• Auto delete oldest collections in a time routed alias: Closed
• Create Time Routed Alias stress-test: Open
• Exception Class to identify out of range docs vs other errors: Open
• add option for deleting an alias to delete collections first: Open
• Document Time Routed Aliases separate from API: Closed
• TRA: Pre-emptively create next collection: Closed
• TRA: evaluate autoDeleteAge independently of when collections are created: Open
• TRA: document re-dating (question, test, docs): Open
• Optimize Queries when sorting by router.field: Patch Available
• Optimize Queries when query filtering by TRA router.field: Patch Available