[SOLR-10144] redesign block-join support

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: query parsers, update
• Labels: gsoc2017 java

Environment

Java. https://wiki.apache.org/solr/HowToContribute

• Assignee: Unassigned
• Reporter: Mikhail Khludnev
• Created: 16/Feb/17 04:48
• Updated: 14/Aug/18 13:38
• Resolved: 14/Aug/18 13:38

Description

This is a placeholder for new block join design. Let's comeup with the name first: whether it will be nested child docs or blocks?

Feature Delivery plan

Naming

Nested Documents

Scopes in schema.xml

1. fields can be grouped with <scope name="parent" default="true">;
   1. such fields' scoping is not mandatory, postponable, might cause rbdbms illusion
2. it can be any level deep, and fan out any subscopes (a scope name is necessary to distinguish between sons and daugthers subscopes);
3. I'm not sure whether name is uniq globally or names in traverse path is unique. I'd like the former;
4. btw, maybe type ;
5. default attribute is necessary to map existing blocks, which has only one nesting dimension: childrenDocuments;
6. fields beside of scopes (global) can appear on any scope. it should work with uniqueKey. What's semantic of uniqueKey across scopes One recently discussed case it searching for child scope documents, when children uniqKeys should not clash;
   1. root doc uniqueKey spans on whole block (it's necessary to be used as deleteTerm for block updates), but every doc is identified with own uniqueKey (otherwise it's not possible to find it with distributed search)
7. coming to roots how many root scopes we can have? I think we can have a few ones that introducing notion of document types.

Updates

1. All formats XML, JSON, JSONdoc, Javabin accept nesting in named scopes;
2. Current format (unnamed nesting) is supported by default marked scope;
3. field are accepted only if they are defined at the certain scope, beside of the global ones. (see consideration above)

Default experience

• 1. submitting q=*:*&fl=* responds root documents with all children from all subscopes nested in according to the scopes hierarchy. i.e. [child] is on by default
  2. fq= aims root docs

Query

1. let's follow Elastic JSON.facet idea, and piggyback on its' request parsing facilities;
2. this query should contains of nested query nodes, every node represent {!parser param=baram v='input'};
3. this syntax should have handy defaluts/shortcut, to search foo:bar in less than fifteen brackets,quotes and commas;
4. it should use existing QParsers including {!func} ;
5. searching scopes is supported by named scope QParser, handled in this syntax by regular way;
6. subscope query should be easily hooked in any occur (should, must, not ) ;
7. it should be available in a dedicated [transformer], and support the following scenarios: search parents by certain children, return them nested in response without repeating query, do the same but return all children of selected parents ;
8. naming standalone nodes and referencing them is cool.

Update/deletes/uniqueness/versioning

1. delete query/id hits root docs, and also nukes subordinate children;
2. updating existing root doc completely removes existing subdocs, and replace all of them with the new ones ;
3. field update fully rewrites whole doc hierarchy, and can aim any subdoc in hierarchy (this won't be easy, how to identify children?)
4. do we need to identify separate subdocs with id? probably yes for field updates
5. version should span whole block (If I've got the recent SOLR-10114 right)

Faceting

1. subject for support by Json.facet

Named scopes support in DIH

1. Old sport, you know.

Inception

comment and the next one

Issue Links

• is related to: SOLR-12638 Support atomic updates of nested/child documents for nested-enabled schema
• is related to: LUCENE-7674 java.lang.IllegalStateException: Child query must not match same docs with parent filter. Combine them as must clauses (+) to find a problem doc. docId=2147483647, class org.apache.lucene.search.ConjunctionScorer
• relates to: SOLR-5211 updating parent as childless makes old children orphans
• relates to: SOLR-5772 duplicate documents between solr "block join" documents and "normal" document
• relates to: SOLR-7606 ToParentBlockJoinQuery fails with AIOOBE under certain circumstances
• relates to: SOLR-6700 ChildDocTransformer doesn't return correct children after updating and optimising solr index
• relates to: SOLR-6596 Atomic update and adding child doc not working together
• relates to: SOLR-6096 Support Update and Delete on nested documents
• relates to: SOLR-11244 Query DSL for Solr