[SOLR-4799] DIH: join="zipper" aka merge join for nested EntityProcessors

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 5.0, 6.0
• Component/s: contrib - DataImportHandler
• Labels: DIH dataimportHandler dih
• Assignee: Noble Paul
• Reporter: Mikhail Khludnev
• Created: 08/May/13 01:04
• Updated: 09/May/16 18:51
• Resolved: 03/Dec/14 12:09

Description

DIH is mostly considered as a playground tool, and real usages end up with SolrJ. I want to contribute few improvements target DIH performance.

This one provides performant approach for joining SQL Entities with miserable memory at contrast to http://wiki.apache.org/solr/DataImportHandler#CachedSqlEntityProcessor

The idea is:

• parent table is explicitly ordered by it’s PK in SQL
• children table is explicitly ordered by parent_id FK in SQL
• children entity processor joins ordered resultsets by ‘zipper’ algorithm.

example usage for what's committed:

<dataConfig>
	<document>
		<entity name="parent" processor="SqlEntityProcessor" query="SELECT * FROM PARENT ORDER BY id">		
			<entity name="child_1" processor="SqlEntityProcessor"
				where="parent_id=parent.id" query="SELECT * FROM CHILD_1 ORDER BY parent_id" join="zipper" >
			</entity>			
		</entity>
	</document>
</dataConfig>

mind about:

1. ordering both sides
2. specifying join="zipper" at children entity
3. note that it works with any entity processors

Issue Links

• links to: blog describes overall context and ideas