[SOLR-2096] DIH should be able read data directly from HDFS for indexing

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.4.1
• Fix Version/s: 4.9, 6.0
• Component/s: contrib - DataImportHandler
• Assignee: Unassigned
• Reporter: Amit Nithian
• Created: 31/Aug/10 06:53
• Updated: 28/Jun/17 22:59
• Resolved: 28/Jun/17 22:59

Description

DIH doesn't support reading from the hdfs:// protocol which makes it hard to index data generated by a M/R job. This tarball contains a subclass of the URLDataSource along with an HDFSReader that allows for this. The data is assumed to be in text format and able to be processed by the LineEntityProcessor.

Here is an example DIH-Config snippet:
<dataSource name="queryData" type="org.apache.solr.handler.dataimport.hdfs.HDFSDataSource"
baseUrl="hdfs://<YOURSERVER>:9000/" encoding="UTF-8"
connectionTimeout="5000" readTimeout="10000"/>
<document name="autoSuggester">
<entity name="jc" processor="LineEntityProcessor"
url="<YOUR FOLDER>/part*" dataSource="queryData">
<!-- Field mappings here if necessary -->
</entity>
</document>
</dataConfig>