[NUTCH-2696] Nutch SegmentReader does not dump non-ASCII characters with Hadoop 3.x

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.16
• Component/s: segment

Environment

Hadoop version : 3.0.0 (CDH 6.1)

Nutch : 1.15

Mode : distributed mode

• Assignee: Sebastian Nagel
• Reporter: Laurent Hervaud
• Created: 21/Feb/19 09:57
• Updated: 28/Jan/21 13:55
• Resolved: 01/Sep/19 15:49

Description

All Nutch tasks work properly with Hadoop 3.x. (except SegmentReader)
SegmentReader with -get option work fine.
SegmentReader with -dump option replace non-ascii character by ?

Exemple url : http://www.wikipedia.fr/index.php

 

command : ./runtime/deploy/bin/nutch readseg -dump /user/nutch/crawl1.15/segments/20190221093756 /tmp/dump1.15 -nocontent -nogenerate -noparse -noparsedata
ParseText::
 Wikipedia.fr - Portail de recherche sur les projets Wikim?dia
 Chercher sur Wikip?dia en fran?ais
 L?encyclop?die librement r?utilisable que chacun peut am?liorer.

 

 

command : ./runtime/deploy/bin/nutch readseg -get /user/nutch/crawl1.15/segments/20190221093756 http://www.wikipedia.fr/index.php -nocontent -nogenerate -noparse -noparsedata
ParseText::
 Wikipedia.fr - Portail de recherche sur les projets Wikimédia
 Chercher sur Wikipédia en français
 L’encyclopédie librement réutilisable que chacun peut améliorer.

 

I try to build with hadoop 3.0.0 dependencies in ivy.xml but i have the same result

It's work fine in local mode.

 

Issue Links

• links to: GitHub Pull Request #440