move NDFS and MapReduce to a separate project

• Key: NUTCH-193
• Type: Task
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Doug Cutting
• Reporter: Doug Cutting
• Component/s: ndfs
• Affects Version/s: 0.8
• Fix Version/s: 0.8

Description

The NDFS and MapReduce code should move from Nutch to a new Lucene sub-project named Hadoop.

My plan is to do this as follows:

1. Move all code in the following packages from Nutch to Hadoop:

org.apache.nutch.fs
org.apache.nutch.io
org.apache.nutch.ipc
org.apache.nutch.mapred
org.apache.nutch.ndfs

These packages will all be renamed to org.apache.hadoop, and Nutch code will be updated to reflect this.

2. Move selected classes from Nutch to Hadoop, as follows:

org.apache.nutch.util.NutchConf -> org.apache.hadoop.conf.Configuration
org.apache.nutch.util.NutchConfigurable -> org.apache.hadoop.Configurable
org.apache.nutch.util.NutchConfigured -> org.apache.hadoop.Configured

org.apache.nutch.util.Progress -> org.apache.hadoop.util.Progress
org.apache.nutch.util.LogFormatter-> org.apache.hadoop.util.LogFormatter
org.apache.nutch.util.Daemon -> org.apache.hadoop.util.Daemon

3. Add a jar containing all of the above the Nutch's lib directory.

Does this plan sound reasonable?

Comments

[Doug Cutting]

Link to the related Nutch issue.

NDFS, the Nutch Distributed Filesystem will be renamed HDFS, the Hadoop Distributed Filesystem. Its code will live in the package org.apache.nutch.dfs, and its fs implementation class will be named DistributedFileSystem.

[Andrzej Bialecki]

What timeframe did you have in mind? There are a few patches in the queue, which will be affected by this split.

Other than that - emphatic yes!

[Sami Siren]

+1

I quess the fuse-j - ndfs work from John/me could be part of hadoop /contrib after this change?

[Doug Cutting]

Andrzej: I'd like to do this soon, this week or next. No matter how long I wait, there will probably always be a few patches queued that will need to be updated. But hopefully we can avoid large patches like NUTCH-169. What other patches are you concerned about in particular?

Sami: yes, the fuse stuff would then make a great hadoop contrib package.

[Otis Gospodnetic]

I assume Doug meant org.apache.hadoop.dfs, not org.apache.nutch.dfs.

[Andrzej Bialecki]

Ok, the sooner the better from my POV. I didn;t have anything in mind that would be included in Hadoop, rather Nutch patches that I'm working on. Affected patches include some of the recent larger ones: the adaptive fetch schedule thing and crawl metadata. No big deal, but we need to know what to shoot for.

[Doug Cutting]

Otis: yes, thanks, I meant org.apache.hadoop.dfs.

Andrzej: I'm awaiting Mike's commit of NUTCH-183, which should happen today. I'll then try to make the split tomorrow.

[John Xing]

what's in the name hadoop? Because "had oops"?

[Doug Cutting]

The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid's term.

[Doug Cutting]

Okay, I've moved the code from Nutch to Hadoop. Now I need to repair Nutch so that it still works!

One remaining problem is the need to separate nutch config files from hadoop config files. There's now a hadoop-default.xml and hadoop-site.xml, which are separate from the similarly-named nutch files. For now, I'll fix this by adding the following methods to Hadoop's Configuration class:

void addDefaultResource(String name);
void addFinalResource(String name);

Then add a Nutch utility class like:

public class NutchConfiguration {
public static Configuration create() { Configuration conf = new Configuration(); addNutchResources(conf); }
public static Configuration addNutchResources(Configuration conf) { addDefaultResource("nutch-default.xml"); addFinalResource("nutch-site.xml"); }
}

Then all of the places which currently call 'new NutchConf()' can be replaced with 'NutchConfiguration().create()'.

Longer-term we might consider a more radical re-design of the configuration API. But first we need to get Hadoop and Nutch split.

[Doug Cutting]

I just committed this. Phew!

[Mike Cafarella]

It should be noted that the name "Nutch" also comes from one of Doug's children.
They seem to have a proud future in advertising and product naming.

[Sami Siren]

closing issues for released versions