move NDFS and MapReduce to a separate project

• Key: NUTCH-193
• Type: Task
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Doug Cutting
• Reporter: Doug Cutting
• Component/s: ndfs
• Affects Version/s: 0.8
• Fix Version/s: 0.8

Description

The NDFS and MapReduce code should move from Nutch to a new Lucene sub-project named Hadoop.

My plan is to do this as follows:

1. Move all code in the following packages from Nutch to Hadoop:

org.apache.nutch.fs
org.apache.nutch.io
org.apache.nutch.ipc
org.apache.nutch.mapred
org.apache.nutch.ndfs

These packages will all be renamed to org.apache.hadoop, and Nutch code will be updated to reflect this.

2. Move selected classes from Nutch to Hadoop, as follows:

org.apache.nutch.util.NutchConf -> org.apache.hadoop.conf.Configuration
org.apache.nutch.util.NutchConfigurable -> org.apache.hadoop.Configurable
org.apache.nutch.util.NutchConfigured -> org.apache.hadoop.Configured

org.apache.nutch.util.Progress -> org.apache.hadoop.util.Progress
org.apache.nutch.util.LogFormatter-> org.apache.hadoop.util.LogFormatter
org.apache.nutch.util.Daemon -> org.apache.hadoop.util.Daemon

3. Add a jar containing all of the above the Nutch's lib directory.

Does this plan sound reasonable?

Comments

[Doug Cutting]

Link to the related Nutch issue.

[Doug Cutting]

NDFS, the Nutch Distributed Filesystem will be renamed HDFS, the Hadoop Distributed Filesystem. Its code will live in the package org.apache.nutch.dfs, and its fs implementation class will be named DistributedFileSystem.

[Andrzej Bialecki]

What timeframe did you have in mind? There are a few patches in the queue, which will be affected by this split.

Other than that - emphatic yes!

[Sami Siren]

+1

I quess the fuse-j - ndfs work from John/me could be part of hadoop /contrib after this change?

[Doug Cutting]

Andrzej: I'd like to do this soon, this week or next. No matter how long I wait, there will probably always be a few patches queued that will need to be updated. But hopefully we can avoid large patches like NUTCH-169. What other patches are you concerned about in particular?

Sami: yes, the fuse stuff would then make a great hadoop contrib package.

[Otis Gospodnetic]

I assume Doug meant org.apache.hadoop.dfs, not org.apache.nutch.dfs.

[Andrzej Bialecki]

Ok, the sooner the better from my POV. I didn;t have anything in mind that would be included in Hadoop, rather Nutch patches that I'm working on. Affected patches include some of the recent larger ones: the adaptive fetch schedule thing and crawl metadata. No big deal, but we need to know what to shoot for.

[Doug Cutting]

Otis: yes, thanks, I meant org.apache.hadoop.dfs.

Andrzej: I'm awaiting Mike's commit of NUTCH-183, which should happen today. I'll then try to make the split tomorrow.

[John Xing]

what's in the name hadoop? Because "had oops"?

[Doug Cutting]

The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid's term.

[Doug Cutting]

Okay, I've moved the code from Nutch to Hadoop. Now I need to repair Nutch so that it still works!

One remaining problem is the need to separate nutch config files from hadoop config files. There's now a hadoop-default.xml and hadoop-site.xml, which are separate from the similarly-named nutch files. For now, I'll fix this by adding the following methods to Hadoop's Configuration class:

void addDefaultResource(String name);
void addFinalResource(String name);

Then add a Nutch utility class like:

public class NutchConfiguration {
public static Configuration create() { Configuration conf = new Configuration(); addNutchResources(conf); }
public static Configuration addNutchResources(Configuration conf) { addDefaultResource("nutch-default.xml"); addFinalResource("nutch-site.xml"); }
}

Then all of the places which currently call 'new NutchConf()' can be replaced with 'NutchConfiguration().create()'.

Longer-term we might consider a more radical re-design of the configuration API. But first we need to get Hadoop and Nutch split.

[Doug Cutting]

I just committed this. Phew!

[Mike Cafarella]

It should be noted that the name "Nutch" also comes from one of Doug's children.
They seem to have a proud future in advertising and product naming.

[Sami Siren]

closing issues for released versions