Issue Details (XML | Word | Printable)

Key: HADOOP-234
Type: New Feature New Feature
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Owen O'Malley
Reporter: Sanjay Dahiya
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

Hadoop Pipes for writing map/reduce jobs in C++ and python

Created: 19/May/06 04:28 PM   Updated: 08/Jul/09 04:41 PM
Return to search
Component/s: None
Affects Version/s: None
Fix Version/s: 0.14.0

Time Tracking:
Not Specified

File Attachments:
  Size
PDF File Hadoop MaReduce Developer doc.pdf 2006-05-30 06:58 PM Sanjay Dahiya 112 kB
HTML File Licensed for inclusion in ASF works hadoop-pipes.html 2007-02-18 08:28 AM Owen O'Malley 11 kB
Text File Licensed for inclusion in ASF works pipes-2.patch 2007-05-16 12:21 AM Owen O'Malley 3.98 MB
Text File Licensed for inclusion in ASF works pipes.patch 2007-04-18 10:30 PM Owen O'Malley 3.98 MB
Issue Links:
Dependants
 

Resolution Date: 16/May/07 07:23 PM


 Description  « Hide
MapReduce C++ support

Requirements

1. Allow users to write Map, Reduce, RecordReader, and RecordWriter functions in C++, rest of the infrastructure already present in Java should be reused.
2. Avoid users having to write both Java and C++ for this to work.
3. Avoid users having to work with JNI methods directly by wrapping them in helper functions.
4. The interface should be SWIG'able.



 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Repository Revision Date User Message
ASF #538693 Wed May 16 19:23:48 UTC 2007 cutting HADOOP-234. Add pipes facility, which permits writing MapReduce programs in C++.
Files Changed
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/BinaryProtocol.java
ADD /lucene/hadoop/trunk/src/c++/utils/depcomp
ADD /lucene/hadoop/trunk/src/c++/pipes/api/hadoop
ADD /lucene/hadoop/trunk/src/c++/utils/m4
ADD /lucene/hadoop/trunk/src/c++/utils/ltmain.sh
ADD /lucene/hadoop/trunk/src/c++/utils/api/hadoop/StringUtils.hh
ADD /lucene/hadoop/trunk/src/c++/utils/aclocal.m4
ADD /lucene/hadoop/trunk/src/examples/pipes/configure
ADD /lucene/hadoop/trunk/src/examples/pipes/missing
ADD /lucene/hadoop/trunk/src/test/org/apache/hadoop/mapred/pipes
ADD /lucene/hadoop/trunk/src/examples/pipes/aclocal.m4
ADD /lucene/hadoop/trunk/src/examples/pipes/.autom4te.cfg
ADD /lucene/hadoop/trunk/src/c++/pipes/install-sh
ADD /lucene/hadoop/trunk/src/c++/utils/m4/hadoop_utils.m4
MODIFY /lucene/hadoop/trunk/bin/hadoop
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/Application.java
ADD /lucene/hadoop/trunk/src/c++/pipes/api/hadoop/Pipes.hh
ADD /lucene/hadoop/trunk/src/c++/pipes/configure
MODIFY /lucene/hadoop/trunk/CHANGES.txt
ADD /lucene/hadoop/trunk/src/c++/pipes/missing
ADD /lucene/hadoop/trunk/src/examples/pipes/impl/wordcount-part.cc
ADD /lucene/hadoop/trunk/src/c++/pipes/.autom4te.cfg
ADD /lucene/hadoop/trunk/src/c++/utils/impl/SerialUtils.cc
ADD /lucene/hadoop/trunk/src/examples/pipes/Makefile.in
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/UpwardProtocol.java
ADD /lucene/hadoop/trunk/src/c++/utils/config.sub
ADD /lucene/hadoop/trunk/src/examples/pipes
ADD /lucene/hadoop/trunk/src/examples/pipes/config.sub
ADD /lucene/hadoop/trunk/src/examples/pipes/Makefile.am
ADD /lucene/hadoop/trunk/src/c++/pipes/aclocal.m4
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/PipesReducer.java
ADD /lucene/hadoop/trunk/src/c++/utils/impl
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/Submitter.java
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/OutputHandler.java
ADD /lucene/hadoop/trunk/src/examples/pipes/impl
ADD /lucene/hadoop/trunk/src/examples/pipes/depcomp
ADD /lucene/hadoop/trunk/src/c++/utils/impl/StringUtils.cc
ADD /lucene/hadoop/trunk/src/c++/pipes/Makefile.in
ADD /lucene/hadoop/trunk/src/examples/pipes/conf
ADD /lucene/hadoop/trunk/src/examples/pipes/impl/wordcount-simple.cc
ADD /lucene/hadoop/trunk/src/c++/utils/impl/config.h.in
ADD /lucene/hadoop/trunk/src/examples/pipes/ltmain.sh
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/DownwardProtocol.java
ADD /lucene/hadoop/trunk/src/c++/utils/configure.ac
ADD /lucene/hadoop/trunk/src/examples/pipes/impl/config.h.in
ADD /lucene/hadoop/trunk/src/c++/pipes
ADD /lucene/hadoop/trunk/src/c++/utils/config.guess
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/package.html
ADD /lucene/hadoop/trunk/src/examples/pipes/configure.ac
ADD /lucene/hadoop/trunk/src/c++/pipes/Makefile.am
ADD /lucene/hadoop/trunk/src/examples/pipes/config.guess
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/PipesPartitioner.java
ADD /lucene/hadoop/trunk/src/c++/utils/configure
ADD /lucene/hadoop/trunk/src/c++/utils/missing
ADD /lucene/hadoop/trunk/src/c++/pipes/api/hadoop/TemplateFactory.hh
ADD /lucene/hadoop/trunk/src/examples/pipes/README.txt
ADD /lucene/hadoop/trunk/src/c++/utils/.autom4te.cfg
MODIFY /lucene/hadoop/trunk/build.xml
ADD /lucene/hadoop/trunk/src/c++/pipes/api
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/PipesMapRunner.java
ADD /lucene/hadoop/trunk/src/c++/pipes/impl/HadoopPipes.cc
ADD /lucene/hadoop/trunk/src/c++/utils/api/hadoop
ADD /lucene/hadoop/trunk/src/c++/pipes/depcomp
ADD /lucene/hadoop/trunk/src/c++/pipes/ltmain.sh
ADD /lucene/hadoop/trunk/src/c++/pipes/config.sub
ADD /lucene/hadoop/trunk/src/examples/pipes/conf/word.xml
ADD /lucene/hadoop/trunk/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java
ADD /lucene/hadoop/trunk/src/c++/utils/install-sh
ADD /lucene/hadoop/trunk/src/examples/pipes/install-sh
ADD /lucene/hadoop/trunk/src/examples/pipes/impl/wordcount-nopipe.cc
ADD /lucene/hadoop/trunk/src/c++/pipes/impl
ADD /lucene/hadoop/trunk/src/c++/utils/api/hadoop/SerialUtils.hh
ADD /lucene/hadoop/trunk/src/c++/pipes/compile
ADD /lucene/hadoop/trunk/src/c++/utils/Makefile.in
ADD /lucene/hadoop/trunk/src/test/org/apache/hadoop/mapred/pipes/WordCountInputFormat.java
ADD /lucene/hadoop/trunk/src/c++/pipes/impl/config.h.in
ADD /lucene/hadoop/trunk/src/c++/pipes/configure.ac
ADD /lucene/hadoop/trunk/src/examples/pipes/conf/word-part.xml
ADD /lucene/hadoop/trunk/src/c++/pipes/config.guess
ADD /lucene/hadoop/trunk/src/c++/utils
ADD /lucene/hadoop/trunk/src/c++/utils/Makefile.am
ADD /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes
ADD /lucene/hadoop/trunk/src/c++/utils/api

Repository Revision Date User Message
ASF #538704 Wed May 16 20:06:31 UTC 2007 cutting HADOOP-234. Make configure scripts executable.
Files Changed
MODIFY /lucene/hadoop/trunk/src/examples/pipes/configure
MODIFY /lucene/hadoop/trunk/src/c++/utils/configure
MODIFY /lucene/hadoop/trunk/src/c++/pipes/configure

Repository Revision Date User Message
ASF #538722 Wed May 16 20:37:44 UTC 2007 omalley HADOOP-234 Fix wording of the package.html to be in the current rather than
future tense.
Files Changed
MODIFY /lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/package.html