pipes should be able to set user counters

• Key: HADOOP-1480
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Arun C Murthy
• Reporter: Owen O'Malley
• Affects Version/s: 0.14.0
• Fix Version/s: 0.19.0

Description

Pipes should expose the user defined counters.

Comments

[Arun C Murthy]

Here is a proposal after discussions with Owen:

Implement a Counter class to the C++ layer in Pipes:

Counter.cpp

class Counter {
  private:
    int id;
  public:
    void increment(long increment);
    static Counter getCounter(std::string group, std::string name);
}

The 'getCounter' contacts the Java Pipes' Mapper/Reducer and registers the given counter (group/name). The registration goes ahead and issues an 'id' which is stored inside the C++ Counter and is then used for incrementing the value of the counter.

Of course we'd need the getCounter/increment api in UpwardProtocol too...

UpwardProtocol.java

interface UpwardProtocol {

  // ...
  // ...
  // ...

  /**
   * Register a counter with the given group and name.
   * @param group counter group
   * @param name counter name
   * @return the id for the registered counter.
   * @throws IOException
   */
  int getCounter(String group, String name) throws IOException;
  
  /**
   * Increment the value of a registered counter.
   * @param id counter id of the registered counter
   * @param amount increment for the counter value
   * @throws IOException
   */
  void incrCounter(int id, long amount) throws IOException;

}

We'd need to add a Counters.Counter getCounter(String group, String name) api to Reporter to facilitate the above registration protocol:

Reporter.java

/**
   * Get the {@link Counter} of the given group with the given name.
   * 
   * @param group counter group
   * @param name counter name
   * @return the <code>Counter</code> of the given group/name.
   */
  public abstract Counter getCounter(String group, String name);

[Arun C Murthy]

Preliminary patch, mostly done - couple of minor loopholes. Testing...

[Arun C Murthy]

Updated, simpler patch. Needs more testing and maybe test-cases...

[Arun C Murthy]

Updated patch with test cases...

[Owen O'Malley]

I just committed this. Thanks, Arun!

[Hudson]

Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/)