Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1328

Hadoop Streaming needs to provide a way for the stream plugin to update global counters

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.18.0
    • None
    • None
    • Reviewed
    • Hide
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.
      Show
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.

    Description

      Sometimes, the stream plugin may want to create/update its own global counters.
      There is no way to do so currently.
      One possible way to enable that is to use the stderr output of the stream process.
      The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
      to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.

      Attachments

        1. hadoop-1328-v5.patch
          7 kB
          Thomas White
        2. hadoop-1328-v4.patch
          7 kB
          Thomas White
        3. hadoop-1328-v3.patch
          7 kB
          Thomas White
        4. hadoop-1328-v2.patch
          7 kB
          Thomas White
        5. hadoop-1328.patch
          7 kB
          Thomas White

        Issue Links

          Activity

            People

              tomwhite Thomas White
              runping Runping Qi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: