Details

      Description

      Use FileSystem.globStatus to list hdfs file usage for all hadopp users

      1. chukwa-59.patch
        40 kB
        Cheng
      2. CHUKWA-59-1.patch
        1 kB
        Eric Yang
      3. CHUKWA-59-2.patch
        1 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          zhangyongjiang Cheng added a comment -

          Changes:

          1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage.

          2. A JPluginMapper to parse the data collected above

          3. Defined table schema and db/seqence file mapping

          4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.

          Show
          zhangyongjiang Cheng added a comment - Changes: 1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage. 2. A JPluginMapper to parse the data collected above 3. Defined table schema and db/seqence file mapping 4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.
          Hide
          zhangyongjiang Cheng added a comment -

          Changes:

          1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage.

          2. A JPluginMapper to parse the data collected above

          3. Defined table schema and db/seqence file mapping

          4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.

          Show
          zhangyongjiang Cheng added a comment - Changes: 1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage. 2. A JPluginMapper to parse the data collected above 3. Defined table schema and db/seqence file mapping 4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.
          Hide
          asrabkin Ari Rabkin added a comment -

          I notice that YahooOrgChart.java references some internal Yahoo! services. This makes me skittish.

          1) Does Yahoo! policy allow publicly released code to reference internal services?
          2) What happens to non-Yahoo! users of Chukwa?

          I'd be okay sticking in the public tree as "example code", provided it was clearly documented as such, provided Yahoo! signs off on that, and provided that users outside Yahoo! don't see weird artifacts.

          Show
          asrabkin Ari Rabkin added a comment - I notice that YahooOrgChart.java references some internal Yahoo! services. This makes me skittish. 1) Does Yahoo! policy allow publicly released code to reference internal services? 2) What happens to non-Yahoo! users of Chukwa? I'd be okay sticking in the public tree as "example code", provided it was clearly documented as such, provided Yahoo! signs off on that, and provided that users outside Yahoo! don't see weird artifacts.
          Hide
          macyang Mac Yang added a comment -

          Thanks for catching this. The file should not be included as is. New patch needed.

          Show
          macyang Mac Yang added a comment - Thanks for catching this. The file should not be included as is. New patch needed.
          Hide
          asrabkin Ari Rabkin added a comment -

          cancelled as per mac's comment.

          Show
          asrabkin Ari Rabkin added a comment - cancelled as per mac's comment.
          Hide
          zhangyongjiang Cheng added a comment -

          Removed code for Yahoo only.

          Show
          zhangyongjiang Cheng added a comment - Removed code for Yahoo only.
          Hide
          zhangyongjiang Cheng added a comment -

          Removed /grid/0/.... from the tools/service/chukwa-hdfsusage/run file

          Show
          zhangyongjiang Cheng added a comment - Removed /grid/0/.... from the tools/service/chukwa-hdfsusage/run file
          Hide
          zhangyongjiang Cheng added a comment -

          removed /grid/0/....

          Show
          zhangyongjiang Cheng added a comment - removed /grid/0/....
          Hide
          eyang Eric Yang added a comment -

          During the build time, the hdfs usage script should modify tools/service/chukwa-hdfsusage/run. This change should go into build.xml file.

          Show
          eyang Eric Yang added a comment - During the build time, the hdfs usage script should modify tools/service/chukwa-hdfsusage/run. This change should go into build.xml file.
          Hide
          eyang Eric Yang added a comment -

          Bundle build time changes to modify startup script for hdfsusage.

          Show
          eyang Eric Yang added a comment - Bundle build time changes to modify startup script for hdfsusage.
          Hide
          eyang Eric Yang added a comment -

          Add the missing LogEntry.java

          Show
          eyang Eric Yang added a comment - Add the missing LogEntry.java
          Hide
          zhangyongjiang Cheng added a comment -

          +1

          Show
          zhangyongjiang Cheng added a comment - +1
          Hide
          eyang Eric Yang added a comment -

          All patches are ready.

          Show
          eyang Eric Yang added a comment - All patches are ready.
          Hide
          eyang Eric Yang added a comment -

          I just committed this, thanks Cheng.

          Show
          eyang Eric Yang added a comment - I just committed this, thanks Cheng.

            People

            • Assignee:
              zhangyongjiang Cheng
              Reporter:
              zhangyongjiang Cheng
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development