Details

      Description

      Use FileSystem.globStatus to list hdfs file usage for all hadopp users

      1. chukwa-59.patch
        40 kB
        Cheng
      2. CHUKWA-59-1.patch
        1 kB
        Eric Yang
      3. CHUKWA-59-2.patch
        1 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Eric Yang added a comment -

          I just committed this, thanks Cheng.

          Show
          Eric Yang added a comment - I just committed this, thanks Cheng.
          Hide
          Eric Yang added a comment -

          All patches are ready.

          Show
          Eric Yang added a comment - All patches are ready.
          Hide
          Cheng added a comment -

          +1

          Show
          Cheng added a comment - +1
          Hide
          Eric Yang added a comment -

          Add the missing LogEntry.java

          Show
          Eric Yang added a comment - Add the missing LogEntry.java
          Hide
          Eric Yang added a comment -

          Bundle build time changes to modify startup script for hdfsusage.

          Show
          Eric Yang added a comment - Bundle build time changes to modify startup script for hdfsusage.
          Hide
          Eric Yang added a comment -

          During the build time, the hdfs usage script should modify tools/service/chukwa-hdfsusage/run. This change should go into build.xml file.

          Show
          Eric Yang added a comment - During the build time, the hdfs usage script should modify tools/service/chukwa-hdfsusage/run. This change should go into build.xml file.
          Hide
          Cheng added a comment -

          removed /grid/0/....

          Show
          Cheng added a comment - removed /grid/0/....
          Hide
          Cheng added a comment -

          Removed /grid/0/.... from the tools/service/chukwa-hdfsusage/run file

          Show
          Cheng added a comment - Removed /grid/0/.... from the tools/service/chukwa-hdfsusage/run file
          Hide
          Cheng added a comment -

          Removed code for Yahoo only.

          Show
          Cheng added a comment - Removed code for Yahoo only.
          Hide
          Ari Rabkin added a comment -

          cancelled as per mac's comment.

          Show
          Ari Rabkin added a comment - cancelled as per mac's comment.
          Hide
          Mac Yang added a comment -

          Thanks for catching this. The file should not be included as is. New patch needed.

          Show
          Mac Yang added a comment - Thanks for catching this. The file should not be included as is. New patch needed.
          Hide
          Ari Rabkin added a comment -

          I notice that YahooOrgChart.java references some internal Yahoo! services. This makes me skittish.

          1) Does Yahoo! policy allow publicly released code to reference internal services?
          2) What happens to non-Yahoo! users of Chukwa?

          I'd be okay sticking in the public tree as "example code", provided it was clearly documented as such, provided Yahoo! signs off on that, and provided that users outside Yahoo! don't see weird artifacts.

          Show
          Ari Rabkin added a comment - I notice that YahooOrgChart.java references some internal Yahoo! services. This makes me skittish. 1) Does Yahoo! policy allow publicly released code to reference internal services? 2) What happens to non-Yahoo! users of Chukwa? I'd be okay sticking in the public tree as "example code", provided it was clearly documented as such, provided Yahoo! signs off on that, and provided that users outside Yahoo! don't see weird artifacts.
          Hide
          Cheng added a comment -

          Changes:

          1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage.

          2. A JPluginMapper to parse the data collected above

          3. Defined table schema and db/seqence file mapping

          4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.

          Show
          Cheng added a comment - Changes: 1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage. 2. A JPluginMapper to parse the data collected above 3. Defined table schema and db/seqence file mapping 4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.
          Hide
          Cheng added a comment -

          Changes:

          1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage.

          2. A JPluginMapper to parse the data collected above

          3. Defined table schema and db/seqence file mapping

          4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.

          Show
          Cheng added a comment - Changes: 1. Use FileSystem.globStatus(new Path("/user/*")) to get all users' file usage. 2. A JPluginMapper to parse the data collected above 3. Defined table schema and db/seqence file mapping 4. Changed exec-data-loader.sh, processSinkFiles.sh and systemDataLoader.sh to have this new metrics included in the pipeline.

            People

            • Assignee:
              Cheng
              Reporter:
              Cheng
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development