Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-648

Make Chukwa Reduce Type to support hierarchy format

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0, 0.6.0
    • Fix Version/s: None
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Currently the Reduce Type decides the final directory-structure of the processed data over HDFS. And it doesn't support a hierarchy format like

      hdfs/namenode
      hdfs/datanode
      ...
      

      A hierarchy structure makes it easy to do the HDFS File Filter versus to the flat storage structure while doing the analysis job.

      1. Chukwa-648.patch
        28 kB
        Jie Huang
      2. Chukwa-648-0_4.patch
        17 kB
        Jie Huang

        Activity

        Hide
        grace.huang Jie Huang added a comment -

        Here attaches the simple solution against 0.4 version. If the feature is acceptable, I will try to upgrade that patch against trunk.

        Show
        grace.huang Jie Huang added a comment - Here attaches the simple solution against 0.4 version. If the feature is acceptable, I will try to upgrade that patch against trunk.
        Hide
        asrabkin Ari Rabkin added a comment -

        Overall approach looks solid.

        A few quibbles about the patch.

        HierarchyDataType shouldn't have an empty main() method.

        HierarchyDataType SHOULD have a top-level Javadoc comment saying a bit about what it's for and where it's used.

        Show
        asrabkin Ari Rabkin added a comment - Overall approach looks solid. A few quibbles about the patch. HierarchyDataType shouldn't have an empty main() method. HierarchyDataType SHOULD have a top-level Javadoc comment saying a bit about what it's for and where it's used.
        Hide
        grace.huang Jie Huang added a comment -

        Thanks for your comments. I will update that file later. thanks.

        Show
        grace.huang Jie Huang added a comment - Thanks for your comments. I will update that file later. thanks.
        Hide
        grace.huang Jie Huang added a comment -

        Here attaches the patch file against trunk. All the unit tests are passed on my local machine.
        @Ari , could you please review the patch file? Thanks for your time.

        Show
        grace.huang Jie Huang added a comment - Here attaches the patch file against trunk. All the unit tests are passed on my local machine. @Ari , could you please review the patch file? Thanks for your time.
        Hide
        asrabkin Ari Rabkin added a comment -

        I committed this one too. Thanks again for your participation and interest in Chukwa.

        Show
        asrabkin Ari Rabkin added a comment - I committed this one too. Thanks again for your participation and interest in Chukwa.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #455 (See https://builds.apache.org/job/Chukwa-trunk/455/)
        CHUKWA-648. Make Chukwa Reduce Type to support hierarchy format. Contributed by Jie Huang. (Revision 1367843)

        Result = FAILURE
        asrabkin : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367843
        Files :

        • /incubator/chukwa/trunk/CHANGES.txt
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/CHUKWA_CONSTANT.java
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/ChukwaRecordOutputFormat.java
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/DailyChukwaRecordRolling.java
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/HourlyChukwaRecordRolling.java
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/MoveToRepository.java
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/PostProcessorManager.java
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #455 (See https://builds.apache.org/job/Chukwa-trunk/455/ ) CHUKWA-648 . Make Chukwa Reduce Type to support hierarchy format. Contributed by Jie Huang. (Revision 1367843) Result = FAILURE asrabkin : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367843 Files : /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/CHUKWA_CONSTANT.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/ChukwaRecordOutputFormat.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/DailyChukwaRecordRolling.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/HourlyChukwaRecordRolling.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/MoveToRepository.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/PostProcessorManager.java
        Hide
        eyang Eric Yang added a comment -

        This patch doesn't compile after check into trunk.

        Show
        eyang Eric Yang added a comment - This patch doesn't compile after check into trunk.
        Hide
        grace.huang Jie Huang added a comment -

        @Eric, I have checked that patch file, it does contain "src/main/java/org/apache/hadoop/chukwa/util/HierarchyDataType.java" file. and I have checked all the unit test before uploading that patch file as well. I guess the possible cause is that svn doesn't detect that new adding file.

        /incubator/chukwa/trunk/CHANGES.txt
        /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/CHUKWA_CONSTANT.java
        /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/ChukwaRecordOutputFormat.java
        /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/DailyChukwaRecordRolling.java
        /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/HourlyChukwaRecordRolling.java
        /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/MoveToRepository.java
        /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/PostProcessorManager.java
        <<< NO HierarchyDataType.java file here.
        
        Show
        grace.huang Jie Huang added a comment - @Eric, I have checked that patch file, it does contain "src/main/java/org/apache/hadoop/chukwa/util/HierarchyDataType.java" file. and I have checked all the unit test before uploading that patch file as well. I guess the possible cause is that svn doesn't detect that new adding file. /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/CHUKWA_CONSTANT.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/ChukwaRecordOutputFormat.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/DailyChukwaRecordRolling.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/HourlyChukwaRecordRolling.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/MoveToRepository.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/PostProcessorManager.java <<< NO HierarchyDataType.java file here.
        Hide
        eyang Eric Yang added a comment -

        Ari, I think you forgot to add the new file to svn. Could you add the file?

        Show
        eyang Eric Yang added a comment - Ari, I think you forgot to add the new file to svn. Could you add the file?
        Hide
        eyang Eric Yang added a comment -

        I just committed the missing file. Thanks Grace.

        Show
        eyang Eric Yang added a comment - I just committed the missing file. Thanks Grace.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #456 (See https://builds.apache.org/job/Chukwa-trunk/456/)
        CHUKWA-648. Make Chukwa Reduce Type to support hierarchy format. (Jie Huang via asrabkin) (Revision 1369519)

        Result = SUCCESS
        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369519
        Files :

        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/util/HierarchyDataType.java
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #456 (See https://builds.apache.org/job/Chukwa-trunk/456/ ) CHUKWA-648 . Make Chukwa Reduce Type to support hierarchy format. (Jie Huang via asrabkin) (Revision 1369519) Result = SUCCESS eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369519 Files : /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/util/HierarchyDataType.java

          People

          • Assignee:
            grace.huang Jie Huang
            Reporter:
            grace.huang Jie Huang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development