Chukwa
  1. Chukwa
  2. CHUKWA-460

Demux depends on conf/chukwa-collector-conf.xml

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.

      The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #330 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/330/ )
        Hide
        Eric Yang added a comment -

        I just committed this to both TRUNK and 0.4. Thanks Ari.

        Show
        Eric Yang added a comment - I just committed this to both TRUNK and 0.4. Thanks Ari.
        Hide
        Eric Yang added a comment -

        The documentation did not show writer.hdfs.filesystem, was required for demux. This change actually conforms to what is currently available on document.

        Show
        Eric Yang added a comment - The documentation did not show writer.hdfs.filesystem, was required for demux. This change actually conforms to what is currently available on document.
        Hide
        Ari Rabkin added a comment -

        +1 to patch. For trunk as well as 0.4 branch. Do we need to do a documentation update?

        Show
        Ari Rabkin added a comment - +1 to patch. For trunk as well as 0.4 branch. Do we need to do a documentation update?
        Hide
        Eric Yang added a comment -

        This patch will remove the reference of writer.hdfs.filesystem from demux and archive, and use HADOOP_CONF_DIR instead.

        Show
        Eric Yang added a comment - This patch will remove the reference of writer.hdfs.filesystem from demux and archive, and use HADOOP_CONF_DIR instead.
        Hide
        Bill Graham added a comment -

        Testing the collector it appears it does not have this dependency, but instead picks up the HDFS configs correctly from the hadoop configs. Which is good, albeit confusing.

        So really this config should be moved to the demux configs for now, to clear up potential confusion over who reads that value. Or better yet, Eric's approach of removing the config entirely and fixing Demux to not use that value should be followed.

        Show
        Bill Graham added a comment - Testing the collector it appears it does not have this dependency, but instead picks up the HDFS configs correctly from the hadoop configs. Which is good, albeit confusing. So really this config should be moved to the demux configs for now, to clear up potential confusion over who reads that value. Or better yet, Eric's approach of removing the config entirely and fixing Demux to not use that value should be followed.
        Hide
        Eric Yang added a comment -

        Demux process should not be using writer.hdfs.filesystem. The original implementation made a mistake to use "writer.hdfs.filesystem" as reference to hdfs. Instead, demux should pick up the proper hdfs url from HADOOP_CONF_DIR. I don't have any objection on the suggested workaround, but less configuration that we need to maintain, the better.

        Show
        Eric Yang added a comment - Demux process should not be using writer.hdfs.filesystem. The original implementation made a mistake to use "writer.hdfs.filesystem" as reference to hdfs. Instead, demux should pick up the proper hdfs url from HADOOP_CONF_DIR. I don't have any objection on the suggested workaround, but less configuration that we need to maintain, the better.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development