Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None

      Description

      I want to add a writer that outputs a separate HDFS file for each post. The use case is performance testing. It's almost certainly useless for any production purpose.

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        This is basically just test code itself, and I don't think it's worth supplying unit tests for it. Works on my end, and I propose to commit it tomorrowish unless there are objections.

        Show
        asrabkin Ari Rabkin added a comment - This is basically just test code itself, and I don't think it's worth supplying unit tests for it. Works on my end, and I propose to commit it tomorrowish unless there are objections.
        Hide
        jboulon Jerome Boulon added a comment -

        Ari, What o you mean by "performance testing" ?

        Show
        jboulon Jerome Boulon added a comment - Ari, What o you mean by "performance testing" ?
        Hide
        asrabkin Ari Rabkin added a comment -

        I want to measure how bad the performance is with one file per post, in order to argue that something like our current rotation scheme is absolutely required.

        Show
        asrabkin Ari Rabkin added a comment - I want to measure how bad the performance is with one file per post, in order to argue that something like our current rotation scheme is absolutely required.
        Hide
        eyang Eric Yang added a comment -

        +1

        Show
        eyang Eric Yang added a comment - +1
        Hide
        asrabkin Ari Rabkin added a comment -

        Just committed this.

        Show
        asrabkin Ari Rabkin added a comment - Just committed this.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #239 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/239/ )
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #247 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/247/ )
        Hide
        asrabkin Ari Rabkin added a comment -

        Did the experiment, didn't get interesting results. For my setup, the namenode was able to keep up with one file per post, and I didn't observe a latency penalty.

        Show
        asrabkin Ari Rabkin added a comment - Did the experiment, didn't get interesting results. For my setup, the namenode was able to keep up with one file per post, and I didn't observe a latency penalty.

          People

          • Assignee:
            asrabkin Ari Rabkin
            Reporter:
            asrabkin Ari Rabkin
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development