Chukwa
  1. Chukwa
  2. CHUKWA-410

Does the BackfillingLoader return only after HDFS blocks are committed?

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: 0.3.0
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None
    • Environment:

      Hadoop 0.20.0, Debian 4 (Etch), Chukwa rev 817532

      Description

      I see that the BackfillingLoader is set to AdaptorShutdownPolicy.WAIT_TILL_FINISHED, what are the semantics of this? Does this mean that the BackfillingLoader returns after the last HDFS write request is made, but the DFSClient could continue to be flushing blocks to the DataNodes in the background? Or does that mean that the entire file has been written/flushed to HDFS and closed and fully available?

      I'm running the Demux immediately after the BackfillingLoader is complete; the raw log files are complete, but the Demux picks up only half of the entries in those log files. Could this be because some blocks are not closed yet?

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Jiaqi Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development