Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-7666

[linearstore] Enhancements to the write buffer default and per-queue options

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: qpid-cpp-1.37.0
    • Fix Version/s: qpid-cpp-1.37.0
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      In applications where large numbers of persistent queues are created on a single broker, control over the default and per-queue store options are important so that resources are not wasted or become exhausted prematurely.

      There are two resources of concern:

      1. Memory allocated per queue for buffering;
      2. Disk resources consumed for journal files.

      Currently, the default setting for memory allocation is 1024kiB/queue (ie 32 pages of 32kiB). The only default setting available is the page size (using the wcache-page-size option), and the number of pages is set internally according to the following table:

      wcache-page-size wcache-num-pages total mmap
      (kiB) (set interanlly) (kiB)
      4 64 256
      8 64 512
      16 32 512
      32* 32 1024
      64 16 1024
      128 8 1024

      * default

      This gives in effect only 3 per-queue memory allocation options: 256kiB, 512kiB and 1024kiB.

      Currently, the values of wcache-page-size is limited to the powers of 2 between 4 and 128.

      It is suggested that the wcache-num-pages value be set as as an option rather than internally, which would give the user much greater control over the per-queue memory footprint and the buffer geometry (which can be optimized for the disk hardware and queue load characteristics).

      In addition, the upper value limit on wcache-page-size should be removed (as it is rather arbitrary) and the default value should be revised.

      Further, it would be valuable to add these parameters to qpid-config so that if a queue is created using this tool, the default values for the store parameters can be overridden on a per-queue basis.

      Disk resource controls are currently in place for the EFP partition (efp-partition) and the size of the EFP files (efp-file-size), both as a default and in qpid-config. The default values should also be revised.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kpvdr Kim van der Riet
                Reporter:
                kpvdr Kim van der Riet
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: