Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-1728

Need a way to assign storage entries to volumes

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.4
    • Component/s: Cache
    • Labels:
      None

      Description

      See http://www.knutsel.com/vol/compare/ for a write up and some test results on this. We are proposing a simple way to hard link storage.config entries to volume.config and thus hosting.config entries.

      1. vol.patch
        9 kB
        Jan van Doorn

        Issue Links

          Activity

          Hide
          zwoop Leif Hedstrom added a comment -

          Where are we with this? I haven't examine the patch yet, but will take a look. One thing that personally interests me is to make sure this is possible to extend into plugin APIs. We can land it without plugin API support, but should try to make sure we do it in a way that extending the plugin APIs later is fairly straight forward (i.e. no or few hoops).

          Show
          zwoop Leif Hedstrom added a comment - Where are we with this? I haven't examine the patch yet, but will take a look. One thing that personally interests me is to make sure this is possible to extend into plugin APIs. We can land it without plugin API support, but should try to make sure we do it in a way that extending the plugin APIs later is fairly straight forward (i.e. no or few hoops).
          Hide
          psudaemon Phil Sorber added a comment -

          I want to take a longer look at it, but from a first glance it looked good.

          I think the plugin requirement is a good one.

          Show
          psudaemon Phil Sorber added a comment - I want to take a longer look at it, but from a first glance it looked good. I think the plugin requirement is a good one.
          Hide
          zym Zhao Yongming added a comment -

          well, while I am looking at the origin docs on http://www.knutsel.com/vol/compare/, I think the design is set to handle very hot & small duration content in memory, for use of live broadcasting stream media etc. Can we assume that if we have ram only feature, and may assigned the way like hosting, then we don't need this anymore?

          Show
          zym Zhao Yongming added a comment - well, while I am looking at the origin docs on http://www.knutsel.com/vol/compare/ , I think the design is set to handle very hot & small duration content in memory, for use of live broadcasting stream media etc. Can we assume that if we have ram only feature, and may assigned the way like hosting, then we don't need this anymore?
          Hide
          zwoop Leif Hedstrom added a comment -

          I don't know for sure, but I don't think a RAM cache only solution works for them. They have a very large amount of content, and (I think?) 20-40TB of disk in each cache box. This proposal is a way to control (via configs and plugins) which contents has priority. It's a neat solution IMO, but not as generic as TS-745. In either case, i always felt it lacking that you can't control which disks a "volume" occupies, I can see plenty of use cases where I want to use expensive (say 15k SAS or SSD) for some content, and less expensive disks for content I care less about (e.g. where high latencies is fine).

          I understand the complication of combining TS-745 and TS-1728. This is also why I'm being a jerk and insisting that we make one config format for all disk / tiered caches control. Even if it means that for example tiered caching takes precedence over the volume identification, it's still a hell of lot clearer where this gets configured.

          Phil, James, Jan, John? Any thoughts on this?

          Show
          zwoop Leif Hedstrom added a comment - I don't know for sure, but I don't think a RAM cache only solution works for them. They have a very large amount of content, and (I think?) 20-40TB of disk in each cache box. This proposal is a way to control (via configs and plugins) which contents has priority. It's a neat solution IMO, but not as generic as TS-745 . In either case, i always felt it lacking that you can't control which disks a "volume" occupies, I can see plenty of use cases where I want to use expensive (say 15k SAS or SSD) for some content, and less expensive disks for content I care less about (e.g. where high latencies is fine). I understand the complication of combining TS-745 and TS-1728 . This is also why I'm being a jerk and insisting that we make one config format for all disk / tiered caches control. Even if it means that for example tiered caching takes precedence over the volume identification, it's still a hell of lot clearer where this gets configured. Phil, James, Jan, John? Any thoughts on this?
          Hide
          zwoop Leif Hedstrom added a comment -

          Alan brought up a good point, there are a few cache related configs, that with this patch would be useful to have as additional "tags" for storage.config as well. E.g.

          /dev/sda object_size=8k volume=1
          /dev/sdb object_size=1M volume=2
          

          This would then allow hosting.config (or a plugin) to assign content most suitable for a particular disk / cache config.

          Show
          zwoop Leif Hedstrom added a comment - Alan brought up a good point, there are a few cache related configs, that with this patch would be useful to have as additional "tags" for storage.config as well. E.g. /dev/sda object_size=8k volume=1 /dev/sdb object_size=1M volume=2 This would then allow hosting.config (or a plugin) to assign content most suitable for a particular disk / cache config.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1d8179df5bfbde4331ee94519c7a7a9d45f2ebe3 in branch refs/heads/issues/TS-1728 from Phil Sorber <sorber@apache.org>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1d8179d ]

          TS-1728: Assign storage entries to volumes

          Introduces the volume assignment in storage.config, on an optional
          basis. No change is needed in the config if you don't want to use
          this feature.

          • Add an optional reference to the volume number in storage.config,
            and the Span object, in Store.h and Store.cc
          • Add the volume reference to DiskVol, CacheProcessor::start_internal
            (in Cache.cc)
          • Change cplist_reconfigure (in Cache.cc) to constrain mapping
          • Fix bytes_used and percent_used per volume stats
          Show
          jira-bot ASF subversion and git services added a comment - Commit 1d8179df5bfbde4331ee94519c7a7a9d45f2ebe3 in branch refs/heads/issues/ TS-1728 from Phil Sorber <sorber@apache.org> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1d8179d ] TS-1728 : Assign storage entries to volumes Introduces the volume assignment in storage.config, on an optional basis. No change is needed in the config if you don't want to use this feature. Add an optional reference to the volume number in storage.config, and the Span object, in Store.h and Store.cc Add the volume reference to DiskVol, CacheProcessor::start_internal (in Cache.cc) Change cplist_reconfigure (in Cache.cc) to constrain mapping Fix bytes_used and percent_used per volume stats
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 28041adbabb766ef4d779762614ea75b745b0ac1 in branch refs/heads/sphinx-docs from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=28041ad ]

          TS-1728: Add to the docs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 28041adbabb766ef4d779762614ea75b745b0ac1 in branch refs/heads/sphinx-docs from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=28041ad ] TS-1728 : Add to the docs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c547df507a845b6c275931c17c0613f73603a425 in branch refs/heads/sphinx-docs from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=c547df5 ]

          Merge branch 'issues/TS-1728' into sphinx-docs

          Show
          jira-bot ASF subversion and git services added a comment - Commit c547df507a845b6c275931c17c0613f73603a425 in branch refs/heads/sphinx-docs from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=c547df5 ] Merge branch 'issues/ TS-1728 ' into sphinx-docs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1d8179df5bfbde4331ee94519c7a7a9d45f2ebe3 in branch refs/heads/sphinx-docs from Phil Sorber <sorber@apache.org>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1d8179d ]

          TS-1728: Assign storage entries to volumes

          Introduces the volume assignment in storage.config, on an optional
          basis. No change is needed in the config if you don't want to use
          this feature.

          • Add an optional reference to the volume number in storage.config,
            and the Span object, in Store.h and Store.cc
          • Add the volume reference to DiskVol, CacheProcessor::start_internal
            (in Cache.cc)
          • Change cplist_reconfigure (in Cache.cc) to constrain mapping
          • Fix bytes_used and percent_used per volume stats
          Show
          jira-bot ASF subversion and git services added a comment - Commit 1d8179df5bfbde4331ee94519c7a7a9d45f2ebe3 in branch refs/heads/sphinx-docs from Phil Sorber <sorber@apache.org> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1d8179d ] TS-1728 : Assign storage entries to volumes Introduces the volume assignment in storage.config, on an optional basis. No change is needed in the config if you don't want to use this feature. Add an optional reference to the volume number in storage.config, and the Span object, in Store.h and Store.cc Add the volume reference to DiskVol, CacheProcessor::start_internal (in Cache.cc) Change cplist_reconfigure (in Cache.cc) to constrain mapping Fix bytes_used and percent_used per volume stats
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6a071eb4d204bc99fb4202d05b0eb04c1cbc9257 in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=6a071eb ]

          Merge branch 'issues/TS-1728' into 'master'

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6a071eb4d204bc99fb4202d05b0eb04c1cbc9257 in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=6a071eb ] Merge branch 'issues/ TS-1728 ' into 'master'
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1d8179df5bfbde4331ee94519c7a7a9d45f2ebe3 in branch refs/heads/master from Phil Sorber <sorber@apache.org>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1d8179d ]

          TS-1728: Assign storage entries to volumes

          Introduces the volume assignment in storage.config, on an optional
          basis. No change is needed in the config if you don't want to use
          this feature.

          • Add an optional reference to the volume number in storage.config,
            and the Span object, in Store.h and Store.cc
          • Add the volume reference to DiskVol, CacheProcessor::start_internal
            (in Cache.cc)
          • Change cplist_reconfigure (in Cache.cc) to constrain mapping
          • Fix bytes_used and percent_used per volume stats
          Show
          jira-bot ASF subversion and git services added a comment - Commit 1d8179df5bfbde4331ee94519c7a7a9d45f2ebe3 in branch refs/heads/master from Phil Sorber <sorber@apache.org> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1d8179d ] TS-1728 : Assign storage entries to volumes Introduces the volume assignment in storage.config, on an optional basis. No change is needed in the config if you don't want to use this feature. Add an optional reference to the volume number in storage.config, and the Span object, in Store.h and Store.cc Add the volume reference to DiskVol, CacheProcessor::start_internal (in Cache.cc) Change cplist_reconfigure (in Cache.cc) to constrain mapping Fix bytes_used and percent_used per volume stats
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1dd14ab9679032d2a6f3a959d9f8da691917c9bf in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1dd14ab ]

          TS-1728: Cast float to int64_t

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1dd14ab9679032d2a6f3a959d9f8da691917c9bf in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1dd14ab ] TS-1728 : Cast float to int64_t
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6605fa880eec5effce56c56a333236f6cbb97f72 in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=6605fa8 ]

          TS-1728: Make some small cleanup as suggested.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6605fa880eec5effce56c56a333236f6cbb97f72 in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=6605fa8 ] TS-1728 : Make some small cleanup as suggested.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1158da9ef9facc408cf15d57b653decf0d5db2d5 in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1158da9 ]

          Revert "TS-1728: Make some small cleanup as suggested."

          This reverts commit 6605fa880eec5effce56c56a333236f6cbb97f72.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1158da9ef9facc408cf15d57b653decf0d5db2d5 in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1158da9 ] Revert " TS-1728 : Make some small cleanup as suggested." This reverts commit 6605fa880eec5effce56c56a333236f6cbb97f72.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3bf9738bd0a31285cb0069b1d3e4ae865440940d in branch refs/heads/issues/TS-1728 from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=3bf9738 ]

          TS-1728: Make compile without C++11

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3bf9738bd0a31285cb0069b1d3e4ae865440940d in branch refs/heads/issues/ TS-1728 from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=3bf9738 ] TS-1728 : Make compile without C++11
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 72216728e6ed1b168a4675fe6089933c5a329b07 in branch refs/heads/issues/TS-1728 from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=7221672 ]

          TS-1728: Make some small cleanup as suggested.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 72216728e6ed1b168a4675fe6089933c5a329b07 in branch refs/heads/issues/ TS-1728 from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=7221672 ] TS-1728 : Make some small cleanup as suggested.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e8658bf9040cd4f413891a60caa2df854f21125a in branch refs/heads/issues/TS-1728 from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=e8658bf ]

          TS-1728: Cast float to int64_t

          Show
          jira-bot ASF subversion and git services added a comment - Commit e8658bf9040cd4f413891a60caa2df854f21125a in branch refs/heads/issues/ TS-1728 from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=e8658bf ] TS-1728 : Cast float to int64_t
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a09dfb396435be105db4b85262f5774a0d0432af in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=a09dfb3 ]

          TS-1728: Make some small cleanup as suggested, but don't introduce
          C++11 only code this time.

          Show
          jira-bot ASF subversion and git services added a comment - Commit a09dfb396435be105db4b85262f5774a0d0432af in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=a09dfb3 ] TS-1728 : Make some small cleanup as suggested, but don't introduce C++11 only code this time.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 28041adbabb766ef4d779762614ea75b745b0ac1 in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=28041ad ]

          TS-1728: Add to the docs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 28041adbabb766ef4d779762614ea75b745b0ac1 in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=28041ad ] TS-1728 : Add to the docs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c547df507a845b6c275931c17c0613f73603a425 in branch refs/heads/master from Phil Sorber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=c547df5 ]

          Merge branch 'issues/TS-1728' into sphinx-docs

          Show
          jira-bot ASF subversion and git services added a comment - Commit c547df507a845b6c275931c17c0613f73603a425 in branch refs/heads/master from Phil Sorber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=c547df5 ] Merge branch 'issues/ TS-1728 ' into sphinx-docs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a8938835a0c62545f471fe79b256ff7c11ea2224 in trafficserver's branch refs/heads/master from Leif Hedstrom
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=a893883 ]

          Fix indentation from TS-1728

          Show
          jira-bot ASF subversion and git services added a comment - Commit a8938835a0c62545f471fe79b256ff7c11ea2224 in trafficserver's branch refs/heads/master from Leif Hedstrom [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=a893883 ] Fix indentation from TS-1728

            People

            • Assignee:
              psudaemon Phil Sorber
              Reporter:
              jvd Jan van Doorn
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development