Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.1.0
    • Component/s: BucketCache
    • Labels:
      None

      Description

      We did an internal brainstorming to study the feasibility of this. Some of our recent tests on SSDs like Optane shows that they are vastly faster in randomreads and can act as effective caches.
      In the current state we have a single tier of Bucket cache and the bucket cache can either be offheap or configured to work with file mode. (The file mode can have multiple files backing it).
      So this model restricts us from using either the memory or the file and not both.
      With the advent of faster devices like Optane SSDs, NVMe based devices it is better we try to utilize all those devices and try using them for the bucket cache so that we can avoid the impact of slower devices where the actual data resides on the HDFS data nodes.

      Combined with this we can allow the user to configure the caching layer per family/table so that one can effectively make use of the caching tiers.

      Can upload a design doc here. Before that, would like to know the suggestions here. Thoughts!!!

        Issue Links

          Activity

          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -
          Show
          ram_krish ramkrishna.s.vasudevan added a comment - Please find the link to the doc describing the feature. https://docs.google.com/document/d/1HF2GOSWXWoPapRwiKgMw2k516uyaTAWoEIunB72GVWU/edit?usp=sharing
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          Done some initial patch with the changes as described in the doc. Can test it in a cluster and report back the savings with different types of tiers. Any thoughts/feedback.

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - Done some initial patch with the changes as described in the doc. Can test it in a cluster and report back the savings with different types of tiers. Any thoughts/feedback.
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          Just did some initial testing with default bucket cache with 10G offheap bucket cache and a Tiered bucket cache with
          L1 - 10 G offheap
          L2 - 24 G optane SSD
          L3 - 24 G fultondale SSD (PCIe SSD)
          HDFS is configured to write and read from HDDs.
          I can see that with 80G data (total data set size) and with 75 threads during random reads we get 23% improvement. But this may not be a real time installation where people can have the HDFS itself in PCIe SSDs. So in that case L1 and L2 cache would be ideal choice. In cloud like deployment where there is only file based bucket cache they can allow memory based bucket cache also to be a tier.
          As mentioned in the doc there are some TODOs and some bucket cache related clean up JIRAs like heapspace occupancy of bucket cache etc will help here too.

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - Just did some initial testing with default bucket cache with 10G offheap bucket cache and a Tiered bucket cache with L1 - 10 G offheap L2 - 24 G optane SSD L3 - 24 G fultondale SSD (PCIe SSD) HDFS is configured to write and read from HDDs. I can see that with 80G data (total data set size) and with 75 threads during random reads we get 23% improvement. But this may not be a real time installation where people can have the HDFS itself in PCIe SSDs. So in that case L1 and L2 cache would be ideal choice. In cloud like deployment where there is only file based bucket cache they can allow memory based bucket cache also to be a tier. As mentioned in the doc there are some TODOs and some bucket cache related clean up JIRAs like heapspace occupancy of bucket cache etc will help here too.

            People

            • Assignee:
              ram_krish ramkrishna.s.vasudevan
              Reporter:
              ram_krish ramkrishna.s.vasudevan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development