Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-876

Dist Cache: Basic Functionality

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • storm-core
    • None

    Description

      Basic functionality for the Dist Cache feature.

      As part of this a new API should be added to support uploading and downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar should be written into the blob store instead of residing locally. We need a default implementation of the blob store that does essentially what nimbus currently does and does not need anything extra. But having an HDFS backend too would be great for scalability and HA.

      The supervisor should provide a way to download and manage these blobs and provide a working directory for the worker process with symlinks to the blobs. It should also allow the blobs to be updated and switch the symlink atomically to point to the new blob once it is downloaded.

      All of this is already done by code internal to Yahoo! we are in the process of getting it ready to push back to open source shortly.

      Attachments

        1. DISTCACHE.md
          14 kB
          Sanket Reddy
        2. DistributedCacheDesignDocument.pdf
          451 kB
          Sanket Reddy

        Issue Links

          Blocked

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. STORM-1174 Divide ACL struct into a union of Others and Users struct for Dist Cache

          • Major - Major loss of function.
          • Open

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. STORM-1166 Allow Groups in ACL for Dist Cache

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          blocks

          Improvement - An improvement or enhancement to an existing feature or task. STORM-877 Dist Cache: Worker API to know when blob has changed

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-878 Dist Cache: UI Displays current version of blobs on supervisor

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-879 Dist Cache: Quota/Usage Metrics Support

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-880 Dist Cache: allow files already in HDFS to be used by HDFS blob store

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-881 Dist Cache: Randomize the order of blobs downloaded

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-882 Dist Cache: Have HDFS blob store download starting at a random block

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-883 Dist Cache: Have supervisor throttle downloads to a set speed

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. STORM-884 Dist Cache: don't let blobstore users remove themselves from ACL

          • Major - Major loss of function.
          • Open
          is part of

          Umbrella - An overarching type made of sub-tasks STORM-411 Extend file uploads to support more distributed cache like semantics

          • Major - Major loss of function.
          • Open
          mentioned in

          Page Loading...

          Activity

            People

              redsanket Sanket Chintapalli
              revans2 Robert Joseph Evans
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: