Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-876

Dist Cache: Basic Functionality

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: storm-core
    • Labels:
      None

      Description

      Basic functionality for the Dist Cache feature.

      As part of this a new API should be added to support uploading and downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar should be written into the blob store instead of residing locally. We need a default implementation of the blob store that does essentially what nimbus currently does and does not need anything extra. But having an HDFS backend too would be great for scalability and HA.

      The supervisor should provide a way to download and manage these blobs and provide a working directory for the worker process with symlinks to the blobs. It should also allow the blobs to be updated and switch the symlink atomically to point to the new blob once it is downloaded.

      All of this is already done by code internal to Yahoo! we are in the process of getting it ready to push back to open source shortly.

        Attachments

        1. DISTCACHE.md
          14 kB
          Sanket Reddy
        2. DistributedCacheDesignDocument.pdf
          451 kB
          Sanket Reddy

          Issue Links

            Activity

              People

              • Assignee:
                redsanket Sanket Chintapalli
                Reporter:
                revans2 Robert Joseph Evans
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: