Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-2467

Allow running Freon validators with limited memory

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • freon

    Description

      Freon validators read each item to be validated completely into a byte[] buffer. This allows timing only the read (and buffer allocation), but not the subsequent digest calculation. However, it also means that memory required for running the validators is proportional to key size.

      I propose to add a command-line flag to allow calculating the digest while reading the input stream. This changes timing results a bit, since values will include the time required for digest calculation. On the other hand, Freon will be able to validate huge keys with limited memory.

      Attachments

        Issue Links

          Activity

            People

              adoroszlai Attila Doroszlai
              adoroszlai Attila Doroszlai
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m