Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1918

Add product enumeration to BPS data generator

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: blueprints
    • Labels:
      None

      Description

      The BigPetStore data generator currently reads products from a JSON file. Manually creating entries is time consuming and limits our ability to add products with lots of fields. We want more fields so we can generate a wider range of patterns.

      I suggest adding a product generator which takes a description of every field and its possible values and generates a Cartesian product of every possible product. Logical rules for be used for filtering out certain combinations. This will significantly reduce the effort required to add a large number of products.

        Activity

        Hide
        rnowling RJ Nowling added a comment -

        This patch:

        1. Removes code for reading products from the JSON file and the JSON file itself.

        2. Adds a new ProductGenerator subsystem which includes routines for computing cartesian products, a simple boolean logic system, iterators for building and filtering products, and a conversion of the JSON file into a class for using the new ProductGenerator.

        Show
        rnowling RJ Nowling added a comment - This patch: 1. Removes code for reading products from the JSON file and the JSON file itself. 2. Adds a new ProductGenerator subsystem which includes routines for computing cartesian products, a simple boolean logic system, iterators for building and filtering products, and a conversion of the JSON file into a class for using the new ProductGenerator.
        Hide
        rnowling RJ Nowling added a comment -

        jay vyas Can you review? thanks!

        Show
        rnowling RJ Nowling added a comment - jay vyas Can you review? thanks!
        Hide
        rnowling RJ Nowling added a comment -

        I wrote up a blog post describing the proposed approach for generating products.

        Show
        rnowling RJ Nowling added a comment - I wrote up a blog post describing the proposed approach for generating products.
        Hide
        jayunit100 jay vyas added a comment - - edited

        +1 working for me and tested, thanks RJ.

        • minor nit you can fix later, or fix now quickly before commiting (no re-review needed from my end) up to you .. add a comment to public static final String PRODUCTS_COLLECTION = "small"; I see the "SmallProductCollection"... But from the code its not clear what "small" represents. Clearly its just an annotation of some sort. b/c i can change it and the code still compiles.

        cc Andrew Purtell this will be useful for large column spaces simulations in hbase

        Show
        jayunit100 jay vyas added a comment - - edited +1 working for me and tested, thanks RJ. minor nit you can fix later, or fix now quickly before commiting (no re-review needed from my end) up to you .. add a comment to public static final String PRODUCTS_COLLECTION = "small"; I see the "SmallProductCollection"... But from the code its not clear what "small" represents. Clearly its just an annotation of some sort. b/c i can change it and the code still compiles. cc Andrew Purtell this will be useful for large column spaces simulations in hbase
        Hide
        rnowling RJ Nowling added a comment -

        Thanks for the review jay vyas! For now, the Constant is a placeholder – I actually ignore it. When I add more product collections, I'll add some comments around that. Thanks!

        Show
        rnowling RJ Nowling added a comment - Thanks for the review jay vyas ! For now, the Constant is a placeholder – I actually ignore it. When I add more product collections, I'll add some comments around that. Thanks!
        Hide
        rnowling RJ Nowling added a comment -

        Committed

        Show
        rnowling RJ Nowling added a comment - Committed

          People

          • Assignee:
            rnowling RJ Nowling
            Reporter:
            rnowling RJ Nowling
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development