Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1744

Add BigPigPetstore File and REST Load Generation.

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: blueprints
    • Labels:
      None

      Description

      We have an example of using BPS to generate data for spark.

      A more realistic thing, however, which is increasingly important, is streaming data.

      So, I would like to now add in https://github.com/jayunit100/PetStoreLoadGenerator/ as a direct project inside the bigtop-bigpetstore repository. Its essentially a way to generate (1) HTTP load or (2) Disk load using the existing BigPetStore data generation algorithm.

      Im tagging this as 0.9, because I think its a great feature to release with. I can create a patch this week. It will be a good way, for example, to do scale tests of flume and kafka.

      Open to ideas/opinions on this.

      1. 13.patch
        43 kB
        jay vyas
      2. BIGTOP-1744.patch
        44 kB
        jay vyas

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        jay vyas I don't think we can tag a new feature against a non-existing release. It it needs to be fixed in 0.9 then it should be marked in "Fix version" as such. But "Affects version" is usually on top of which the fix or new feature will get added. Hopefully, I making sense

        Show
        cos Konstantin Boudnik added a comment - jay vyas I don't think we can tag a new feature against a non-existing release. It it needs to be fixed in 0.9 then it should be marked in "Fix version" as such. But "Affects version" is usually on top of which the fix or new feature will get added. Hopefully, I making sense
        Hide
        rnowling RJ Nowling added a comment -

        jay vyas With some clean up, I think this would be a great addition to BigPetStore. Right now, you support output to rotating files and REST, right? If we added support for Flume or Kafka, this could be used to generate streaming data for BigPetStore Spark. I'd be happy to review any patches.

        I'm wondering if we should calling it the BigPetStore Streaming Data Generator Driver since it can be used for more than just load testing?

        Show
        rnowling RJ Nowling added a comment - jay vyas With some clean up, I think this would be a great addition to BigPetStore. Right now, you support output to rotating files and REST, right? If we added support for Flume or Kafka, this could be used to generate streaming data for BigPetStore Spark. I'd be happy to review any patches. I'm wondering if we should calling it the BigPetStore Streaming Data Generator Driver since it can be used for more than just load testing?
        Hide
        jayunit100 jay vyas added a comment -

        flume kafka can easily use it to read from the queue it creates as-is.

        i'll be cleaning it up this wk !

        Show
        jayunit100 jay vyas added a comment - flume kafka can easily use it to read from the queue it creates as-is. i'll be cleaning it up this wk !
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jayunit100 opened a pull request:

        https://github.com/apache/bigtop/pull/13

        BIGTOP-1744. Add BigPetStore REST/File ransaction queue to Apache BigTo...

        I'll manually attach a patch later, but here is a pull request to get started.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jayunit100/bigtop BIGTOP-1744

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/13.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #13


        commit c9b97283edbdfc2036ae3931708c199162b216be
        Author: jayunit100 <jayunit100@gmail.com>
        Date: 2015-03-10T17:01:03Z

        BIGTOP-1744. Add BigPetStore REST/File ransaction queue to Apache BigTop


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jayunit100 opened a pull request: https://github.com/apache/bigtop/pull/13 BIGTOP-1744 . Add BigPetStore REST/File ransaction queue to Apache BigTo... I'll manually attach a patch later, but here is a pull request to get started. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayunit100/bigtop BIGTOP-1744 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/13.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13 commit c9b97283edbdfc2036ae3931708c199162b216be Author: jayunit100 <jayunit100@gmail.com> Date: 2015-03-10T17:01:03Z BIGTOP-1744 . Add BigPetStore REST/File ransaction queue to Apache BigTop
        Hide
        jayunit100 jay vyas added a comment -

        RJ Nowling initial thoughts? I'll submit a patch shortly thereafter !

        Show
        jayunit100 jay vyas added a comment - RJ Nowling initial thoughts? I'll submit a patch shortly thereafter !
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26142986

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,147 @@
        +# BigPetStore transaction queue #
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        +
        +It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done.
        +
        +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack.
        +
        +# I'm in a hude rush, how do I use it? #
        — End diff –

        s/hude/huge/

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26142986 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,147 @@ +# BigPetStore transaction queue # + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. + +It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done. + +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack. + +# I'm in a hude rush, how do I use it? # — End diff – s/hude/huge/
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143016

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,147 @@
        +# BigPetStore transaction queue #
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        — End diff –

        Also describe HTTP REST?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143016 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,147 @@ +# BigPetStore transaction queue # + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. — End diff – Also describe HTTP REST?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143236

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/build.gradle —
        @@ -0,0 +1,42 @@
        +apply plugin: 'java'
        +apply plugin: 'idea'
        +apply plugin:'application'
        +mainClassName = "org.apache.bigtop.load.LoadGen"
        +
        +sourceCompatibility = 1.7
        +version = '1.0'
        +
        +// Begin fat jar reuirements..
        +buildscript {
        + repositories

        { + jcenter() + }

        +
        + dependencies

        { + classpath 'eu.appsatori:gradle-fatjar-plugin:0.3' + }

        +}
        +apply plugin: 'eu.appsatori.fatjar'
        +//end fat jar reuirements..
        +
        +fatJar

        { + exclude 'META-INF/*.DSA' +}

        +
        +repositories

        { + mavenCentral() +}

        +
        +dependencies {
        + compile 'com.fasterxml:jackson-module-json-org:0.9.1'
        + compile 'org.apache.commons:commons-io:1.3.2'
        + compile 'org.apache.httpcomponents:httpclient:4.4'
        + compile group:'org.apache.commons', name:'commons-lang3',version:'3.3.2'
        + compile files('libs/bigpetstore-data-generator-0.2.jar')
        — End diff –

        We should pull the jar from bintray. Look at the BPS Spark gradle file for directions on how to do this. Make sure to use the 0.2.1.jar not, 0.2. Important bug fix in there.

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143236 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/build.gradle — @@ -0,0 +1,42 @@ +apply plugin: 'java' +apply plugin: 'idea' +apply plugin:'application' +mainClassName = "org.apache.bigtop.load.LoadGen" + +sourceCompatibility = 1.7 +version = '1.0' + +// Begin fat jar reuirements.. +buildscript { + repositories { + jcenter() + } + + dependencies { + classpath 'eu.appsatori:gradle-fatjar-plugin:0.3' + } +} +apply plugin: 'eu.appsatori.fatjar' +//end fat jar reuirements.. + +fatJar { + exclude 'META-INF/*.DSA' +} + +repositories { + mavenCentral() +} + +dependencies { + compile 'com.fasterxml:jackson-module-json-org:0.9.1' + compile 'org.apache.commons:commons-io:1.3.2' + compile 'org.apache.httpcomponents:httpclient:4.4' + compile group:'org.apache.commons', name:'commons-lang3',version:'3.3.2' + compile files('libs/bigpetstore-data-generator-0.2.jar') — End diff – We should pull the jar from bintray. Look at the BPS Spark gradle file for directions on how to do this. Make sure to use the 0.2.1.jar not, 0.2. Important bug fix in there.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143331

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGen.java —
        @@ -0,0 +1,185 @@
        +/*
        + * Licensed to the Apache Software Foundation (ASF) under one or more
        + * contributor license agreements. See the NOTICE file distributed with
        + * this work for additional information regarding copyright ownership.
        + * The ASF licenses this file to You under the Apache License, Version 2.0
        + * (the "License"); you may not use this file except in compliance with
        + * the License. You may obtain a copy of the License at
        + *
        + * http://www.apache.org/licenses/LICENSE-2.0
        + *
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.bigtop.load;
        +
        +import com.github.rnowling.bps.datagenerator.datamodels.inputs.InputData;
        +import com.github.rnowling.bps.datagenerator.datamodels.inputs.ProductCategory;
        +import com.github.rnowling.bps.datagenerator.datamodels.inputs.ZipcodeRecord;
        +import com.github.rnowling.bps.datagenerator.datamodels.*;
        +//import com.github.rnowling.bps.datagenerator.*

        {DataLoader,StoreGenerator,CustomerGenerator => CustGen, PurchasingProfileGenerator,TransactionGenerator}

        +import com.github.rnowling.bps.datagenerator.*;
        +import com.github.rnowling.bps.datagenerator.framework.SeedFactory;
        +import com.google.common.collect.Lists;
        +
        +import java.io.DataOutput;
        +import java.io.DataOutputStream;
        +import java.net.HttpURLConnection;
        +import java.net.URL;
        +import java.nio.charset.Charset;
        +import java.nio.file.Files;
        +import java.nio.file.Path;
        +import java.nio.file.Paths;
        +import java.util.*;
        +import java.util.concurrent.LinkedBlockingDeque;
        +import java.util.concurrent.LinkedBlockingQueue;
        +
        +/**
        + * This driver uses the data generator API to generate
        + * an arbitrarily large data set of petstore transactions.
        + * <p/>
        + * Each "transaction" consists of many "products", each of which
        + * is stringified into what is often called a "line item".
        + * <p/>
        + * Then, spark writes those line items out as a distributed hadoop file glob.
        — End diff –

        No Spark here

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143331 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGen.java — @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.bigtop.load; + +import com.github.rnowling.bps.datagenerator.datamodels.inputs.InputData; +import com.github.rnowling.bps.datagenerator.datamodels.inputs.ProductCategory; +import com.github.rnowling.bps.datagenerator.datamodels.inputs.ZipcodeRecord; +import com.github.rnowling.bps.datagenerator.datamodels.*; +//import com.github.rnowling.bps.datagenerator.* {DataLoader,StoreGenerator,CustomerGenerator => CustGen, PurchasingProfileGenerator,TransactionGenerator} +import com.github.rnowling.bps.datagenerator.*; +import com.github.rnowling.bps.datagenerator.framework.SeedFactory; +import com.google.common.collect.Lists; + +import java.io.DataOutput; +import java.io.DataOutputStream; +import java.net.HttpURLConnection; +import java.net.URL; +import java.nio.charset.Charset; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.*; +import java.util.concurrent.LinkedBlockingDeque; +import java.util.concurrent.LinkedBlockingQueue; + +/** + * This driver uses the data generator API to generate + * an arbitrarily large data set of petstore transactions. + * <p/> + * Each "transaction" consists of many "products", each of which + * is stringified into what is often called a "line item". + * <p/> + * Then, spark writes those line items out as a distributed hadoop file glob. — End diff – No Spark here
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143423

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGenFactory.java —
        @@ -0,0 +1,106 @@
        +package org.apache.bigtop.load;
        +
        +import java.net.URL;
        +import java.nio.file.Paths;
        +import java.util.Random;
        +
        +/*
        +* Licensed to the Apache Software Foundation (ASF) under one or more
        +* contributor license agreements. See the NOTICE file distributed with
        +* this work for additional information regarding copyright ownership.
        +* The ASF licenses this file to You under the Apache License, Version 2.0
        +* (the "License"); you may not use this file except in compliance with
        +* the License. You may obtain a copy of the License at
        +*
        +* http://www.apache.org/licenses/LICENSE-2.0
        +*
        +* Unless required by applicable law or agreed to in writing, software
        +* distributed under the License is distributed on an "AS IS" BASIS,
        +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        +* See the License for the specific language governing permissions and
        +* limitations under the License.
        +*/
        +
        +/**
        + * Build a load generator based on command line args.
        + * This class decides, i.e. wether to use HTTP, FileLoadGen, and so on.
        — End diff –

        s/wether/whether/

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143423 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGenFactory.java — @@ -0,0 +1,106 @@ +package org.apache.bigtop.load; + +import java.net.URL; +import java.nio.file.Paths; +import java.util.Random; + +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to You under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ + +/** + * Build a load generator based on command line args. + * This class decides, i.e. wether to use HTTP, FileLoadGen, and so on. — End diff – s/wether/whether/
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143697

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/build.gradle —
        @@ -0,0 +1,42 @@
        +apply plugin: 'java'
        +apply plugin: 'idea'
        +apply plugin:'application'
        +mainClassName = "org.apache.bigtop.load.LoadGen"
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143697 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/build.gradle — @@ -0,0 +1,42 @@ +apply plugin: 'java' +apply plugin: 'idea' +apply plugin:'application' +mainClassName = "org.apache.bigtop.load.LoadGen" — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jayunit100 commented on the pull request:

        https://github.com/apache/bigtop/pull/13#issuecomment-78103460

        thanks ! also its not building ATM, will look into why....

        Show
        githubbot ASF GitHub Bot added a comment - Github user jayunit100 commented on the pull request: https://github.com/apache/bigtop/pull/13#issuecomment-78103460 thanks ! also its not building ATM, will look into why....
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143726

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/FileLoadGen.java —
        @@ -0,0 +1,75 @@
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143726 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/FileLoadGen.java — @@ -0,0 +1,75 @@ +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143745

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/HttpLoadGen.java —
        @@ -0,0 +1,97 @@
        +/*
        + * Licensed to the Apache Software Foundation (ASF) under one or more
        + * contributor license agreements. See the NOTICE file distributed with
        + * this work for additional information regarding copyright ownership.
        + * The ASF licenses this file to You under the Apache License, Version 2.0
        + * (the "License"); you may not use this file except in compliance with
        + * the License. You may obtain a copy of the License at
        + *
        + * http://www.apache.org/licenses/LICENSE-2.0
        + *
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143745 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/HttpLoadGen.java — @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143753

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGen.java —
        @@ -0,0 +1,185 @@
        +/*
        + * Licensed to the Apache Software Foundation (ASF) under one or more
        + * contributor license agreements. See the NOTICE file distributed with
        + * this work for additional information regarding copyright ownership.
        + * The ASF licenses this file to You under the Apache License, Version 2.0
        + * (the "License"); you may not use this file except in compliance with
        + * the License. You may obtain a copy of the License at
        + *
        + * http://www.apache.org/licenses/LICENSE-2.0
        + *
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143753 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGen.java — @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143771

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGenFactory.java —
        @@ -0,0 +1,106 @@
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143771 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/LoadGenFactory.java — @@ -0,0 +1,106 @@ +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143790

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/SimpleHttpServer.java —
        @@ -0,0 +1,129 @@
        +/*
        +* Licensed to the Apache Software Foundation (ASF) under one or more
        +* contributor license agreements. See the NOTICE file distributed with
        +* this work for additional information regarding copyright ownership.
        +* The ASF licenses this file to You under the Apache License, Version 2.0
        +* (the "License"); you may not use this file except in compliance with
        +* the License. You may obtain a copy of the License at
        +*
        +* http://www.apache.org/licenses/LICENSE-2.0
        +*
        +* Unless required by applicable law or agreed to in writing, software
        +* distributed under the License is distributed on an "AS IS" BASIS,
        +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        +* See the License for the specific language governing permissions and
        +* limitations under the License.
        +*/
        +
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143790 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/SimpleHttpServer.java — @@ -0,0 +1,129 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to You under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ + +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143799

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/Utils.java —
        @@ -0,0 +1,105 @@
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143799 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/main/java/org/apache/bigtop/load/Utils.java — @@ -0,0 +1,105 @@ +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26143810

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/test/java/org/apache/bigtop/load/TestLoadGen.java —
        @@ -0,0 +1,100 @@
        +/*
        +* Licensed to the Apache Software Foundation (ASF) under one or more
        +* contributor license agreements. See the NOTICE file distributed with
        +* this work for additional information regarding copyright ownership.
        +* The ASF licenses this file to You under the Apache License, Version 2.0
        +* (the "License"); you may not use this file except in compliance with
        +* the License. You may obtain a copy of the License at
        +*
        +* http://www.apache.org/licenses/LICENSE-2.0
        +*
        +* Unless required by applicable law or agreed to in writing, software
        +* distributed under the License is distributed on an "AS IS" BASIS,
        +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        +* See the License for the specific language governing permissions and
        +* limitations under the License.
        +*/
        +
        +package org.apache.bigtop.load;
        — End diff –

        I think the package should be "org.apache.bigtop.bigpetstore.load"

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26143810 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/src/test/java/org/apache/bigtop/load/TestLoadGen.java — @@ -0,0 +1,100 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to You under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ + +package org.apache.bigtop.load; — End diff – I think the package should be "org.apache.bigtop.bigpetstore.load"
        Hide
        rnowling RJ Nowling added a comment -

        jay vyas when it's building, let me know – I'll pull it down and test it out. For now, the three bigger issues I see are :

        • Package should probably be org.apache.bigtop.bigpetstore.load instead of org.apache.bigtop.bigpetstore.load. Maybe we should use .driver? This is a generally useful driver for the data generator – not just load testing
        • Gradle should pull the BPS data generator jar from bin tray, not depend on a copied jar file in /lib
        • We should use version 0.2.1 of the jar, not 0.2 – 0.2.1 includes a small but important bug fix for generating patterns in the data. Although it'll have no effect on load, it will affect anyone who analyzes the resulting data.
        Show
        rnowling RJ Nowling added a comment - jay vyas when it's building, let me know – I'll pull it down and test it out. For now, the three bigger issues I see are : Package should probably be org.apache.bigtop.bigpetstore.load instead of org.apache.bigtop.bigpetstore.load. Maybe we should use .driver? This is a generally useful driver for the data generator – not just load testing Gradle should pull the BPS data generator jar from bin tray, not depend on a copied jar file in /lib We should use version 0.2.1 of the jar, not 0.2 – 0.2.1 includes a small but important bug fix for generating patterns in the data. Although it'll have no effect on load, it will affect anyone who analyzes the resulting data.
        Hide
        jayunit100 jay vyas added a comment -

        Attaching official patch for review....

        Show
        jayunit100 jay vyas added a comment - Attaching official patch for review....
        Hide
        rnowling RJ Nowling added a comment -

        jay vyas When building, the tests never finishing running for me.

        Show
        rnowling RJ Nowling added a comment - jay vyas When building, the tests never finishing running for me.
        Hide
        jayunit100 jay vyas added a comment -

        never is a strong word...? really ?

        Show
        jayunit100 jay vyas added a comment - never is a strong word...? really ?
        Hide
        rnowling RJ Nowling added a comment -

        My cpu load jumps for a while then goes to 0. Can you tell me how long they run for you?

        Show
        rnowling RJ Nowling added a comment - My cpu load jumps for a while then goes to 0. Can you tell me how long they run for you?
        Hide
        rnowling RJ Nowling added a comment -

        I lie... finished after 5 min.

        Show
        rnowling RJ Nowling added a comment - I lie... finished after 5 min.
        Hide
        rnowling RJ Nowling added a comment -

        Comments:

        1. Tests Pass. Yay!

        2. There is no gradlew script. Should we change the README to use gradle since we don't include the gradlew script? Or do you want to add gradlew?

        3. You're using fatjar but other BPS projects (MR, Spark) use shadowJar. What are the advantages / disadvantages of one over the other?

        4. Noticed that when I unzipped the distribution created from distZip that it has the same jar twice and asks if I want to overwrite.

        5. instructions in README need to be updated for new package paths – none of them work right now. Dockerfile also needs to be updated because distribution zipfile name has changed.

        6. Can we include a manifest in the JAR so it can be run more succinctly (java -jar path/to/jar parameters...)?

        7. Can you add descriptions for the command-line arguments to the README?

        8. distZip instructions come first but aren't used – maybe move them to the end and add instructions for running from the distribution zipfile?

        9. Should we add instructions for building and running the Docker container locally? (vs asking users to pull it from your personal account?)

        10. I can't get it to run out of the distribution – fails to find the main class. This also prevents me from running the Docker container I built.

        Show
        rnowling RJ Nowling added a comment - Comments: 1. Tests Pass. Yay! 2. There is no gradlew script. Should we change the README to use gradle since we don't include the gradlew script? Or do you want to add gradlew? 3. You're using fatjar but other BPS projects (MR, Spark) use shadowJar. What are the advantages / disadvantages of one over the other? 4. Noticed that when I unzipped the distribution created from distZip that it has the same jar twice and asks if I want to overwrite. 5. instructions in README need to be updated for new package paths – none of them work right now. Dockerfile also needs to be updated because distribution zipfile name has changed. 6. Can we include a manifest in the JAR so it can be run more succinctly (java -jar path/to/jar parameters...)? 7. Can you add descriptions for the command-line arguments to the README? 8. distZip instructions come first but aren't used – maybe move them to the end and add instructions for running from the distribution zipfile? 9. Should we add instructions for building and running the Docker container locally? (vs asking users to pull it from your personal account?) 10. I can't get it to run out of the distribution – fails to find the main class. This also prevents me from running the Docker container I built.
        Hide
        jayunit100 jay vyas added a comment - - edited

        we don't really need shadowJar here : Its not running under a hadoop type system. shadowJar iirc has all the extension points. i think we generally use it only when we need to (i.e. mapreduce/spark).

        Im +1 on the other 9 points.

        Shall I resubmit w/ those fixes and keep fatJar? Or is there another reason to use shadowJar that you have in mind?

        Show
        jayunit100 jay vyas added a comment - - edited we don't really need shadowJar here : Its not running under a hadoop type system. shadowJar iirc has all the extension points. i think we generally use it only when we need to (i.e. mapreduce/spark). Im +1 on the other 9 points. Shall I resubmit w/ those fixes and keep fatJar? Or is there another reason to use shadowJar that you have in mind?
        Hide
        rnowling RJ Nowling added a comment -

        I have no strong feelings on shadowJar vs fatJar at the moment. I thought shadowJar may fix the duplicate jar issue but not worried if you fix it another way.

        If you fix the other 9, I'd be happy.

        Show
        rnowling RJ Nowling added a comment - I have no strong feelings on shadowJar vs fatJar at the moment. I thought shadowJar may fix the duplicate jar issue but not worried if you fix it another way. If you fix the other 9, I'd be happy.
        Hide
        jayunit100 jay vyas added a comment - - edited

        Okay folks. The patch is here now https://github.com/apache/bigtop/pull/13.patch

        • fixed up the jar manifests, so we can run w/ java -jar build/libs/bigpetstore-transaction-queue-all-1.0.jar /tmp 1 1 10000 1 now.
        • the commons-io transitive dependency was causing that zip overwrite. fixed it,using exclude directive from gradle... wow... that was harder than I thought it would be ...
        • Also patched up the README with updated invocations and some arguments info.

        RJ Nowling feel free to go ahead and verify it now. I think the README is pretty good now, might have missed some stuff, but hoping if any other udpates we can just add them in as time goes on.

        Show
        jayunit100 jay vyas added a comment - - edited Okay folks. The patch is here now https://github.com/apache/bigtop/pull/13.patch fixed up the jar manifests, so we can run w/ java -jar build/libs/bigpetstore-transaction-queue-all-1.0.jar /tmp 1 1 10000 1 now. the commons-io transitive dependency was causing that zip overwrite. fixed it,using exclude directive from gradle... wow... that was harder than I thought it would be ... Also patched up the README with updated invocations and some arguments info. RJ Nowling feel free to go ahead and verify it now. I think the README is pretty good now, might have missed some stuff, but hoping if any other udpates we can just add them in as time goes on.
        Hide
        rnowling RJ Nowling added a comment -

        Great! Thanks! I look forward to giving it another spin tomorrow

        Show
        rnowling RJ Nowling added a comment - Great! Thanks! I look forward to giving it another spin tomorrow
        Hide
        rnowling RJ Nowling added a comment -

        I was able to successfully:

        1. Create a fat jar and run the transaction queue from the fat jar
        2. Built a distribution zip, create a Docker container, and run the generator from the Docker container
        3. Run tests

        Comments:

        1. Jar name on lines 65 and 79 of the README should be bigpetstore-transaction-queue-all-1.0.jar, not bigpetstore-transaction-queue-1.0.jar
        2. On line 79 of the README, can we use the more succinct java -jar ... instead of the more verbose java -cp ... ?
        3. One lines 60 and 74 of the README, the path to the binary should /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue not /opt/bigpetstore-transaction-queue

        Once the README is updated, I can give a +1

        Show
        rnowling RJ Nowling added a comment - I was able to successfully: 1. Create a fat jar and run the transaction queue from the fat jar 2. Built a distribution zip, create a Docker container, and run the generator from the Docker container 3. Run tests Comments: 1. Jar name on lines 65 and 79 of the README should be bigpetstore-transaction-queue-all-1.0.jar, not bigpetstore-transaction-queue-1.0.jar 2. On line 79 of the README, can we use the more succinct java -jar ... instead of the more verbose java -cp ... ? 3. One lines 60 and 74 of the README, the path to the binary should /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue not /opt/bigpetstore-transaction-queue Once the README is updated, I can give a +1
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26237249

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,164 @@
        +# Introduction to the BigPetStore transaction queue#
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        +
        +There are many ways that data streams are consumed,
        +
        +- one is Files (files written every n seconds)
        +- another is via the web (peridoic HTTP PUTs)
        +
        +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done.
        +
        +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack.
        +
        +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer).
        +
        +# I'm in a huge rush, how do I use it? #
        +
        +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub).
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator
        +```
        +
        +After you run it, you can read below to customize it.
        +
        +# BigPetStore Data Generator #
        +
        +See related project on github, jayunit100/PetStoreBook, for example usage.
        +
        +An infiniterator for bigpetstore !
        +
        +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765). This repository writes bigpetstore transactions to
        +
        +- a rotating file queue OR
        +- a REST API calling queue.
        +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly !
        +- Its a little raw: I can clean it up later if people start getting interested.
        +
        +# Get Started by building the jar #
        +Clone this repository. Then just run
        +
        +```
        +gradle fatJar
        +```
        +
        +## Arguments ##
        +
        +The command line arguments are:
        +
        +- path (URL, for REST loads or directory on local disk, for file loads)
        +- stores (total number)
        +- customers (total number)
        +- simulation length (in days)
        +- seed (random seed for generator)
        +
        +## Generating FileSystem load ##
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0
        +```
        +or Java
        +```
        + gradle clean fatJar
        + java -jar ./build/libs/bigpetstore-transaction-queue-1.0.jar /tmp 1 5 10000 123
        — End diff –

        Jar name should be bigpetstore-transaction-queue-all-1.0.jar

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26237249 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,164 @@ +# Introduction to the BigPetStore transaction queue# + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. + +There are many ways that data streams are consumed, + +- one is Files (files written every n seconds) +- another is via the web (peridoic HTTP PUTs) + +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done. + +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack. + +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer). + +# I'm in a huge rush, how do I use it? # + +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub). +``` +docker run -t -i jayunit100/bigpetstore-load-generator +``` + +After you run it, you can read below to customize it. + +# BigPetStore Data Generator # + +See related project on github, jayunit100/PetStoreBook, for example usage. + +An infiniterator for bigpetstore ! + +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator ( http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765 ). This repository writes bigpetstore transactions to + +- a rotating file queue OR +- a REST API calling queue. +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly ! +- Its a little raw: I can clean it up later if people start getting interested. + +# Get Started by building the jar # +Clone this repository. Then just run + +``` +gradle fatJar +``` + +## Arguments ## + +The command line arguments are: + +- path (URL, for REST loads or directory on local disk, for file loads) +- stores (total number) +- customers (total number) +- simulation length (in days) +- seed (random seed for generator) + +## Generating FileSystem load ## + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0 +``` +or Java +``` + gradle clean fatJar + java -jar ./build/libs/bigpetstore-transaction-queue-1.0.jar /tmp 1 5 10000 123 — End diff – Jar name should be bigpetstore-transaction-queue-all-1.0.jar
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26237298

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,164 @@
        +# Introduction to the BigPetStore transaction queue#
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        +
        +There are many ways that data streams are consumed,
        +
        +- one is Files (files written every n seconds)
        +- another is via the web (peridoic HTTP PUTs)
        +
        +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done.
        +
        +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack.
        +
        +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer).
        +
        +# I'm in a huge rush, how do I use it? #
        +
        +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub).
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator
        +```
        +
        +After you run it, you can read below to customize it.
        +
        +# BigPetStore Data Generator #
        +
        +See related project on github, jayunit100/PetStoreBook, for example usage.
        +
        +An infiniterator for bigpetstore !
        +
        +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765). This repository writes bigpetstore transactions to
        +
        +- a rotating file queue OR
        +- a REST API calling queue.
        +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly !
        +- Its a little raw: I can clean it up later if people start getting interested.
        +
        +# Get Started by building the jar #
        +Clone this repository. Then just run
        +
        +```
        +gradle fatJar
        +```
        +
        +## Arguments ##
        +
        +The command line arguments are:
        +
        +- path (URL, for REST loads or directory on local disk, for file loads)
        +- stores (total number)
        +- customers (total number)
        +- simulation length (in days)
        +- seed (random seed for generator)
        +
        +## Generating FileSystem load ##
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0
        +```
        +or Java
        +```
        + gradle clean fatJar
        + java -jar ./build/libs/bigpetstore-transaction-queue-1.0.jar /tmp 1 5 10000 123
        +```
        +
        +## Generating REST load ##
        +OR Replace the file path with a REST API root (it will jsonify the transactions, and send them as the final url portion).
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue http://localhost:3000/restapi/rpush/ 1 5 10000 123
        +```
        +Or from java
        +```
        +gradle clean fatJar
        +java -cp ./build/libs/bigpetstore-transaction-queue-1.0.jar org.apache.bigtop.qstream.LoadGen http://localhost:3000/rpush/guestbook/ 4 4 1000 123
        — End diff –

        Jar name should be bigpetstore-transaction-queue-all-1.0.jar. Can shorter java -jar ... format be used here now?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26237298 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,164 @@ +# Introduction to the BigPetStore transaction queue# + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. + +There are many ways that data streams are consumed, + +- one is Files (files written every n seconds) +- another is via the web (peridoic HTTP PUTs) + +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done. + +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack. + +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer). + +# I'm in a huge rush, how do I use it? # + +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub). +``` +docker run -t -i jayunit100/bigpetstore-load-generator +``` + +After you run it, you can read below to customize it. + +# BigPetStore Data Generator # + +See related project on github, jayunit100/PetStoreBook, for example usage. + +An infiniterator for bigpetstore ! + +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator ( http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765 ). This repository writes bigpetstore transactions to + +- a rotating file queue OR +- a REST API calling queue. +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly ! +- Its a little raw: I can clean it up later if people start getting interested. + +# Get Started by building the jar # +Clone this repository. Then just run + +``` +gradle fatJar +``` + +## Arguments ## + +The command line arguments are: + +- path (URL, for REST loads or directory on local disk, for file loads) +- stores (total number) +- customers (total number) +- simulation length (in days) +- seed (random seed for generator) + +## Generating FileSystem load ## + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0 +``` +or Java +``` + gradle clean fatJar + java -jar ./build/libs/bigpetstore-transaction-queue-1.0.jar /tmp 1 5 10000 123 +``` + +## Generating REST load ## +OR Replace the file path with a REST API root (it will jsonify the transactions, and send them as the final url portion). + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue http://localhost:3000/restapi/rpush/ 1 5 10000 123 +``` +Or from java +``` +gradle clean fatJar +java -cp ./build/libs/bigpetstore-transaction-queue-1.0.jar org.apache.bigtop.qstream.LoadGen http://localhost:3000/rpush/guestbook/ 4 4 1000 123 — End diff – Jar name should be bigpetstore-transaction-queue-all-1.0.jar. Can shorter java -jar ... format be used here now?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26237353

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,164 @@
        +# Introduction to the BigPetStore transaction queue#
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        +
        +There are many ways that data streams are consumed,
        +
        +- one is Files (files written every n seconds)
        +- another is via the web (peridoic HTTP PUTs)
        +
        +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done.
        +
        +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack.
        +
        +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer).
        +
        +# I'm in a huge rush, how do I use it? #
        +
        +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub).
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator
        +```
        +
        +After you run it, you can read below to customize it.
        +
        +# BigPetStore Data Generator #
        +
        +See related project on github, jayunit100/PetStoreBook, for example usage.
        +
        +An infiniterator for bigpetstore !
        +
        +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765). This repository writes bigpetstore transactions to
        +
        +- a rotating file queue OR
        +- a REST API calling queue.
        +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly !
        +- Its a little raw: I can clean it up later if people start getting interested.
        +
        +# Get Started by building the jar #
        +Clone this repository. Then just run
        +
        +```
        +gradle fatJar
        +```
        +
        +## Arguments ##
        +
        +The command line arguments are:
        +
        +- path (URL, for REST loads or directory on local disk, for file loads)
        +- stores (total number)
        +- customers (total number)
        +- simulation length (in days)
        +- seed (random seed for generator)
        +
        +## Generating FileSystem load ##
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0
        — End diff –

        Binary path should be /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26237353 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,164 @@ +# Introduction to the BigPetStore transaction queue# + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. + +There are many ways that data streams are consumed, + +- one is Files (files written every n seconds) +- another is via the web (peridoic HTTP PUTs) + +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done. + +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack. + +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer). + +# I'm in a huge rush, how do I use it? # + +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub). +``` +docker run -t -i jayunit100/bigpetstore-load-generator +``` + +After you run it, you can read below to customize it. + +# BigPetStore Data Generator # + +See related project on github, jayunit100/PetStoreBook, for example usage. + +An infiniterator for bigpetstore ! + +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator ( http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765 ). This repository writes bigpetstore transactions to + +- a rotating file queue OR +- a REST API calling queue. +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly ! +- Its a little raw: I can clean it up later if people start getting interested. + +# Get Started by building the jar # +Clone this repository. Then just run + +``` +gradle fatJar +``` + +## Arguments ## + +The command line arguments are: + +- path (URL, for REST loads or directory on local disk, for file loads) +- stores (total number) +- customers (total number) +- simulation length (in days) +- seed (random seed for generator) + +## Generating FileSystem load ## + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0 — End diff – Binary path should be /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26237369

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,164 @@
        +# Introduction to the BigPetStore transaction queue#
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        +
        +There are many ways that data streams are consumed,
        +
        +- one is Files (files written every n seconds)
        +- another is via the web (peridoic HTTP PUTs)
        +
        +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done.
        +
        +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack.
        +
        +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer).
        +
        +# I'm in a huge rush, how do I use it? #
        +
        +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub).
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator
        +```
        +
        +After you run it, you can read below to customize it.
        +
        +# BigPetStore Data Generator #
        +
        +See related project on github, jayunit100/PetStoreBook, for example usage.
        +
        +An infiniterator for bigpetstore !
        +
        +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765). This repository writes bigpetstore transactions to
        +
        +- a rotating file queue OR
        +- a REST API calling queue.
        +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly !
        +- Its a little raw: I can clean it up later if people start getting interested.
        +
        +# Get Started by building the jar #
        +Clone this repository. Then just run
        +
        +```
        +gradle fatJar
        +```
        +
        +## Arguments ##
        +
        +The command line arguments are:
        +
        +- path (URL, for REST loads or directory on local disk, for file loads)
        +- stores (total number)
        +- customers (total number)
        +- simulation length (in days)
        +- seed (random seed for generator)
        +
        +## Generating FileSystem load ##
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0
        +```
        +or Java
        +```
        + gradle clean fatJar
        + java -jar ./build/libs/bigpetstore-transaction-queue-1.0.jar /tmp 1 5 10000 123
        +```
        +
        +## Generating REST load ##
        +OR Replace the file path with a REST API root (it will jsonify the transactions, and send them as the final url portion).
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue http://localhost:3000/restapi/rpush/ 1 5 10000 123
        — End diff –

        Binary path should be /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26237369 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,164 @@ +# Introduction to the BigPetStore transaction queue# + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. + +There are many ways that data streams are consumed, + +- one is Files (files written every n seconds) +- another is via the web (peridoic HTTP PUTs) + +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done. + +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack. + +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer). + +# I'm in a huge rush, how do I use it? # + +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub). +``` +docker run -t -i jayunit100/bigpetstore-load-generator +``` + +After you run it, you can read below to customize it. + +# BigPetStore Data Generator # + +See related project on github, jayunit100/PetStoreBook, for example usage. + +An infiniterator for bigpetstore ! + +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator ( http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765 ). This repository writes bigpetstore transactions to + +- a rotating file queue OR +- a REST API calling queue. +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly ! +- Its a little raw: I can clean it up later if people start getting interested. + +# Get Started by building the jar # +Clone this repository. Then just run + +``` +gradle fatJar +``` + +## Arguments ## + +The command line arguments are: + +- path (URL, for REST loads or directory on local disk, for file loads) +- stores (total number) +- customers (total number) +- simulation length (in days) +- seed (random seed for generator) + +## Generating FileSystem load ## + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue /tmp/ 1 1 3.0 +``` +or Java +``` + gradle clean fatJar + java -jar ./build/libs/bigpetstore-transaction-queue-1.0.jar /tmp 1 5 10000 123 +``` + +## Generating REST load ## +OR Replace the file path with a REST API root (it will jsonify the transactions, and send them as the final url portion). + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue http://localhost:3000/restapi/rpush/ 1 5 10000 123 — End diff – Binary path should be /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user rnowling commented on a diff in the pull request:

        https://github.com/apache/bigtop/pull/13#discussion_r26238608

        — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md —
        @@ -0,0 +1,165 @@
        +# Introduction to the BigPetStore transaction queue#
        +
        +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data.
        +
        +There are many ways that data streams are consumed,
        +
        +- one is Files (files written every n seconds)
        +- another is via the web (peridoic HTTP PUTs)
        +
        +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done.
        +
        +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack.
        +
        +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer).
        +
        +# I'm in a huge rush, how do I use it? #
        +
        +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub).
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator
        +```
        +
        +After you run it, you can read below to customize it.
        +
        +# BigPetStore Data Generator #
        +
        +See related project on github, jayunit100/PetStoreBook, for example usage.
        +
        +An infiniterator for bigpetstore !
        +
        +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765). This repository writes bigpetstore transactions to
        +
        +- a rotating file queue OR
        +- a REST API calling queue.
        +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly !
        +- Its a little raw: I can clean it up later if people start getting interested.
        +
        +# Get Started by building the jar #
        +Clone this repository. Then just run
        +
        +```
        +gradle fatJar
        +```
        +
        +## Arguments ##
        +
        +The command line arguments are:
        +
        +- path (URL, for REST loads or directory on local disk, for file loads)
        +- stores (total number)
        +- customers (total number)
        +- simulation length (in days)
        +- seed (random seed for generator)
        +
        +## Generating FileSystem load ##
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue /tmp/ 1 1 3.0
        +```
        +or Java
        +```
        + gradle clean fatJar
        + java -jar ./build/libs/bigpetstore-transaction-queue-all-1.0.jar /tmp 1 5 10000 123
        +```
        +
        +## Generating REST load ##
        +OR Replace the file path with a REST API root (it will jsonify the transactions, and send them as the final url portion).
        +
        +From docker
        +
        +```
        +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue
        + http://localhost:3000/restapi/rpush/ 1 5 10000 123
        +```
        +Or from java
        +```
        +gradle clean fatJar
        +java -cp ./build/libs/bigpetstore-transaction-queue-all-1.0.jar org.apache.bigtop.qstream.LoadGen http://localhost:3000/rpush/guestbook/ 4 4 1000 123
        — End diff –

        Can you use the shorter java -jar command?

        Show
        githubbot ASF GitHub Bot added a comment - Github user rnowling commented on a diff in the pull request: https://github.com/apache/bigtop/pull/13#discussion_r26238608 — Diff: bigtop-bigpetstore/bigpetstore-transaction-queue/README.md — @@ -0,0 +1,165 @@ +# Introduction to the BigPetStore transaction queue# + +This is a rotating queue for Apache BigTop which uses the BigPetStore libraries to generate realistic streaming transactional data. + +There are many ways that data streams are consumed, + +- one is Files (files written every n seconds) +- another is via the web (peridoic HTTP PUTs) + +Both of these are supported by this library. It runs infinitely as a single java process, repeatedly re running itself every time a simulation is done. + +It is a part of the BigPetStore ecosystem of projects which Apache BigTop leverages for real world validation of the stack. + +As a proof of concept, it has been used to test google's kubernetes framework by generating large load against Redis (a in memory caching data layer). + +# I'm in a huge rush, how do I use it? # + +Just run the dockerfile (eventually these dockerfiles will be stored in bigtop/ on dockerhub). +``` +docker run -t -i jayunit100/bigpetstore-load-generator +``` + +After you run it, you can read below to customize it. + +# BigPetStore Data Generator # + +See related project on github, jayunit100/PetStoreBook, for example usage. + +An infiniterator for bigpetstore ! + +If you're not familiar with bigpetstore, check out the apache bigtop project, which uses it to process data with spark, and the original whitepaper on the data generator ( http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7034765 ). This repository writes bigpetstore transactions to + +- a rotating file queue OR +- a REST API calling queue. +- You better have a consumer somewhere, otherwise it will fill up your disk very quickly ! +- Its a little raw: I can clean it up later if people start getting interested. + +# Get Started by building the jar # +Clone this repository. Then just run + +``` +gradle fatJar +``` + +## Arguments ## + +The command line arguments are: + +- path (URL, for REST loads or directory on local disk, for file loads) +- stores (total number) +- customers (total number) +- simulation length (in days) +- seed (random seed for generator) + +## Generating FileSystem load ## + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue /tmp/ 1 1 3.0 +``` +or Java +``` + gradle clean fatJar + java -jar ./build/libs/bigpetstore-transaction-queue-all-1.0.jar /tmp 1 5 10000 123 +``` + +## Generating REST load ## +OR Replace the file path with a REST API root (it will jsonify the transactions, and send them as the final url portion). + +From docker + +``` +docker run -t -i jayunit100/bigpetstore-load-generator /opt/bigpetstore-transaction-queue-1.0/bin/bigpetstore-transaction-queue + http://localhost:3000/restapi/rpush/ 1 5 10000 123 +``` +Or from java +``` +gradle clean fatJar +java -cp ./build/libs/bigpetstore-transaction-queue-all-1.0.jar org.apache.bigtop.qstream.LoadGen http://localhost:3000/rpush/guestbook/ 4 4 1000 123 — End diff – Can you use the shorter java -jar command?
        Hide
        jayunit100 jay vyas added a comment -

        final official patch for commit .

        Show
        jayunit100 jay vyas added a comment - final official patch for commit .
        Hide
        rnowling RJ Nowling added a comment -

        LGTM. +1. I'll go ahead and commit.

        Show
        rnowling RJ Nowling added a comment - LGTM. +1. I'll go ahead and commit.
        Hide
        rnowling RJ Nowling added a comment -

        Committed!

        Show
        rnowling RJ Nowling added a comment - Committed!

          People

          • Assignee:
            jayunit100 jay vyas
            Reporter:
            jayunit100 jay vyas
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development