Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1653

Add queries for customer, state, and product statistics w/ d3 friendly JSON output to analytics phase.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Component/s: blueprints
    • Labels:
      None

      Description

      Follow on to BIGTOP-1536, this time we can use a scala json library if a good one exists.

      1. BIGTOP-1653.patch
        19 kB
        jay vyas
      2. BIGTOP-1653.patch
        17 kB
        jay vyas
      3. BIGTOP-1653.patch
        28 kB
        jay vyas
      4. BIGTOP-1653.patch
        4 kB
        jay vyas
      5. diff.patch
        8 kB
        jay vyas
      6. diff.patch
        6 kB
        jay vyas

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          Update. For the first part of this, I've implemented a simple spark sql query in PetStoreStatistics.scala like this.

                   def totalTransactions(r:(_,_,_,_,RDD[Transaction]), sc: SparkContext): Statistics = {
               val sqlContext = new org.apache.spark.sql.SQLContext(sc);
                import sqlContext._;
                r._5.registerTempTable("transactions");
                val results: SchemaRDD = sql("SELECT * FROM transactions")
                return Statistics(results.count());
          

          This fails currently, possibly a serialization error... ?

          java.util.Calendar (of class scala.reflect.internal.Types$TypeRef$$anon$6)
          scala.MatchError: java.util.Calendar (of class scala.reflect.internal.Types$TypeRef$$anon$6)
          	at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:53)
          	at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:64)
          	at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:62)
          	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
          	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
          	at scala.collection.immutable.List.foreach(List.scala:318)
          	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
          	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
          	at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:62)
          	at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:50)
          	at org.apache.spark.sql.catalyst.ScalaReflection$.attributesFor(ScalaReflection.scala:44)
          	at org.apache.spark.sql.execution.ExistingRdd$.fromProductRdd(basicOperators.scala:229)
          	at org.apache.spark.sql.SQLContext.createSchemaRDD(SQLContext.scala:94)
          	at org.apache.bigtop.bigpetstore.spark.generator.PetStoreStatistics$.totalTransactions(PetStoreStatistics.scala:76)
          

          will look more later

          Show
          jayunit100 jay vyas added a comment - Update. For the first part of this, I've implemented a simple spark sql query in PetStoreStatistics.scala like this. def totalTransactions(r:(_,_,_,_,RDD[Transaction]), sc: SparkContext): Statistics = { val sqlContext = new org.apache.spark.sql.SQLContext(sc); import sqlContext._; r._5.registerTempTable("transactions"); val results: SchemaRDD = sql("SELECT * FROM transactions") return Statistics(results.count()); This fails currently, possibly a serialization error... ? java.util.Calendar (of class scala.reflect.internal.Types$TypeRef$$anon$6) scala.MatchError: java.util.Calendar (of class scala.reflect.internal.Types$TypeRef$$anon$6) at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:53) at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:64) at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:62) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:62) at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:50) at org.apache.spark.sql.catalyst.ScalaReflection$.attributesFor(ScalaReflection.scala:44) at org.apache.spark.sql.execution.ExistingRdd$.fromProductRdd(basicOperators.scala:229) at org.apache.spark.sql.SQLContext.createSchemaRDD(SQLContext.scala:94) at org.apache.bigtop.bigpetstore.spark.generator.PetStoreStatistics$.totalTransactions(PetStoreStatistics.scala:76) will look more later
          Hide
          jayunit100 jay vyas added a comment - - edited

          confirmed its a error specific to Calendar (may or may not be serialization)... The same code works fine w/ Locations or other case classes...

          Show
          jayunit100 jay vyas added a comment - - edited confirmed its a error specific to Calendar (may or may not be serialization)... The same code works fine w/ Locations or other case classes...
          Hide
          jayunit100 jay vyas added a comment -

          update patch. now we can query by month year etc.. workaround was to use jodaTime, and then extract month/yr/ so on into separate fields w/ RDD conversion.

          Show
          jayunit100 jay vyas added a comment - update patch. now we can query by month year etc.. workaround was to use jodaTime, and then extract month/yr/ so on into separate fields w/ RDD conversion.
          Hide
          jayunit100 jay vyas added a comment -

          note that this isnt yet ready for review. Im going to make all the summary stats into a single JSON and also create the HTML page.

          Show
          jayunit100 jay vyas added a comment - note that this isnt yet ready for review. Im going to make all the summary stats into a single JSON and also create the HTML page.
          Hide
          jayunit100 jay vyas added a comment -

          Updated patch. now we get product group frequencies.

          grouped product:zip [59,13,95949]
          grouped product:zip [70,41,04079]
          grouped product:zip [105,47,84015]
          grouped product:zip [57,4,21042]
          grouped product:zip [76,42,32459]
          grouped product:zip [167,14,04079]
          grouped product:zip [43,20,84015]
          grouped product:zip [51,15,32459]
          grouped product:zip [58,36,93311]
          grouped product:zip [140,9,93311]
          grouped product:zip [77,37,53058]
          grouped product:zip [65,56,07039]
          grouped product:zip [136,10,53058]
          grouped product:zip [86,54,95361]
          grouped product:zip [69,56,77583]
          grouped product:zip [87,29,07039]
          grouped product:zip [50,27,95361]
          grouped product:zip [43,51,95949]
          grouped product:zip [70,29,77583]
          grouped product:zip [68,42,21042]
          grouped product:zip [76,2,07039]
          grouped product:zip [174,0,95361]
          grouped product:zip [87,24,95949]
          grouped product:zip [72,2,77583]
          grouped product:zip [97,52,04079]
          grouped product:zip [44,15,21042]
          grouped product:zip [40,25,04079]
          grouped product:zip [133,26,32459]
          grouped product:zip [67,31,84015]
          grouped product:zip [54,4,84015]
          grouped product:zip [126,47,93311]
          grouped product:zip [68,48,53058]
          grouped product:zip [52,20,93311]
          grouped product:zip [61,21,53058]
          grouped product:zip [52,40,07039]
          grouped product:zip [38,13,07039]
          grouped product:zip [54,40,77583]
          grouped product:zip [64,38,95361]
          grouped product:zip [79,11,95361]
          grouped product:zip [136,35,95949]
          grouped product:zip [44,13,77583]
          grouped product:zip [155,26,21042]
          grouped product:zip [148,8,95949]
          grouped product:zip [71,37,32459]
          grouped product:zip [69,42,84015]
          grouped product:zip [49,36,04079]
          grouped product:zip [131,9,04079]
          grouped product:zip [134,10,32459]
          grouped product:zip [56,15,84015]
          grouped product:zip [79,31,93311]
          grouped product:zip [35,32,53058]
          grouped product:zip [46,4,93311]
          grouped product:zip [48,51,07039]
          grouped product:zip [72,5,53058]
          grouped product:zip [43,46,95949]
          grouped product:zip [95,47,04079]
          

          still in intermediate state. just pasting a patch as an update. Its getting close to feature equivalence now w/ bigpetstore-mapreduce, just need to have a json transformation from the query outputs (right now printing the collect() output to screen)>

          Show
          jayunit100 jay vyas added a comment - Updated patch. now we get product group frequencies. grouped product:zip [59,13,95949] grouped product:zip [70,41,04079] grouped product:zip [105,47,84015] grouped product:zip [57,4,21042] grouped product:zip [76,42,32459] grouped product:zip [167,14,04079] grouped product:zip [43,20,84015] grouped product:zip [51,15,32459] grouped product:zip [58,36,93311] grouped product:zip [140,9,93311] grouped product:zip [77,37,53058] grouped product:zip [65,56,07039] grouped product:zip [136,10,53058] grouped product:zip [86,54,95361] grouped product:zip [69,56,77583] grouped product:zip [87,29,07039] grouped product:zip [50,27,95361] grouped product:zip [43,51,95949] grouped product:zip [70,29,77583] grouped product:zip [68,42,21042] grouped product:zip [76,2,07039] grouped product:zip [174,0,95361] grouped product:zip [87,24,95949] grouped product:zip [72,2,77583] grouped product:zip [97,52,04079] grouped product:zip [44,15,21042] grouped product:zip [40,25,04079] grouped product:zip [133,26,32459] grouped product:zip [67,31,84015] grouped product:zip [54,4,84015] grouped product:zip [126,47,93311] grouped product:zip [68,48,53058] grouped product:zip [52,20,93311] grouped product:zip [61,21,53058] grouped product:zip [52,40,07039] grouped product:zip [38,13,07039] grouped product:zip [54,40,77583] grouped product:zip [64,38,95361] grouped product:zip [79,11,95361] grouped product:zip [136,35,95949] grouped product:zip [44,13,77583] grouped product:zip [155,26,21042] grouped product:zip [148,8,95949] grouped product:zip [71,37,32459] grouped product:zip [69,42,84015] grouped product:zip [49,36,04079] grouped product:zip [131,9,04079] grouped product:zip [134,10,32459] grouped product:zip [56,15,84015] grouped product:zip [79,31,93311] grouped product:zip [35,32,53058] grouped product:zip [46,4,93311] grouped product:zip [48,51,07039] grouped product:zip [72,5,53058] grouped product:zip [43,46,95949] grouped product:zip [95,47,04079] still in intermediate state. just pasting a patch as an update. Its getting close to feature equivalence now w/ bigpetstore-mapreduce , just need to have a json transformation from the query outputs (right now printing the collect() output to screen)>
          Hide
          jayunit100 jay vyas added a comment -

          heres official patch for review.

          Show
          jayunit100 jay vyas added a comment - heres official patch for review.
          Hide
          jayunit100 jay vyas added a comment -

          possibly Matt Fenwick might be interested in architecting up a data visualization solution for us.
          I'd like to go with http://trifacta.github.io/vega/ .

          Show
          jayunit100 jay vyas added a comment - possibly Matt Fenwick might be interested in architecting up a data visualization solution for us. I'd like to go with http://trifacta.github.io/vega/ .
          Hide
          rnowling RJ Nowling added a comment -

          jay vyas

          I can't apply the patch (see the errors below). Am I applying it incorrectly? Is it possible that you didn't generate the patch against head?

          [rnowling@miles bigtop_1653]$ git apply BIGTOP-1653.patch
          BIGTOP-1653.patch:470: trailing whitespace.
              
          error: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/analytics/PetStoreStatistics.scala: already exists in working directory
          error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/datamodel/DataModel.scala:28
          error: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/datamodel/DataModel.scala: patch does not apply
          error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/etl/ETL.scala:31
          error: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/etl/ETL.scala: patch does not apply
          error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/TestFullPipeline.scala: already exists in working directory
          BIGTOP-1653.patch:540: new blank line at EOF.
          +
          error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/datamodel/IOUtilsSuite.scala:26
          error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/datamodel/IOUtilsSuite.scala: patch does not apply
          error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/etl/ETLSuite.scala:17
          error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/etl/ETLSuite.scala: patch does not apply
          error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/generator/SparkDriverSuite.scala:17
          error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/generator/SparkDriverSuite.scala: patch does not apply
          
          Show
          rnowling RJ Nowling added a comment - jay vyas I can't apply the patch (see the errors below). Am I applying it incorrectly? Is it possible that you didn't generate the patch against head? [rnowling@miles bigtop_1653]$ git apply BIGTOP-1653.patch BIGTOP-1653.patch:470: trailing whitespace. error: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/analytics/PetStoreStatistics.scala: already exists in working directory error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/datamodel/DataModel.scala:28 error: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/datamodel/DataModel.scala: patch does not apply error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/etl/ETL.scala:31 error: bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/etl/ETL.scala: patch does not apply error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/TestFullPipeline.scala: already exists in working directory BIGTOP-1653.patch:540: new blank line at EOF. + error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/datamodel/IOUtilsSuite.scala:26 error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/datamodel/IOUtilsSuite.scala: patch does not apply error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/etl/ETLSuite.scala:17 error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/etl/ETLSuite.scala: patch does not apply error: patch failed: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/generator/SparkDriverSuite.scala:17 error: bigtop-bigpetstore/bigpetstore-spark/src/test/scala/org/apache/bigpetstore/spark/generator/SparkDriverSuite.scala: patch does not apply
          Hide
          jayunit100 jay vyas added a comment -

          RJ Nowling I'll fix it tonite. I have about 5 commits in this that I re based,
          Probably I have one missing . Thanks for testing!

          Show
          jayunit100 jay vyas added a comment - RJ Nowling I'll fix it tonite. I have about 5 commits in this that I re based, Probably I have one missing . Thanks for testing!
          Hide
          rnowling RJ Nowling added a comment -

          Great! I'm looking forward to a detailed review – this is some really great work! Just a note – the patch noted the creation of multiple files that had been added in BIGTOP-1536 so you might just double check that as part of your investigation.

          Show
          rnowling RJ Nowling added a comment - Great! I'm looking forward to a detailed review – this is some really great work! Just a note – the patch noted the creation of multiple files that had been added in BIGTOP-1536 so you might just double check that as part of your investigation.
          Hide
          jayunit100 jay vyas added a comment -

          yup probably rebased one too many commits. im goingt to just cp and patch onto a fresh copy of bigtop.

          Show
          jayunit100 jay vyas added a comment - yup probably rebased one too many commits. im goingt to just cp and patch onto a fresh copy of bigtop.
          Hide
          jayunit100 jay vyas added a comment -

          yup probably rebased one too many commits. im goingt to just cp and patch onto a fresh copy of bigtop.

          Show
          jayunit100 jay vyas added a comment - yup probably rebased one too many commits. im goingt to just cp and patch onto a fresh copy of bigtop.
          Hide
          jayunit100 jay vyas added a comment -

          okay heres the updated patch. go for it RJ Nowling .

          Show
          jayunit100 jay vyas added a comment - okay heres the updated patch. go for it RJ Nowling .
          Hide
          jayunit100 jay vyas added a comment - - edited

          And, BTW, this is what the data looks like, on output.

          {
             "totalTransaction":12,
             "transactionsByZip":[
            {"count":64,"productId":54,"zipcode":"94583"},{"count":38,"productId":18,"zipcode":"34761"},
          
             {"count":158,"productId":14,"zipcode":"11368"},{"count":66,"productId":46,"zipcode":"33027"},
          
             {"count":52,"productId":27,"zipcode":"94583"},{"count":84,"productId":19,"zipcode":"33027"},
          
             {"count":143,"productId":0,"zipcode":"94583"},{"count":58,"productId":41,"zipcode":"72715"},
          
             {"count":76,"productId":54,"zipcode":"15014"},{"count":118,"productId":52,"zipcode":"45439"}},   
               ..... (several more) .... 
          
             "productDetails":[
                {
                   "productId":0,
                   "category":"kitty litter",
                   "attributes":{
                      "category":"kitty litter",
                      "brand":"Pretty Cat",
                      "size":"7.0",
                      "per_unit_cost":"1.43"
                   }
                },
                {
                   "productId":2,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice"
                   }
                },
                {
                   "productId":4,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Pork"
                   }
                },
                {
                   "productId":6,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Tuna"
                   }
                },
                {
                   "productId":8,
                   "category":"poop bags",
                   "attributes":{
                      "size":"120.0",
                      "color":"Blue",
                      "brand":"Happy Pup",
                      "per_unit_cost":"0.17",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":10,
                   "category":"kitty litter",
                   "attributes":{
                      "category":"kitty litter",
                      "brand":"Pretty Cat",
                      "size":"14.0",
                      "per_unit_cost":"1.43"
                   }
                },
                {
                   "productId":12,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Lamb & Rice"
                   }
                },
                {
                   "productId":14,
                   "category":"poop bags",
                   "attributes":{
                      "size":"60.0",
                      "color":"multicolor",
                      "brand":"Dog Days",
                      "per_unit_cost":"0.21",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":16,
                   "category":"poop bags",
                   "attributes":{
                      "size":"60.0",
                      "color":"Blue",
                      "brand":"Dog Days",
                      "per_unit_cost":"0.21",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":18,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Chicken"
                   }
                },
                {
                   "productId":20,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Fish & Potato"
                   }
                },
                {
                   "productId":22,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"14.0",
                      "brand":"Feisty Feline",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Tuna"
                   }
                },
                {
                   "productId":24,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Feisty Feline",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice"
                   }
                },
                {
                   "productId":26,
                   "category":"poop bags",
                   "attributes":{
                      "size":"120.0",
                      "color":"multicolor",
                      "brand":"Happy Pup",
                      "per_unit_cost":"0.17",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":28,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Pork"
                   }
                },
                {
                   "productId":30,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Chicken"
                   }
                },
                {
                   "productId":32,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"14.0",
                      "brand":"Feisty Feline",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice"
                   }
                },
                {
                   "productId":34,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Feisty Feline",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Tuna"
                   }
                },
                {
                   "productId":36,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Lamb & Rice"
                   }
                },
                {
                   "productId":38,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Lamb & Rice"
                   }
                },
                {
                   "productId":40,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Chicken"
                   }
                },
                {
                   "productId":42,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"14.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice"
                   }
                },
                {
                   "productId":44,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Fish & Potato"
                   }
                },
                {
                   "productId":46,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Pretty Cat",
                      "per_unit_cost":"2.86",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice (Hairball Management)"
                   }
                },
                {
                   "productId":48,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Pork"
                   }
                },
                {
                   "productId":50,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Pretty Cat",
                      "per_unit_cost":"2.86",
                      "category":"dry cat food",
                      "flavor":"Tuna"
                   }
                },
                {
                   "productId":52,
                   "category":"kitty litter",
                   "attributes":{
                      "category":"kitty litter",
                      "brand":"Feisty Feline",
                      "size":"28.0",
                      "per_unit_cost":"1.5"
                   }
                },
                {
                   "productId":54,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"14.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Tuna"
                   }
                },
                {
                   "productId":56,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Fish & Potato"
                   }
                },
                {
                   "productId":1,
                   "category":"poop bags",
                   "attributes":{
                      "size":"120.0",
                      "color":"multicolor",
                      "brand":"Dog Days",
                      "per_unit_cost":"0.21",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":3,
                   "category":"poop bags",
                   "attributes":{
                      "size":"60.0",
                      "color":"multicolor",
                      "brand":"Happy Pup",
                      "per_unit_cost":"0.17",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":5,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Pretty Cat",
                      "per_unit_cost":"2.86",
                      "category":"dry cat food",
                      "flavor":"Tuna"
                   }
                },
                {
                   "productId":7,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Chicken"
                   }
                },
                {
                   "productId":9,
                   "category":"kitty litter",
                   "attributes":{
                      "category":"kitty litter",
                      "brand":"Fiesty Feline",
                      "size":"14.0",
                      "per_unit_cost":"1.5"
                   }
                },
                {
                   "productId":11,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Feisty Feline",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice (Hairball Management)"
                   }
                },
                {
                   "productId":13,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Lamb & Rice"
                   }
                },
                {
                   "productId":15,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Pork"
                   }
                },
                {
                   "productId":17,
                   "category":"poop bags",
                   "attributes":{
                      "size":"60.0",
                      "color":"Blue",
                      "brand":"Happy Pup",
                      "per_unit_cost":"0.17",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":19,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice (Hairball Management)"
                   }
                },
                {
                   "productId":21,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Fish & Potato"
                   }
                },
                {
                   "productId":23,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Chicken"
                   }
                },
                {
                   "productId":25,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Pork"
                   }
                },
                {
                   "productId":27,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Chicken"
                   }
                },
                {
                   "productId":29,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"7.0",
                      "brand":"Pretty Cat",
                      "per_unit_cost":"2.86",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice (Hairball Management)"
                   }
                },
                {
                   "productId":31,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Pretty Cat",
                      "per_unit_cost":"2.86",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice"
                   }
                },
                {
                   "productId":33,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Fish & Potato"
                   }
                },
                {
                   "productId":35,
                   "category":"kitty litter",
                   "attributes":{
                      "category":"kitty litter",
                      "brand":"Fiesty Feline",
                      "size":"7.0",
                      "per_unit_cost":"1.5"
                   }
                },
                {
                   "productId":37,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"14.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice (Hairball Management)"
                   }
                },
                {
                   "productId":39,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Happy Pup",
                      "per_unit_cost":"2.67",
                      "category":"dry dog food",
                      "flavor":"Fish & Potato"
                   }
                },
                {
                   "productId":41,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"15.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Lamb & Rice"
                   }
                },
                {
                   "productId":43,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Wellfed",
                      "per_unit_cost":"2.0",
                      "category":"dry dog food",
                      "flavor":"Pork"
                   }
                },
                {
                   "productId":45,
                   "category":"poop bags",
                   "attributes":{
                      "size":"120.0",
                      "color":"Blue",
                      "brand":"Dog Days",
                      "per_unit_cost":"0.21",
                      "category":"poop bags"
                   }
                },
                {
                   "productId":47,
                   "category":"kitty litter",
                   "attributes":{
                      "category":"kitty litter",
                      "brand":"Pretty Cat",
                      "size":"28.0",
                      "per_unit_cost":"1.43"
                   }
                },
                {
                   "productId":49,
                   "category":"dry cat food",
                   "attributes":{
                      "size":"14.0",
                      "brand":"Feisty Feline",
                      "per_unit_cost":"2.14",
                      "category":"dry cat food",
                      "flavor":"Chicken & Rice (Hairball Management)"
                   }
                },
                {
                   "productId":51,
                   "category":"dry dog food",
                   "attributes":{
                      "size":"30.0",
                      "brand":"Dog Days",
                      "per_unit_cost":"3.0",
                      "category":"dry dog food",
                      "flavor":"Lamb & Rice"
                   }
                }
             ]
          }
          
          Show
          jayunit100 jay vyas added a comment - - edited And, BTW, this is what the data looks like, on output. { "totalTransaction":12, "transactionsByZip":[ {"count":64,"productId":54,"zipcode":"94583"},{"count":38,"productId":18,"zipcode":"34761"}, {"count":158,"productId":14,"zipcode":"11368"},{"count":66,"productId":46,"zipcode":"33027"}, {"count":52,"productId":27,"zipcode":"94583"},{"count":84,"productId":19,"zipcode":"33027"}, {"count":143,"productId":0,"zipcode":"94583"},{"count":58,"productId":41,"zipcode":"72715"}, {"count":76,"productId":54,"zipcode":"15014"},{"count":118,"productId":52,"zipcode":"45439"}}, ..... (several more) .... "productDetails":[ { "productId":0, "category":"kitty litter", "attributes":{ "category":"kitty litter", "brand":"Pretty Cat", "size":"7.0", "per_unit_cost":"1.43" } }, { "productId":2, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Wellfed", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice" } }, { "productId":4, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Pork" } }, { "productId":6, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Wellfed", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Tuna" } }, { "productId":8, "category":"poop bags", "attributes":{ "size":"120.0", "color":"Blue", "brand":"Happy Pup", "per_unit_cost":"0.17", "category":"poop bags" } }, { "productId":10, "category":"kitty litter", "attributes":{ "category":"kitty litter", "brand":"Pretty Cat", "size":"14.0", "per_unit_cost":"1.43" } }, { "productId":12, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Lamb & Rice" } }, { "productId":14, "category":"poop bags", "attributes":{ "size":"60.0", "color":"multicolor", "brand":"Dog Days", "per_unit_cost":"0.21", "category":"poop bags" } }, { "productId":16, "category":"poop bags", "attributes":{ "size":"60.0", "color":"Blue", "brand":"Dog Days", "per_unit_cost":"0.21", "category":"poop bags" } }, { "productId":18, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Chicken" } }, { "productId":20, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Fish & Potato" } }, { "productId":22, "category":"dry cat food", "attributes":{ "size":"14.0", "brand":"Feisty Feline", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Tuna" } }, { "productId":24, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Feisty Feline", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice" } }, { "productId":26, "category":"poop bags", "attributes":{ "size":"120.0", "color":"multicolor", "brand":"Happy Pup", "per_unit_cost":"0.17", "category":"poop bags" } }, { "productId":28, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Pork" } }, { "productId":30, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Chicken" } }, { "productId":32, "category":"dry cat food", "attributes":{ "size":"14.0", "brand":"Feisty Feline", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice" } }, { "productId":34, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Feisty Feline", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Tuna" } }, { "productId":36, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Lamb & Rice" } }, { "productId":38, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Lamb & Rice" } }, { "productId":40, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Chicken" } }, { "productId":42, "category":"dry cat food", "attributes":{ "size":"14.0", "brand":"Wellfed", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice" } }, { "productId":44, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Fish & Potato" } }, { "productId":46, "category":"dry cat food", "attributes":{ "size":"15.0", "brand":"Pretty Cat", "per_unit_cost":"2.86", "category":"dry cat food", "flavor":"Chicken & Rice (Hairball Management)" } }, { "productId":48, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Pork" } }, { "productId":50, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Pretty Cat", "per_unit_cost":"2.86", "category":"dry cat food", "flavor":"Tuna" } }, { "productId":52, "category":"kitty litter", "attributes":{ "category":"kitty litter", "brand":"Feisty Feline", "size":"28.0", "per_unit_cost":"1.5" } }, { "productId":54, "category":"dry cat food", "attributes":{ "size":"14.0", "brand":"Wellfed", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Tuna" } }, { "productId":56, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Fish & Potato" } }, { "productId":1, "category":"poop bags", "attributes":{ "size":"120.0", "color":"multicolor", "brand":"Dog Days", "per_unit_cost":"0.21", "category":"poop bags" } }, { "productId":3, "category":"poop bags", "attributes":{ "size":"60.0", "color":"multicolor", "brand":"Happy Pup", "per_unit_cost":"0.17", "category":"poop bags" } }, { "productId":5, "category":"dry cat food", "attributes":{ "size":"15.0", "brand":"Pretty Cat", "per_unit_cost":"2.86", "category":"dry cat food", "flavor":"Tuna" } }, { "productId":7, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Chicken" } }, { "productId":9, "category":"kitty litter", "attributes":{ "category":"kitty litter", "brand":"Fiesty Feline", "size":"14.0", "per_unit_cost":"1.5" } }, { "productId":11, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Feisty Feline", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice (Hairball Management)" } }, { "productId":13, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Lamb & Rice" } }, { "productId":15, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Pork" } }, { "productId":17, "category":"poop bags", "attributes":{ "size":"60.0", "color":"Blue", "brand":"Happy Pup", "per_unit_cost":"0.17", "category":"poop bags" } }, { "productId":19, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Wellfed", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice (Hairball Management)" } }, { "productId":21, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Fish & Potato" } }, { "productId":23, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Chicken" } }, { "productId":25, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Pork" } }, { "productId":27, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Chicken" } }, { "productId":29, "category":"dry cat food", "attributes":{ "size":"7.0", "brand":"Pretty Cat", "per_unit_cost":"2.86", "category":"dry cat food", "flavor":"Chicken & Rice (Hairball Management)" } }, { "productId":31, "category":"dry cat food", "attributes":{ "size":"15.0", "brand":"Pretty Cat", "per_unit_cost":"2.86", "category":"dry cat food", "flavor":"Chicken & Rice" } }, { "productId":33, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Fish & Potato" } }, { "productId":35, "category":"kitty litter", "attributes":{ "category":"kitty litter", "brand":"Fiesty Feline", "size":"7.0", "per_unit_cost":"1.5" } }, { "productId":37, "category":"dry cat food", "attributes":{ "size":"14.0", "brand":"Wellfed", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice (Hairball Management)" } }, { "productId":39, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Happy Pup", "per_unit_cost":"2.67", "category":"dry dog food", "flavor":"Fish & Potato" } }, { "productId":41, "category":"dry dog food", "attributes":{ "size":"15.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Lamb & Rice" } }, { "productId":43, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Wellfed", "per_unit_cost":"2.0", "category":"dry dog food", "flavor":"Pork" } }, { "productId":45, "category":"poop bags", "attributes":{ "size":"120.0", "color":"Blue", "brand":"Dog Days", "per_unit_cost":"0.21", "category":"poop bags" } }, { "productId":47, "category":"kitty litter", "attributes":{ "category":"kitty litter", "brand":"Pretty Cat", "size":"28.0", "per_unit_cost":"1.43" } }, { "productId":49, "category":"dry cat food", "attributes":{ "size":"14.0", "brand":"Feisty Feline", "per_unit_cost":"2.14", "category":"dry cat food", "flavor":"Chicken & Rice (Hairball Management)" } }, { "productId":51, "category":"dry dog food", "attributes":{ "size":"30.0", "brand":"Dog Days", "per_unit_cost":"3.0", "category":"dry dog food", "flavor":"Lamb & Rice" } } ] }
          Hide
          rnowling RJ Nowling added a comment -

          jay vyas neat work! It's great to have an example for how to do SQL queries in Spark.

          The following needs to be fixed before we can commit the patch:

          1. Change package in PetStoreStatistics.scala from generator to analytics

          2. Remove

          :Any = 

          from the main method of PetStoreStatistics so that it can be executed directly. Related compiler warning:

          /home/rnowling/Projects/bigtop_1653/bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/analytics/PetStoreStatistics.scala:40: PetStoreStatistics has a main method with parameter type Array[String], but org.apache.bigtop.bigpetstore.spark.generator.PetStoreStatistics will not be a runnable program.
            Reason: main method must have exact signature (Array[String])Unit
          object PetStoreStatistics {
          

          3. Change app name from "BPS Data Generator" to something like "PetStoreStatistics" in main method of PetStoreStatistics

          4. Parameters should be checked before creating a SparkContext, otherwise the output is hard to read.

          5. Usage isn't printed when the wrong number of parameters are given.

          6. There are stray semicolons in the source code. Scala doesn't need semicolons so I suggest grepping the files for all instances and deleting them.

          7. Comment on lines 65-69 in PetStoreStatistics seems to be leftover from Generator code.

          8. Move import of sql context on line 74 of PetStoreStatistics to top of file

          9. What are lines 86-91 of PetStoreStatistics? I see comments saying that the lines shouldn't be used except for testing and a commented out line.

          10. Instead of just writing out total transactions, why not write out the number of transactions by month?

          11. Can you add a meaningful variable name for the result of SQL query 2 like you did with SQL query 1 instead of just calling collect() in the return statement?

          12. Please add a section to the README with instructions for running PetStoreStatistics from the CLI.

          Show
          rnowling RJ Nowling added a comment - jay vyas neat work! It's great to have an example for how to do SQL queries in Spark. The following needs to be fixed before we can commit the patch: 1. Change package in PetStoreStatistics.scala from generator to analytics 2. Remove :Any = from the main method of PetStoreStatistics so that it can be executed directly. Related compiler warning: /home/rnowling/Projects/bigtop_1653/bigtop-bigpetstore/bigpetstore-spark/src/main/scala/org/apache/bigpetstore/spark/analytics/PetStoreStatistics.scala:40: PetStoreStatistics has a main method with parameter type Array[String], but org.apache.bigtop.bigpetstore.spark.generator.PetStoreStatistics will not be a runnable program. Reason: main method must have exact signature (Array[String])Unit object PetStoreStatistics { 3. Change app name from "BPS Data Generator" to something like "PetStoreStatistics" in main method of PetStoreStatistics 4. Parameters should be checked before creating a SparkContext, otherwise the output is hard to read. 5. Usage isn't printed when the wrong number of parameters are given. 6. There are stray semicolons in the source code. Scala doesn't need semicolons so I suggest grepping the files for all instances and deleting them. 7. Comment on lines 65-69 in PetStoreStatistics seems to be leftover from Generator code. 8. Move import of sql context on line 74 of PetStoreStatistics to top of file 9. What are lines 86-91 of PetStoreStatistics? I see comments saying that the lines shouldn't be used except for testing and a commented out line. 10. Instead of just writing out total transactions, why not write out the number of transactions by month? 11. Can you add a meaningful variable name for the result of SQL query 2 like you did with SQL query 1 instead of just calling collect() in the return statement? 12. Please add a section to the README with instructions for running PetStoreStatistics from the CLI.
          Hide
          jayunit100 jay vyas added a comment -

          awesome , easy fixes . lemee put em in now

          Show
          jayunit100 jay vyas added a comment - awesome , easy fixes . lemee put em in now
          Hide
          jayunit100 jay vyas added a comment -

          (4,5,6,8,10,11) seem like incremental improvements / stylistic improvements , that can be done in another scope. the other changes are probably best to bundle here. shall we get the core analytics improvements in, and then you can implement those in a separate patch ?

          Show
          jayunit100 jay vyas added a comment - (4,5,6,8,10,11) seem like incremental improvements / stylistic improvements , that can be done in another scope. the other changes are probably best to bundle here. shall we get the core analytics improvements in, and then you can implement those in a separate patch ?
          Hide
          jayunit100 jay vyas added a comment -

          also, appears (8) sqlContext import has to be done after SQLContext is created. i think its an implicits thing. i know its weird but it gives me some errors.

          Show
          jayunit100 jay vyas added a comment - also, appears (8) sqlContext import has to be done after SQLContext is created. i think its an implicits thing. i know its weird but it gives me some errors.
          Hide
          rnowling RJ Nowling added a comment -

          I'm on board with handling 4-6, 8, 10, and 11 in separate JIRA. If you fix 1-3, 7, 9 and 12, I can give it a +1.

          Show
          rnowling RJ Nowling added a comment - I'm on board with handling 4-6, 8, 10, and 11 in separate JIRA. If you fix 1-3, 7, 9 and 12, I can give it a +1.
          Hide
          jayunit100 jay vyas added a comment - - edited

          looks about right now ? (see diff.patch) . If thats good, ill merge it into the commit and you can re-review.

          I put it in a diff for you to make it easier to evaluate.

          Show
          jayunit100 jay vyas added a comment - - edited looks about right now ? (see diff.patch) . If thats good, ill merge it into the commit and you can re-review. I put it in a diff for you to make it easier to evaluate.
          Hide
          jayunit100 jay vyas added a comment -

          here we go. this diff.patch just tested fully against whole pipeline, passes all integration tests etc. if it looks good ill create the final cleaned patch for review.

          Show
          jayunit100 jay vyas added a comment - here we go. this diff.patch just tested fully against whole pipeline, passes all integration tests etc. if it looks good ill create the final cleaned patch for review.
          Hide
          jayunit100 jay vyas added a comment -

          Attached cleaned up patch w/ README updates, fixes to command line error messages, removal of Any (good catch), and removal of single partitioner which would have been very slow (also great catch). Also like you suggested i cleaned up intermediate variable names for sparkSql temp tables.

          Show
          jayunit100 jay vyas added a comment - Attached cleaned up patch w/ README updates, fixes to command line error messages, removal of Any (good catch), and removal of single partitioner which would have been very slow (also great catch). Also like you suggested i cleaned up intermediate variable names for sparkSql temp tables.
          Hide
          rnowling RJ Nowling added a comment -

          +1 to commit. Patch works. Tested integration tests and CLI in docs.

          Show
          rnowling RJ Nowling added a comment - +1 to commit. Patch works. Tested integration tests and CLI in docs.
          Hide
          evans_ye Evans Ye added a comment -

          +1 please commit.

          Show
          evans_ye Evans Ye added a comment - +1 please commit.
          Hide
          jayunit100 jay vyas added a comment -

          pushed, !

          Show
          jayunit100 jay vyas added a comment - pushed, !

            People

            • Assignee:
              jayunit100 jay vyas
              Reporter:
              jayunit100 jay vyas
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development