[MADLIB-1462] DL - Fit multiple does not print timing for validation evaluate

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v1.18.0
• Component/s: Deep Learning
• Assignee: Nikhil Kak
• Reporter: Nikhil Kak
• Created: 17/Dec/20 22:17
• Updated: 22/Jan/21 00:35
• Resolved: 22/Jan/21 00:35

Description

Currently when running fit_multiple with validation dataset, we don't print the timing for the validation runs

select madlib.madlib_keras_fit_multiple_model('cifar10_train_batched', 'cifar10_out', 'cifar10_mst_table', 100, TRUE, 'cifar10_train_batched', 1);
INFO:
 Time for training in iteration 1: 33.6217501163 sec
DETAIL:
 Training set after iteration 1:
 mst_key=12: metric=0.260340005159, loss=2.13081121445
 ...
 mst_key=2: metric=0.164859995246, loss=2.25495767593
 Validation set after iteration 1:
 mst_key=12: metric=0.260340005159, loss=2.13081121445
 ...
 mst_key=2: metric=0.164859995246, loss=2.25495767593
CONTEXT: PL/Python function "madlib_keras_fit_multiple_model"
INFO:
 Time for training in iteration 2: 24.7699511051 sec
DETAIL:
 ....

We should print the time it took to run validation evaluate for both training and validation dataset

 

If the user specifies only the training dataset, then we should add the following to the existing output
1. The cumulative time it took for all the msts to run eval for the training dataset for that iteration

select madlib.madlib_keras_fit_multiple_model('iris_data_packed','iris_multiple_model','mst_table_4row',2, FALSE,NULL,1);

INFO:
 Time for training in iteration 1: 2.24381709099 sec
DETAIL:
 Training set after iteration 1:
 mst_key=2: metric=0.333333343267, loss=1.33550834656
 mst_key=1: metric=0.333333343267, loss=1.12043237686
 mst_key=4: metric=0.333333343267, loss=3.90859818459
 mst_key=3: metric=0.333333343267, loss=4.37875080109
 Time for evaluating training dataset in iteration 1: 0.652065515518
CONTEXT: PL/Python function "madlib_keras_fit_multiple_model"
INFO:
 Time for training in iteration 2: 2.32056617737 sec
DETAIL:
 Training set after iteration 2:
 mst_key=2: metric=0.666666686535, loss=1.14192306995
 mst_key=1: metric=0.666666686535, loss=0.917088747025
 mst_key=4: metric=0.340000003576, loss=2.98958563805
 mst_key=3: metric=0.333333343267, loss=3.86314368248
 Time for evaluating training dataset in iteration 2: 0.679529428482

If the user specifies a validation dataset, then we should add the following to the existing output
1. The cumulative time it took for all the msts to run eval for the training dataset for that iteration
1. The cumulative time it took for all the msts to run eval for the validation dataset for that iteration

select madlib.madlib_keras_fit_multiple_model('iris_data_packed','iris_multiple_model','mst_table_4row',2, FALSE,'iris_data_packed',1);


INFO:
 Time for training in iteration 1: 4.27021813393 sec
DETAIL:
 Training set after iteration 1:
 mst_key=2: metric=0.333333343267, loss=1.39633440971
 mst_key=1: metric=0.333333343267, loss=1.04632723331
 mst_key=4: metric=0.333333343267, loss=3.96611213684
 mst_key=3: metric=0.333333343267, loss=4.38052940369
 Time for evaluating training dataset in iteration 1: 0.649274587631

Validation set after iteration 1:
 mst_key=2: metric=0.333333343267, loss=1.39633440971
 mst_key=1: metric=0.333333343267, loss=1.04632723331
 mst_key=4: metric=0.333333343267, loss=3.96611213684
 mst_key=3: metric=0.333333343267, loss=4.38052940369
 Time for evaluating validation dataset in iteration 1: 0.75797867775
CONTEXT: PL/Python function "madlib_keras_fit_multiple_model"
INFO:
 Time for training in iteration 2: 2.1767308712 sec
DETAIL:
 Training set after iteration 2:
 mst_key=2: metric=0.666666686535, loss=1.10426521301
 mst_key=1: metric=0.666666686535, loss=1.02108848095
 mst_key=4: metric=0.333333343267, loss=3.10222005844
 mst_key=3: metric=0.333333343267, loss=3.85620188713
 Time for evaluating training dataset in iteration 2: 0.784633874893

Validation set after iteration 2:
 mst_key=2: metric=0.666666686535, loss=1.10426521301
 mst_key=1: metric=0.666666686535, loss=1.02108848095
 mst_key=4: metric=0.333333343267, loss=3.10222005844
 mst_key=3: metric=0.333333343267, loss=3.85620188713
 Time for evaluating validation dataset in iteration 2: 0.639858007431