Repo: parlai
Filename: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/projects/parlai/tests/test_transformers.py
ClassName: TestTransformerRanker
Testname: test_repeater
Params: learning_rate,35,29,ParamType.LR,0.007
batchsize,36,26,ParamType.CHAINS,16
Assertion: self.assertGreaterEqual(test['hits@1'], 0.90)
Original runtime: 14.317
>>Getting original runtime
Optimization Iteration: 1
Running with params: {'learning_rate': 0.007, 'batchsize': 16}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_0.007_16
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 10.680666666666669, Max: 15.26, Min: 7.6000000000000005
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.9999999999999942
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 10.220500000000003, Max: 15.26, Min: 7.5200000000000005
Passed tests : 60
Failed tests : 0
Converged: True
Convergence score: 0.6226461114121385
Evaluating 60 values out of 60
Overall-timings: Avg: 10.220500000000003, Max: 15.26, Min: 7.5200000000000005
Choosing distribution: gumbel_l
Probability of fail (non-zero): 4.0290371724898187e-07
All-Passed: True
Probabilty of failure: 4.0290371724898187e-07
Runtime: 10.220500000000003
Score: 10.220500000000003
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
>>Setting original runtime to 10.220500000000003
Optimization Iteration: 1
Running with params: {'batchsize': 64, 'learning_rate': 0.0038053042078382885}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0038053042078382885
Optimization Iteration: 2
Running with params: {'batchsize': 256, 'learning_rate': 0.00042559078304402296}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.00042559078304402296
Optimization Iteration: 3
Running with params: {'batchsize': 16, 'learning_rate': 5.364287629794711e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 5.364287629794711e-05
Optimization Iteration: 4
Running with params: {'batchsize': 8, 'learning_rate': 0.00016939442229211082}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.00016939442229211082
Optimization Iteration: 5
Running with params: {'batchsize': 2, 'learning_rate': 0.003716002364858732}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.003716002364858732
Optimization Iteration: 6
Running with params: {'batchsize': 64, 'learning_rate': 0.0026854689417369114}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0026854689417369114
Optimization Iteration: 7
Running with params: {'batchsize': 256, 'learning_rate': 0.0048982426450085385}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0048982426450085385
Optimization Iteration: 8
Running with params: {'batchsize': 32, 'learning_rate': 0.19656478497768018}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_32_0.19656478497768018
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Timings: Avg: 40.02733333333333, Max: 56.88, Min: 26.38
Passed tests : 0
Failed tests : 30
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 9
Running with params: {'batchsize': 128, 'learning_rate': 0.8103686111643624}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_128_0.8103686111643624
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 20...
Iter 19...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Timings: Avg: 28.27766666666667, Max: 31.599999999999998, Min: 23.88
Passed tests : 0
Failed tests : 30
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 10
Running with params: {'batchsize': 64, 'learning_rate': 0.5387634110628335}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_64_0.5387634110628335
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 28...
Iter 27...
Iter 29...
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Timings: Avg: 27.18566666666667, Max: 33.38, Min: 22.8
Passed tests : 0
Failed tests : 30
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 11
Running with params: {'batchsize': 32, 'learning_rate': 0.7271633464272925}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_32_0.7271633464272925
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 6...
Iter 5...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 29...
Iter 28...
Iter 27...
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Not found ::: 
Timings: Avg: 33.42433333333334, Max: 39.92, Min: 28.060000000000002
Passed tests : 0
Failed tests : 30
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 12
Running with params: {'batchsize': 8, 'learning_rate': 0.007381451183531364}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_8_0.007381451183531364
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 17.647333333333336, Max: 27.630000000000003, Min: 9.66
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.538727687130385
Exceeding current best timeout: 17.647333333333336 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 13
Running with params: {'batchsize': 256, 'learning_rate': 1.0341578005327797e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 1.0341578005327797e-05
Optimization Iteration: 14
Running with params: {'batchsize': 8, 'learning_rate': 7.21473297991731e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 7.21473297991731e-05
Optimization Iteration: 15
Running with params: {'batchsize': 32, 'learning_rate': 0.00018233815633267395}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.00018233815633267395
Optimization Iteration: 16
Running with params: {'batchsize': 64, 'learning_rate': 0.00382161211546053}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.00382161211546053
Optimization Iteration: 17
Running with params: {'batchsize': 64, 'learning_rate': 0.05752430961442976}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_64_0.05752430961442976
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 19...
Iter 18...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 18.526333333333334, Max: 26.099999999999998, Min: 9.719999999999999
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.2360679774997734
Exceeding current best timeout: 18.526333333333334 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 18
Running with params: {'batchsize': 2, 'learning_rate': 0.030349399182205593}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_2_0.030349399182205593
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 17.263666666666666, Max: 28.26, Min: 9.77
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.3608276348795343
Exceeding current best timeout: 17.263666666666666 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 19
Running with params: {'batchsize': 2, 'learning_rate': 0.002425918951755371}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.002425918951755371
Optimization Iteration: 20
Running with params: {'batchsize': 256, 'learning_rate': 4.201507884026965e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 4.201507884026965e-05
Optimization Iteration: 21
Running with params: {'batchsize': 2, 'learning_rate': 0.03589770329040312}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_2_0.03589770329040312
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 14.96466666666667, Max: 23.310000000000002, Min: 9.520000000000001
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.178511301977574
Exceeding current best timeout: 14.96466666666667 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 22
Running with params: {'batchsize': 2, 'learning_rate': 0.0009515785033614905}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0009515785033614905
Optimization Iteration: 23
Running with params: {'batchsize': 256, 'learning_rate': 0.017775187194179753}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_256_0.017775187194179753
Launching 30 jobs, 30 in parallel
Iter 1...
Iter 0...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 28...
Iter 29...
Iter 27...
Timings: Avg: 16.699333333333335, Max: 25.82, Min: 9.600000000000001
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.084435336938071
Exceeding current best timeout: 16.699333333333335 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 24
Running with params: {'batchsize': 16, 'learning_rate': 0.11779086103913758}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_16_0.11779086103913758
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 17.706, Max: 26.259999999999998, Min: 9.64
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.568929081105481
Exceeding current best timeout: 17.706 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 25
Running with params: {'batchsize': 128, 'learning_rate': 0.0007997839243775609}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0007997839243775609
Optimization Iteration: 26
Running with params: {'batchsize': 256, 'learning_rate': 0.010473171593944296}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_256_0.010473171593944296
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 18...
Iter 16...
Iter 19...
Iter 17...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Iter 20...
Timings: Avg: 15.694333333333331, Max: 21.17, Min: 9.52
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.942571724714527
Exceeding current best timeout: 15.694333333333331 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 27
Running with params: {'batchsize': 16, 'learning_rate': 0.2246474951830929}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_16_0.2246474951830929
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 11...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 25...
Iter 26...
Iter 24...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 15.929333333333338, Max: 24.169999999999998, Min: 9.61
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.8973665961010275
Exceeding current best timeout: 15.929333333333338 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 28
Running with params: {'batchsize': 128, 'learning_rate': 0.0009643982638282253}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0009643982638282253
Optimization Iteration: 29
Running with params: {'batchsize': 64, 'learning_rate': 0.015945246150321016}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_64_0.015945246150321016
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 23...
Iter 22...
Iter 24...
Iter 25...
Iter 26...
Iter 29...
Iter 28...
Iter 27...
Timings: Avg: 16.881, Max: 26.39, Min: 9.540000000000001
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.9448482384566073
Exceeding current best timeout: 16.881 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 30
Running with params: {'batchsize': 16, 'learning_rate': 0.31735755891806455}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_16_0.31735755891806455
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 17.406333333333333, Max: 28.25, Min: 9.53
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.8380365552345193
Exceeding current best timeout: 17.406333333333333 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 31
Running with params: {'batchsize': 128, 'learning_rate': 0.0009312583607408095}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0009312583607408095
Optimization Iteration: 32
Running with params: {'batchsize': 64, 'learning_rate': 0.0003621198831789243}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0003621198831789243
Optimization Iteration: 33
Running with params: {'batchsize': 16, 'learning_rate': 1.1455995349277229e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 1.1455995349277229e-05
Optimization Iteration: 34
Running with params: {'batchsize': 128, 'learning_rate': 0.001505582142777588}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.001505582142777588
Optimization Iteration: 35
Running with params: {'batchsize': 64, 'learning_rate': 0.00027076100356116905}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.00027076100356116905
Optimization Iteration: 36
Running with params: {'batchsize': 256, 'learning_rate': 2.957140559295787e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 2.957140559295787e-05
Optimization Iteration: 37
Running with params: {'batchsize': 2, 'learning_rate': 0.06757433052224515}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_2_0.06757433052224515
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 15...
Iter 16...
Iter 14...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 17.103666666666665, Max: 21.44, Min: 9.6
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.366431913239849
Exceeding current best timeout: 17.103666666666665 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 38
Running with params: {'batchsize': 2, 'learning_rate': 0.02813078038367177}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_2_0.02813078038367177
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 17.605333333333338, Max: 25.22, Min: 9.639999999999999
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.1641710001743946
Exceeding current best timeout: 17.605333333333338 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 39
Running with params: {'batchsize': 2, 'learning_rate': 9.727649624980515e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 9.727649624980515e-05
Optimization Iteration: 40
Running with params: {'batchsize': 8, 'learning_rate': 0.0021886916725221525}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0021886916725221525
Optimization Iteration: 41
Running with params: {'batchsize': 256, 'learning_rate': 0.013374701604608824}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_256_0.013374701604608824
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 16.387, Max: 24.220000000000002, Min: 9.6
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.5855725062711246
Exceeding current best timeout: 16.387 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 42
Running with params: {'batchsize': 256, 'learning_rate': 0.006260272796924018}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.006260272796924018
Optimization Iteration: 43
Running with params: {'batchsize': 16, 'learning_rate': 0.08037407778845848}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_16_0.08037407778845848
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 25...
Iter 24...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 16.298, Max: 25.72, Min: 9.67
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.0000000000000002
Exceeding current best timeout: 16.298 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 44
Running with params: {'batchsize': 16, 'learning_rate': 0.11741008316449439}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_16_0.11741008316449439
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 23...
Iter 22...
Iter 24...
Iter 25...
Iter 26...
Iter 28...
Iter 27...
Iter 29...
Timings: Avg: 17.31233333333334, Max: 26.57, Min: 9.71
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.23606797749979
Exceeding current best timeout: 17.31233333333334 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 45
Running with params: {'batchsize': 128, 'learning_rate': 0.0005710404733676672}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.0005710404733676672
Optimization Iteration: 46
Running with params: {'batchsize': 32, 'learning_rate': 1.8053991387738394e-05}
Lower learning rate.. returning... Best: 0.007, Proposed: 1.8053991387738394e-05
Optimization Iteration: 47
Running with params: {'batchsize': 64, 'learning_rate': 0.00011849500262525265}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.00011849500262525265
Optimization Iteration: 48
Running with params: {'batchsize': 8, 'learning_rate': 0.005286719004451675}
Lower learning rate.. returning... Best: 0.007, Proposed: 0.005286719004451675
Optimization Iteration: 49
Running with params: {'batchsize': 16, 'learning_rate': 0.3167682629355798}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_16_0.3167682629355798
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 17.06566666666667, Max: 25.919999999999998, Min: 9.809999999999999
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.0542032715824328
Exceeding current best timeout: 17.06566666666667 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Optimization Iteration: 50
Running with params: {'batchsize': 32, 'learning_rate': 0.16941003646730596}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/logs/optim_1597619505_parlai/run_1597619505/assert_38468249_32_0.16941003646730596
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 16.083666666666666, Max: 31.0, Min: 9.49
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.717911380774684
Exceeding current best timeout: 16.083666666666666 > 10.220500000000003
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 10.220500000000003
Best-param: {'learning_rate': 0.007, 'batchsize': 16}
Exception>>>
Traceback (most recent call last):
  File "optimizer.py", line 65, in run_optimizer
    result, trialsobj = bayesopt.sample()
  File "/mnt/batch/tasks/workitems/opt1/job-1/Task_parlai/wd/borntobeflaky/tool/src/strategy/bayesopt.py", line 143, in sample
    verbose=True, timeout=6000)
  File "/usr/local/lib/python3.6/dist-packages/hyperopt/fmin.py", line 482, in fmin
    show_progressbar=show_progressbar,
  File "/usr/local/lib/python3.6/dist-packages/hyperopt/base.py", line 686, in fmin
    show_progressbar=show_progressbar,
  File "/usr/local/lib/python3.6/dist-packages/hyperopt/fmin.py", line 516, in fmin
    return trials.argmin
  File "/usr/local/lib/python3.6/dist-packages/hyperopt/base.py", line 622, in argmin
    best_trial = self.best_trial
  File "/usr/local/lib/python3.6/dist-packages/hyperopt/base.py", line 613, in best_trial
    raise AllTrialsFailed
hyperopt.exceptions.AllTrialsFailed

Optimizer time: 634.3384127616882
