Repo: cleverhans
Filename: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/projects/cleverhans/cleverhans/future/torch/tests/test_attacks.py
ClassName: TestSPSA
Testname: test_adv_example_success_rate_linf
Params: nb_iter,365,19,ParamType.ITER,50
Assertion: self.assertLess(adv_acc, .5)
Original runtime: 5.857
>>Getting original runtime
Optimization Iteration: 1
Running with params: {'nb_iter': 50}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614712/assert_19482812_50
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 3...
Iter 4...
Iter 2...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 23...
Iter 21...
Iter 24...
Iter 22...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 5.6403333333333325, Max: 6.0600000000000005, Min: 5.319999999999999
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.122790929762998
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 5.663666666666667, Max: 6.13, Min: 5.319999999999999
Passed tests : 60
Failed tests : 0
Converged: False
Convergence score: 1.1529849127953868
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 5.663888888888889, Max: 6.13, Min: 5.319999999999999
Passed tests : 90
Failed tests : 0
Converged: True
Convergence score: 0.6738508977077393
Evaluating 90 values out of 90
Overall-timings: Avg: 5.663888888888889, Max: 6.13, Min: 5.319999999999999
Choosing distribution: logistic
Probability of fail (non-zero): 1.3748711058525487e-09
All-Passed: True
Probabilty of failure: 1.3748711058525487e-09
Runtime: 5.663888888888889
Score: 5.663888888888889
Best-score: 5.663888888888889
Best-param: {'nb_iter': 50}
>>Setting original runtime to 5.663888888888889
Optimization Iteration: 1
Running with params: {'nb_iter': 30}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614712/assert_19482812_30
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 4.075666666666667, Max: 4.260000000000001, Min: 3.8600000000000003
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.927674999088624
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 4.061333333333334, Max: 4.260000000000001, Min: 3.8600000000000003
Passed tests : 60
Failed tests : 0
Converged: False
Convergence score: 1.346530716294432
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 4.067, Max: 4.260000000000001, Min: 3.83
Passed tests : 90
Failed tests : 0
Converged: False
Convergence score: 1.0004730562141158
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 26...
Iter 25...
Iter 28...
Iter 29...
Iter 27...
Timings: Avg: 4.063083333333334, Max: 4.35, Min: 3.83
Passed tests : 120
Failed tests : 0
Converged: True
Convergence score: 0.8874248707880544
Evaluating 120 values out of 120
Overall-timings: Avg: 4.063083333333334, Max: 4.35, Min: 3.83
Choosing distribution: beta
Probability of fail (non-zero): 7.148578851268184e-05
All-Passed: True
Probabilty of failure: 7.148578851268184e-05
Runtime: 4.063083333333334
Score: 4.063083333333334
Best-score: 4.063083333333334
Best-param: {'nb_iter': 30}
Optimization Iteration: 2
Running with params: {'nb_iter': 10}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614712/assert_19482812_10
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 28...
Iter 29...
Iter 25...
Iter 27...
Iter 26...
Timings: Avg: 2.6503333333333337, Max: 2.78, Min: 2.4899999999999998
Passed tests : 0
Failed tests : 30
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 4.063083333333334
Best-param: {'nb_iter': 30}
Optimization Iteration: 3
Running with params: {'nb_iter': 30}
Optimization Iteration: 4
Running with params: {'nb_iter': 50}
Optimization Iteration: 5
Running with params: {'nb_iter': 10}
Optimization Iteration: 6
Running with params: {'nb_iter': 20}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614712/assert_19482812_20
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 3.324333333333334, Max: 3.41, Min: 3.1799999999999997
Passed tests : 16
Failed tests : 14
Converged: False
Convergence score: 2.230619365854195
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 3.318666666666667, Max: 3.42, Min: 3.0900000000000003
Passed tests : 33
Failed tests : 27
Converged: True
Convergence score: 0.9759418573238514
Evaluating 60 values out of 60
Overall-timings: Avg: 3.318666666666667, Max: 3.42, Min: 3.0900000000000003
Choosing distribution: logistic
Probability of fail (non-zero): 0.3707914309926821
Exceeding max probability of failure: 0.3707914309926821
All-Passed: False
Probabilty of failure: 0.3707914309926821
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 4.063083333333334
Best-param: {'nb_iter': 30}
Optimization Iteration: 7
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 8
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 9
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 10
Running with params: {'nb_iter': 20}
Optimization Iteration: 11
Running with params: {'nb_iter': 20}
Optimization Iteration: 12
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 13
Running with params: {'nb_iter': 10}
Optimization Iteration: 14
Running with params: {'nb_iter': 20}
Optimization Iteration: 15
Running with params: {'nb_iter': 20}
Optimization Iteration: 16
Running with params: {'nb_iter': 20}
Optimization Iteration: 17
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 18
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 19
Running with params: {'nb_iter': 30}
Optimization Iteration: 20
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 21
Running with params: {'nb_iter': 30}
Optimization Iteration: 22
Running with params: {'nb_iter': 30}
Optimization Iteration: 23
Running with params: {'nb_iter': 30}
Optimization Iteration: 24
Running with params: {'nb_iter': 30}
Optimization Iteration: 25
Running with params: {'nb_iter': 50}
Optimization Iteration: 26
Running with params: {'nb_iter': 30}
Optimization Iteration: 27
Running with params: {'nb_iter': 30}
Optimization Iteration: 28
Running with params: {'nb_iter': 50}
Optimization Iteration: 29
Running with params: {'nb_iter': 10}
Optimization Iteration: 30
Running with params: {'nb_iter': 20}
Optimization Iteration: 31
Running with params: {'nb_iter': 10}
Optimization Iteration: 32
Running with params: {'nb_iter': 30}
Optimization Iteration: 33
Running with params: {'nb_iter': 30}
Optimization Iteration: 34
Running with params: {'nb_iter': 50}
Optimization Iteration: 35
Running with params: {'nb_iter': 10}
Optimization Iteration: 36
Running with params: {'nb_iter': 20}
Optimization Iteration: 37
Running with params: {'nb_iter': 30}
Optimization Iteration: 38
Running with params: {'nb_iter': 20}
Optimization Iteration: 39
Running with params: {'nb_iter': 20}
Optimization Iteration: 40
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 41
Running with params: {'nb_iter': 30}
Optimization Iteration: 42
Running with params: {'nb_iter': 50}
Optimization Iteration: 43
Running with params: {'nb_iter': 10}
Optimization Iteration: 44
Running with params: {'nb_iter': 20}
Optimization Iteration: 45
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 46
Running with params: {'nb_iter': 20}
Optimization Iteration: 47
Running with params: {'nb_iter': 30}
Optimization Iteration: 48
Running with params: {'nb_iter': 20}
Optimization Iteration: 49
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 50
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Upto iteration 50: {'nb_iter': 30.0}
Optimization Iteration: 51
Running with params: {'nb_iter': 30}
Optimization Iteration: 52
Running with params: {'nb_iter': 20}
Optimization Iteration: 53
Running with params: {'nb_iter': 30}
Optimization Iteration: 54
Running with params: {'nb_iter': 10}
Optimization Iteration: 55
Running with params: {'nb_iter': 50}
Optimization Iteration: 56
Running with params: {'nb_iter': 20}
Optimization Iteration: 57
Running with params: {'nb_iter': 30}
Optimization Iteration: 58
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 59
Running with params: {'nb_iter': 30}
Optimization Iteration: 60
Running with params: {'nb_iter': 10}
Optimization Iteration: 61
Running with params: {'nb_iter': 20}
Optimization Iteration: 62
Running with params: {'nb_iter': 30}
Optimization Iteration: 63
Running with params: {'nb_iter': 50}
Optimization Iteration: 64
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 65
Running with params: {'nb_iter': 20}
Optimization Iteration: 66
Running with params: {'nb_iter': 30}
Optimization Iteration: 67
Running with params: {'nb_iter': 30}
Optimization Iteration: 68
Running with params: {'nb_iter': 30}
Optimization Iteration: 69
Running with params: {'nb_iter': 30}
Optimization Iteration: 70
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 71
Running with params: {'nb_iter': 30}
Optimization Iteration: 72
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 73
Running with params: {'nb_iter': 20}
Optimization Iteration: 74
Running with params: {'nb_iter': 30}
Optimization Iteration: 75
Running with params: {'nb_iter': 20}
Optimization Iteration: 76
Running with params: {'nb_iter': 30}
Optimization Iteration: 77
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 78
Running with params: {'nb_iter': 30}
Optimization Iteration: 79
Running with params: {'nb_iter': 20}
Optimization Iteration: 80
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 81
Running with params: {'nb_iter': 30}
Optimization Iteration: 82
Running with params: {'nb_iter': 50}
Optimization Iteration: 83
Running with params: {'nb_iter': 20}
Optimization Iteration: 84
Running with params: {'nb_iter': 30}
Optimization Iteration: 85
Running with params: {'nb_iter': 20}
Optimization Iteration: 86
Running with params: {'nb_iter': 20}
Optimization Iteration: 87
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 88
Running with params: {'nb_iter': 30}
Optimization Iteration: 89
Running with params: {'nb_iter': 50}
Optimization Iteration: 90
Running with params: {'nb_iter': 30}
Optimization Iteration: 91
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 92
Running with params: {'nb_iter': 10}
Optimization Iteration: 93
Running with params: {'nb_iter': 30}
Optimization Iteration: 94
Running with params: {'nb_iter': 20}
Optimization Iteration: 95
Running with params: {'nb_iter': 30}
Optimization Iteration: 96
Running with params: {'nb_iter': 30}
Optimization Iteration: 97
Running with params: {'nb_iter': 10}
Optimization Iteration: 98
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 99
Running with params: {'nb_iter': 20}
Optimization Iteration: 100
Running with params: {'nb_iter': 50}
Upto iteration 100: {'nb_iter': 30.0}
Breaking...
{'nb_iter': 30.0}
Best score: 4.063083333333334
Repeated params: 77
Trials: 101
Best param {'nb_iter': 30.0}
Reduction: 28.263364394310923%
Speedup: 1.3939878718269512x
Optimizer time: 48.14083957672119
