Repo: cleverhans
Filename: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/projects/cleverhans/cleverhans/future/torch/tests/test_attacks.py
ClassName: TestSPSA
Testname: test_adv_example_success_rate_l2
Params: nb_iter,365,19,ParamType.ITER,50
Assertion: self.assertLess(adv_acc, .5)
Original runtime: 5.904999999999999
>>Getting original runtime
Optimization Iteration: 1
Running with params: {'nb_iter': 50}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614539/assert_19752420_50
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 6.055999999999998, Max: 6.670000000000001, Min: 5.75
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.4035119245307959
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 9...
Iter 8...
Iter 10...
Iter 12...
Iter 11...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 6.070833333333332, Max: 6.670000000000001, Min: 5.75
Passed tests : 60
Failed tests : 0
Converged: True
Convergence score: 0.7545973343833211
Evaluating 60 values out of 60
Overall-timings: Avg: 6.070833333333332, Max: 6.670000000000001, Min: 5.75
Choosing distribution: beta
Probability of fail : 0.0
All-Passed: True
Probabilty of failure: 0.0
Runtime: 6.070833333333332
Score: 6.070833333333332
Best-score: 6.070833333333332
Best-param: {'nb_iter': 50}
>>Setting original runtime to 6.070833333333332
Optimization Iteration: 1
Running with params: {'nb_iter': 40}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614539/assert_19752420_40
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 5.2026666666666666, Max: 5.340000000000001, Min: 5.05
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.8344120219478686
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 5.231833333333333, Max: 5.41, Min: 5.05
Passed tests : 60
Failed tests : 0
Converged: False
Convergence score: 1.0476648626996952
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 14...
Iter 15...
Iter 12...
Iter 13...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 29...
Iter 28...
Iter 27...
Timings: Avg: 5.220555555555555, Max: 5.41, Min: 5.05
Passed tests : 90
Failed tests : 0
Converged: True
Convergence score: 0.7868588726146666
Evaluating 90 values out of 90
Overall-timings: Avg: 5.220555555555555, Max: 5.41, Min: 5.05
Choosing distribution: logistic
Probability of fail (non-zero): 7.433993890693102e-05
All-Passed: True
Probabilty of failure: 7.433993890693102e-05
Runtime: 5.220555555555555
Score: 5.220555555555555
Best-score: 5.220555555555555
Best-param: {'nb_iter': 40}
Optimization Iteration: 2
Running with params: {'nb_iter': 30}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614539/assert_19752420_30
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 26...
Iter 25...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 4.316999999999999, Max: 4.4399999999999995, Min: 4.11
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.2188007849009166
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 4.328999999999999, Max: 4.7299999999999995, Min: 4.11
Passed tests : 60
Failed tests : 0
Converged: True
Convergence score: 0.8764051782982841
Evaluating 60 values out of 60
Overall-timings: Avg: 4.328999999999999, Max: 4.7299999999999995, Min: 4.11
Choosing distribution: gumbel_l
Probability of fail : 0.0
All-Passed: True
Probabilty of failure: 0.0
Runtime: 4.328999999999999
Score: 4.328999999999999
Best-score: 4.328999999999999
Best-param: {'nb_iter': 30}
Optimization Iteration: 3
Running with params: {'nb_iter': 30}
Optimization Iteration: 4
Running with params: {'nb_iter': 20}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614539/assert_19752420_20
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 3.475, Max: 3.73, Min: 3.3000000000000003
Passed tests : 21
Failed tests : 9
Converged: False
Convergence score: 1.453092203551647
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 24...
Iter 23...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 3.504833333333333, Max: 3.73, Min: 3.3000000000000003
Passed tests : 35
Failed tests : 25
Converged: False
Convergence score: 1.0739434959940202
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 16...
Iter 15...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 25...
Iter 26...
Iter 27...
Iter 24...
Iter 28...
Iter 29...
Timings: Avg: 3.5057777777777783, Max: 3.73, Min: 3.3000000000000003
Passed tests : 51
Failed tests : 39
Converged: True
Convergence score: 0.9299738348696075
Evaluating 90 values out of 90
Overall-timings: Avg: 3.5057777777777783, Max: 3.73, Min: 3.3000000000000003
Choosing distribution: beta
Probability of fail (non-zero): 0.4070229978049883
Exceeding max probability of failure: 0.4070229978049883
All-Passed: False
Probabilty of failure: 0.4070229978049883
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 4.328999999999999
Best-param: {'nb_iter': 30}
Optimization Iteration: 5
Running with params: {'nb_iter': 40}
Optimization Iteration: 6
Running with params: {'nb_iter': 20}
Optimization Iteration: 7
Running with params: {'nb_iter': 40}
Optimization Iteration: 8
Running with params: {'nb_iter': 30}
Optimization Iteration: 9
Running with params: {'nb_iter': 20}
Optimization Iteration: 10
Running with params: {'nb_iter': 40}
Optimization Iteration: 11
Running with params: {'nb_iter': 30}
Optimization Iteration: 12
Running with params: {'nb_iter': 10}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614539/assert_19752420_10
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 15...
Iter 14...
Iter 17...
Iter 18...
Iter 16...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 2.7466666666666666, Max: 2.9, Min: 2.65
Passed tests : 0
Failed tests : 30
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 4.328999999999999
Best-param: {'nb_iter': 30}
Optimization Iteration: 13
Running with params: {'nb_iter': 20}
Optimization Iteration: 14
Running with params: {'nb_iter': 30}
Optimization Iteration: 15
Running with params: {'nb_iter': 10}
Optimization Iteration: 16
Running with params: {'nb_iter': 40}
Optimization Iteration: 17
Running with params: {'nb_iter': 10}
Optimization Iteration: 18
Running with params: {'nb_iter': 20}
Optimization Iteration: 19
Running with params: {'nb_iter': 10}
Optimization Iteration: 20
Running with params: {'nb_iter': 30}
Optimization Iteration: 21
Running with params: {'nb_iter': 50}
Optimization Iteration: 22
Running with params: {'nb_iter': 30}
Optimization Iteration: 23
Running with params: {'nb_iter': 30}
Optimization Iteration: 24
Running with params: {'nb_iter': 20}
Optimization Iteration: 25
Running with params: {'nb_iter': 30}
Optimization Iteration: 26
Running with params: {'nb_iter': 50}
Optimization Iteration: 27
Running with params: {'nb_iter': 50}
Optimization Iteration: 28
Running with params: {'nb_iter': 20}
Optimization Iteration: 29
Running with params: {'nb_iter': 30}
Optimization Iteration: 30
Running with params: {'nb_iter': 40}
Optimization Iteration: 31
Running with params: {'nb_iter': 20}
Optimization Iteration: 32
Running with params: {'nb_iter': 40}
Optimization Iteration: 33
Running with params: {'nb_iter': 20}
Optimization Iteration: 34
Running with params: {'nb_iter': 30}
Optimization Iteration: 35
Running with params: {'nb_iter': 30}
Optimization Iteration: 36
Running with params: {'nb_iter': 20}
Optimization Iteration: 37
Running with params: {'nb_iter': 30}
Optimization Iteration: 38
Running with params: {'nb_iter': 40}
Optimization Iteration: 39
Running with params: {'nb_iter': 40}
Optimization Iteration: 40
Running with params: {'nb_iter': 10}
Optimization Iteration: 41
Running with params: {'nb_iter': 20}
Optimization Iteration: 42
Running with params: {'nb_iter': 30}
Optimization Iteration: 43
Running with params: {'nb_iter': 30}
Optimization Iteration: 44
Running with params: {'nb_iter': 50}
Optimization Iteration: 45
Running with params: {'nb_iter': 10}
Optimization Iteration: 46
Running with params: {'nb_iter': 20}
Optimization Iteration: 47
Running with params: {'nb_iter': 30}
Optimization Iteration: 48
Running with params: {'nb_iter': 40}
Optimization Iteration: 49
Running with params: {'nb_iter': 20}
Optimization Iteration: 50
Running with params: {'nb_iter': 30}
Upto iteration 50: {'nb_iter': 30.0}
Optimization Iteration: 51
Running with params: {'nb_iter': 40}
Optimization Iteration: 52
Running with params: {'nb_iter': 10}
Optimization Iteration: 53
Running with params: {'nb_iter': 20}
Optimization Iteration: 54
Running with params: {'nb_iter': 30}
Optimization Iteration: 55
Running with params: {'nb_iter': 50}
Optimization Iteration: 56
Running with params: {'nb_iter': 20}
Optimization Iteration: 57
Running with params: {'nb_iter': 10}
Optimization Iteration: 58
Running with params: {'nb_iter': 40}
Optimization Iteration: 59
Running with params: {'nb_iter': 30}
Optimization Iteration: 60
Running with params: {'nb_iter': 20}
Optimization Iteration: 61
Running with params: {'nb_iter': 50}
Optimization Iteration: 62
Running with params: {'nb_iter': 30}
Optimization Iteration: 63
Running with params: {'nb_iter': 10}
Optimization Iteration: 64
Running with params: {'nb_iter': 20}
Optimization Iteration: 65
Running with params: {'nb_iter': 30}
Optimization Iteration: 66
Running with params: {'nb_iter': 30}
Optimization Iteration: 67
Running with params: {'nb_iter': 30}
Optimization Iteration: 68
Running with params: {'nb_iter': 30}
Optimization Iteration: 69
Running with params: {'nb_iter': 40}
Optimization Iteration: 70
Running with params: {'nb_iter': 40}
Optimization Iteration: 71
Running with params: {'nb_iter': 30}
Optimization Iteration: 72
Running with params: {'nb_iter': 30}
Optimization Iteration: 73
Running with params: {'nb_iter': 40}
Optimization Iteration: 74
Running with params: {'nb_iter': 20}
Optimization Iteration: 75
Running with params: {'nb_iter': 30}
Optimization Iteration: 76
Running with params: {'nb_iter': 30}
Optimization Iteration: 77
Running with params: {'nb_iter': 20}
Optimization Iteration: 78
Running with params: {'nb_iter': 50}
Optimization Iteration: 79
Running with params: {'nb_iter': 30}
Optimization Iteration: 80
Running with params: {'nb_iter': 40}
Optimization Iteration: 81
Running with params: {'nb_iter': 20}
Optimization Iteration: 82
Running with params: {'nb_iter': 30}
Optimization Iteration: 83
Running with params: {'nb_iter': 20}
Optimization Iteration: 84
Running with params: {'nb_iter': 40}
Optimization Iteration: 85
Running with params: {'nb_iter': 30}
Optimization Iteration: 86
Running with params: {'nb_iter': 20}
Optimization Iteration: 87
Running with params: {'nb_iter': 30}
Optimization Iteration: 88
Running with params: {'nb_iter': 30}
Optimization Iteration: 89
Running with params: {'nb_iter': 20}
Optimization Iteration: 90
Running with params: {'nb_iter': 40}
Optimization Iteration: 91
Running with params: {'nb_iter': 40}
Optimization Iteration: 92
Running with params: {'nb_iter': 20}
Optimization Iteration: 93
Running with params: {'nb_iter': 30}
Optimization Iteration: 94
Running with params: {'nb_iter': 50}
Optimization Iteration: 95
Running with params: {'nb_iter': 40}
Optimization Iteration: 96
Running with params: {'nb_iter': 20}
Optimization Iteration: 97
Running with params: {'nb_iter': 30}
Optimization Iteration: 98
Running with params: {'nb_iter': 30}
Optimization Iteration: 99
Running with params: {'nb_iter': 50}
Optimization Iteration: 100
Running with params: {'nb_iter': 10}
Upto iteration 100: {'nb_iter': 30.0}
Breaking...
{'nb_iter': 30.0}
Best score: 4.328999999999999
Repeated params: 96
Trials: 101
Best param {'nb_iter': 30.0}
Reduction: 28.69183253260124%
Speedup: 1.4023639023639025x
Optimizer time: 56.31655931472778
