Repo: cleverhans
Filename: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/projects/cleverhans/cleverhans/future/torch/tests/test_attacks.py
ClassName: TestSPSA
Testname: test_targeted_adv_example_success_rate_linf
Params: nb_iter,365,19,ParamType.ITER,50
Assertion: self.assertGreater(adv_success, .7)
Original runtime: 5.515000000000001
>>Getting original runtime
Optimization Iteration: 1
Running with params: {'nb_iter': 50}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614672/assert_24568584_50
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 5.611, Max: 5.8100000000000005, Min: 5.430000000000001
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.0485376614107658
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 5.669, Max: 6.180000000000001, Min: 5.430000000000001
Passed tests : 60
Failed tests : 0
Converged: True
Convergence score: 0.6984302957695753
Evaluating 60 values out of 60
Overall-timings: Avg: 5.669, Max: 6.180000000000001, Min: 5.430000000000001
Choosing distribution: gumbel_l
Probability of fail (non-zero): 2.787161758361853e-06
All-Passed: True
Probabilty of failure: 2.787161758361853e-06
Runtime: 5.669
Score: 5.669
Best-score: 5.669
Best-param: {'nb_iter': 50}
>>Setting original runtime to 5.669
Optimization Iteration: 1
Running with params: {'nb_iter': 10}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614672/assert_24568584_10
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 2.6086666666666667, Max: 2.67, Min: 2.34
Passed tests : 2
Failed tests : 28
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 5.669
Best-param: {'nb_iter': 50}
Optimization Iteration: 2
Running with params: {'nb_iter': 10}
Optimization Iteration: 3
Running with params: {'nb_iter': 30}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614672/assert_24568584_30
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 5...
Iter 6...
Iter 4...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 27...
Iter 26...
Iter 28...
Iter 29...
Timings: Avg: 4.064000000000001, Max: 4.28, Min: 3.9
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 2.8490144114909532
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 4.078333333333333, Max: 4.28, Min: 3.83
Passed tests : 60
Failed tests : 0
Converged: True
Convergence score: 0.9415717588421928
Evaluating 60 values out of 60
Overall-timings: Avg: 4.078333333333333, Max: 4.28, Min: 3.83
Choosing distribution: norm
Probability of fail (non-zero): 0.0009255236607067241
All-Passed: True
Probabilty of failure: 0.0009255236607067241
Runtime: 4.078333333333333
Score: 4.078333333333333
Best-score: 4.078333333333333
Best-param: {'nb_iter': 30}
Optimization Iteration: 4
Running with params: {'nb_iter': 20}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614672/assert_24568584_20
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 3.328, Max: 3.4699999999999998, Min: 3.12
Passed tests : 21
Failed tests : 9
Converged: False
Convergence score: 4.985272427507907
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 3.3350000000000004, Max: 3.4699999999999998, Min: 3.12
Passed tests : 40
Failed tests : 20
Converged: True
Convergence score: 0.7999204764958227
Evaluating 60 values out of 60
Overall-timings: Avg: 3.3350000000000004, Max: 3.4699999999999998, Min: 3.12
Choosing distribution: dweibull
Probability of fail (non-zero): 0.21886365637761365
Exceeding max probability of failure: 0.21886365637761365
All-Passed: False
Probabilty of failure: 0.21886365637761365
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 4.078333333333333
Best-param: {'nb_iter': 30}
Optimization Iteration: 5
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 6
Running with params: {'nb_iter': 30}
Optimization Iteration: 7
Running with params: {'nb_iter': 20}
Optimization Iteration: 8
Running with params: {'nb_iter': 10}
Optimization Iteration: 9
Running with params: {'nb_iter': 30}
Optimization Iteration: 10
Running with params: {'nb_iter': 10}
Optimization Iteration: 11
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 12
Running with params: {'nb_iter': 20}
Optimization Iteration: 13
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 14
Running with params: {'nb_iter': 30}
Optimization Iteration: 15
Running with params: {'nb_iter': 10}
Optimization Iteration: 16
Running with params: {'nb_iter': 10}
Optimization Iteration: 17
Running with params: {'nb_iter': 10}
Optimization Iteration: 18
Running with params: {'nb_iter': 20}
Optimization Iteration: 19
Running with params: {'nb_iter': 20}
Optimization Iteration: 20
Running with params: {'nb_iter': 20}
Optimization Iteration: 21
Running with params: {'nb_iter': 30}
Optimization Iteration: 22
Running with params: {'nb_iter': 30}
Optimization Iteration: 23
Running with params: {'nb_iter': 30}
Optimization Iteration: 24
Running with params: {'nb_iter': 50}
Optimization Iteration: 25
Running with params: {'nb_iter': 50}
Optimization Iteration: 26
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 27
Running with params: {'nb_iter': 30}
Optimization Iteration: 28
Running with params: {'nb_iter': 30}
Optimization Iteration: 29
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 30
Running with params: {'nb_iter': 50}
Optimization Iteration: 31
Running with params: {'nb_iter': 20}
Optimization Iteration: 32
Running with params: {'nb_iter': 30}
Optimization Iteration: 33
Running with params: {'nb_iter': 20}
Optimization Iteration: 34
Running with params: {'nb_iter': 30}
Optimization Iteration: 35
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 36
Running with params: {'nb_iter': 30}
Optimization Iteration: 37
Running with params: {'nb_iter': 20}
Optimization Iteration: 38
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 39
Running with params: {'nb_iter': 30}
Optimization Iteration: 40
Running with params: {'nb_iter': 20}
Optimization Iteration: 41
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 42
Running with params: {'nb_iter': 30}
Optimization Iteration: 43
Running with params: {'nb_iter': 20}
Optimization Iteration: 44
Running with params: {'nb_iter': 50}
Optimization Iteration: 45
Running with params: {'nb_iter': 10}
Optimization Iteration: 46
Running with params: {'nb_iter': 30}
Optimization Iteration: 47
Running with params: {'nb_iter': 20}
Optimization Iteration: 48
Running with params: {'nb_iter': 10}
Optimization Iteration: 49
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 50
Running with params: {'nb_iter': 20}
Upto iteration 50: {'nb_iter': 30.0}
Optimization Iteration: 51
Running with params: {'nb_iter': 30}
Optimization Iteration: 52
Running with params: {'nb_iter': 20}
Optimization Iteration: 53
Running with params: {'nb_iter': 10}
Optimization Iteration: 54
Running with params: {'nb_iter': 30}
Optimization Iteration: 55
Running with params: {'nb_iter': 50}
Optimization Iteration: 56
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 57
Running with params: {'nb_iter': 30}
Optimization Iteration: 58
Running with params: {'nb_iter': 20}
Optimization Iteration: 59
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 60
Running with params: {'nb_iter': 30}
Optimization Iteration: 61
Running with params: {'nb_iter': 10}
Optimization Iteration: 62
Running with params: {'nb_iter': 20}
Optimization Iteration: 63
Running with params: {'nb_iter': 50}
Optimization Iteration: 64
Running with params: {'nb_iter': 30}
Optimization Iteration: 65
Running with params: {'nb_iter': 20}
Optimization Iteration: 66
Running with params: {'nb_iter': 30}
Optimization Iteration: 67
Running with params: {'nb_iter': 30}
Optimization Iteration: 68
Running with params: {'nb_iter': 30}
Optimization Iteration: 69
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 70
Running with params: {'nb_iter': 30}
Optimization Iteration: 71
Running with params: {'nb_iter': 30}
Optimization Iteration: 72
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 73
Running with params: {'nb_iter': 30}
Optimization Iteration: 74
Running with params: {'nb_iter': 20}
Optimization Iteration: 75
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 76
Running with params: {'nb_iter': 30}
Optimization Iteration: 77
Running with params: {'nb_iter': 20}
Optimization Iteration: 78
Running with params: {'nb_iter': 50}
Optimization Iteration: 79
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 80
Running with params: {'nb_iter': 30}
Optimization Iteration: 81
Running with params: {'nb_iter': 20}
Optimization Iteration: 82
Running with params: {'nb_iter': 30}
Optimization Iteration: 83
Running with params: {'nb_iter': 20}
Optimization Iteration: 84
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 85
Running with params: {'nb_iter': 30}
Optimization Iteration: 86
Running with params: {'nb_iter': 30}
Optimization Iteration: 87
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 88
Running with params: {'nb_iter': 20}
Optimization Iteration: 89
Running with params: {'nb_iter': 20}
Optimization Iteration: 90
Running with params: {'nb_iter': 30}
Optimization Iteration: 91
Running with params: {'nb_iter': 50}
Optimization Iteration: 92
Running with params: {'nb_iter': 30}
Optimization Iteration: 93
Running with params: {'nb_iter': 20}
Optimization Iteration: 94
Running with params: {'nb_iter': 10}
Optimization Iteration: 95
Running with params: {'nb_iter': 30}
Optimization Iteration: 96
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Optimization Iteration: 97
Running with params: {'nb_iter': 20}
Optimization Iteration: 98
Running with params: {'nb_iter': 30}
Optimization Iteration: 99
Running with params: {'nb_iter': 50}
Optimization Iteration: 100
Running with params: {'nb_iter': 40}
Higher iteration... returning...
Upto iteration 100: {'nb_iter': 30.0}
Breaking...
{'nb_iter': 30.0}
Best score: 4.078333333333333
Repeated params: 78
Trials: 101
Best param {'nb_iter': 30.0}
Reduction: 28.05903451519962%
Speedup: 1.3900286064568859x
Optimizer time: 33.13195776939392
