Repo: cleverhans
Filename: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/projects/cleverhans/cleverhans/future/torch/tests/test_attacks.py
ClassName: TestSPSA
Testname: test_targeted_adv_example_success_rate_l2
Params: nb_iter,365,19,ParamType.ITER,50
Assertion: self.assertGreater(adv_success, .7)
Original runtime: 5.814
>>Getting original runtime
Optimization Iteration: 1
Running with params: {'nb_iter': 50}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614602/assert_96811720_50
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 5.974, Max: 6.109999999999999, Min: 5.84
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.222720737296715
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 13...
Iter 12...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 6.027833333333333, Max: 6.409999999999999, Min: 5.7
Passed tests : 60
Failed tests : 0
Converged: False
Convergence score: 1.1759606143201788
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 6.0535555555555565, Max: 6.77, Min: 5.7
Passed tests : 90
Failed tests : 0
Converged: False
Convergence score: 1.008958165219276
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 6.049333333333333, Max: 6.77, Min: 5.7
Passed tests : 120
Failed tests : 0
Converged: True
Convergence score: 0.7559331271263403
Evaluating 120 values out of 120
Overall-timings: Avg: 6.049333333333333, Max: 6.77, Min: 5.7
Choosing distribution: logistic
Probability of fail (non-zero): 1.5454022345344677e-06
All-Passed: True
Probabilty of failure: 1.5454022345344677e-06
Runtime: 6.049333333333333
Score: 6.049333333333333
Best-score: 6.049333333333333
Best-param: {'nb_iter': 50}
>>Setting original runtime to 6.049333333333333
Optimization Iteration: 1
Running with params: {'nb_iter': 40}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614602/assert_96811720_40
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 29...
Iter 26...
Iter 27...
Iter 28...
Timings: Avg: 5.1563333333333325, Max: 5.3500000000000005, Min: 4.909999999999999
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.820691387103216
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 5.1626666666666665, Max: 5.43, Min: 4.909999999999999
Passed tests : 60
Failed tests : 0
Converged: True
Convergence score: 0.9915366199924259
Evaluating 60 values out of 60
Overall-timings: Avg: 5.1626666666666665, Max: 5.43, Min: 4.909999999999999
Choosing distribution: beta
Probability of fail : 0.0
All-Passed: True
Probabilty of failure: 0.0
Runtime: 5.1626666666666665
Score: 5.1626666666666665
Best-score: 5.1626666666666665
Best-param: {'nb_iter': 40}
Optimization Iteration: 2
Running with params: {'nb_iter': 20}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614602/assert_96811720_20
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 28...
Iter 29...
Timings: Avg: 3.4969999999999994, Max: 3.59, Min: 3.31
Passed tests : 22
Failed tests : 8
Converged: False
Convergence score: 1.5149196810877097
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 3.4963333333333333, Max: 3.85, Min: 3.31
Passed tests : 41
Failed tests : 19
Converged: True
Convergence score: 0.6143231925708165
Evaluating 60 values out of 60
Overall-timings: Avg: 3.4963333333333333, Max: 3.85, Min: 3.31
Choosing distribution: gumbel_l
Probability of fail (non-zero): 0.27272794427581676
Exceeding max probability of failure: 0.27272794427581676
All-Passed: False
Probabilty of failure: 0.27272794427581676
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 5.1626666666666665
Best-param: {'nb_iter': 40}
Optimization Iteration: 3
Running with params: {'nb_iter': 10}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614602/assert_96811720_10
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 26...
Iter 27...
Iter 23...
Iter 24...
Iter 29...
Iter 28...
Iter 25...
Timings: Avg: 2.7106666666666666, Max: 2.79, Min: 2.56
Passed tests : 2
Failed tests : 28
Half of samples failed, exiting...
All-Passed: False
Probabilty of failure: 1.0
Runtime: 10000.0
Score: 3.4028234663852886e+38
Best-score: 5.1626666666666665
Best-param: {'nb_iter': 40}
Optimization Iteration: 4
Running with params: {'nb_iter': 50}
Optimization Iteration: 5
Running with params: {'nb_iter': 20}
Optimization Iteration: 6
Running with params: {'nb_iter': 10}
Optimization Iteration: 7
Running with params: {'nb_iter': 30}
Logdir: /mnt/batch/tasks/workitems/opt1/job-1/Task_cleverhans/wd/borntobeflaky/tool/logs/optim_1597614539_cleverhans/run_1597614602/assert_96811720_30
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 28...
Iter 29...
Iter 27...
Timings: Avg: 4.322666666666666, Max: 4.63, Min: 4.159999999999999
Passed tests : 30
Failed tests : 0
Converged: False
Convergence score: 1.709963920141924
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 26...
Iter 25...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 4.348, Max: 4.63, Min: 4.1000000000000005
Passed tests : 60
Failed tests : 0
Converged: False
Convergence score: 1.3727739764491775
Continuing to next batch...
Launching 30 jobs, 30 in parallel
Iter 0...
Iter 1...
Iter 2...
Iter 3...
Iter 4...
Iter 5...
Iter 6...
Iter 7...
Iter 8...
Iter 9...
Iter 10...
Iter 11...
Iter 12...
Iter 13...
Iter 14...
Iter 15...
Iter 16...
Iter 17...
Iter 18...
Iter 19...
Iter 20...
Iter 21...
Iter 22...
Iter 23...
Iter 24...
Iter 25...
Iter 26...
Iter 27...
Iter 29...
Iter 28...
Timings: Avg: 4.343444444444445, Max: 4.63, Min: 4.1000000000000005
Passed tests : 90
Failed tests : 0
Converged: True
Convergence score: 0.7787200190553294
Evaluating 90 values out of 90
Overall-timings: Avg: 4.343444444444445, Max: 4.63, Min: 4.1000000000000005
Choosing distribution: lognorm
Probability of fail (non-zero): 0.0034273420876275926
All-Passed: True
Probabilty of failure: 0.0034273420876275926
Runtime: 4.343444444444445
Score: 4.343444444444445
Best-score: 4.343444444444445
Best-param: {'nb_iter': 30}
Optimization Iteration: 8
Running with params: {'nb_iter': 10}
Optimization Iteration: 9
Running with params: {'nb_iter': 20}
Optimization Iteration: 10
Running with params: {'nb_iter': 20}
Optimization Iteration: 11
Running with params: {'nb_iter': 10}
Optimization Iteration: 12
Running with params: {'nb_iter': 40}
Optimization Iteration: 13
Running with params: {'nb_iter': 20}
Optimization Iteration: 14
Running with params: {'nb_iter': 30}
Optimization Iteration: 15
Running with params: {'nb_iter': 20}
Optimization Iteration: 16
Running with params: {'nb_iter': 50}
Optimization Iteration: 17
Running with params: {'nb_iter': 20}
Optimization Iteration: 18
Running with params: {'nb_iter': 10}
Optimization Iteration: 19
Running with params: {'nb_iter': 20}
Optimization Iteration: 20
Running with params: {'nb_iter': 30}
Optimization Iteration: 21
Running with params: {'nb_iter': 30}
Optimization Iteration: 22
Running with params: {'nb_iter': 30}
Optimization Iteration: 23
Running with params: {'nb_iter': 30}
Optimization Iteration: 24
Running with params: {'nb_iter': 40}
Optimization Iteration: 25
Running with params: {'nb_iter': 30}
Optimization Iteration: 26
Running with params: {'nb_iter': 40}
Optimization Iteration: 27
Running with params: {'nb_iter': 30}
Optimization Iteration: 28
Running with params: {'nb_iter': 30}
Optimization Iteration: 29
Running with params: {'nb_iter': 40}
Optimization Iteration: 30
Running with params: {'nb_iter': 40}
Optimization Iteration: 31
Running with params: {'nb_iter': 50}
Optimization Iteration: 32
Running with params: {'nb_iter': 30}
Optimization Iteration: 33
Running with params: {'nb_iter': 40}
Optimization Iteration: 34
Running with params: {'nb_iter': 20}
Optimization Iteration: 35
Running with params: {'nb_iter': 30}
Optimization Iteration: 36
Running with params: {'nb_iter': 50}
Optimization Iteration: 37
Running with params: {'nb_iter': 20}
Optimization Iteration: 38
Running with params: {'nb_iter': 10}
Optimization Iteration: 39
Running with params: {'nb_iter': 30}
Optimization Iteration: 40
Running with params: {'nb_iter': 20}
Optimization Iteration: 41
Running with params: {'nb_iter': 50}
Optimization Iteration: 42
Running with params: {'nb_iter': 10}
Optimization Iteration: 43
Running with params: {'nb_iter': 20}
Optimization Iteration: 44
Running with params: {'nb_iter': 30}
Optimization Iteration: 45
Running with params: {'nb_iter': 40}
Optimization Iteration: 46
Running with params: {'nb_iter': 20}
Optimization Iteration: 47
Running with params: {'nb_iter': 10}
Optimization Iteration: 48
Running with params: {'nb_iter': 30}
Optimization Iteration: 49
Running with params: {'nb_iter': 20}
Optimization Iteration: 50
Running with params: {'nb_iter': 40}
Upto iteration 50: {'nb_iter': 30.0}
Optimization Iteration: 51
Running with params: {'nb_iter': 30}
Optimization Iteration: 52
Running with params: {'nb_iter': 20}
Optimization Iteration: 53
Running with params: {'nb_iter': 10}
Optimization Iteration: 54
Running with params: {'nb_iter': 50}
Optimization Iteration: 55
Running with params: {'nb_iter': 30}
Optimization Iteration: 56
Running with params: {'nb_iter': 20}
Optimization Iteration: 57
Running with params: {'nb_iter': 40}
Optimization Iteration: 58
Running with params: {'nb_iter': 30}
Optimization Iteration: 59
Running with params: {'nb_iter': 10}
Optimization Iteration: 60
Running with params: {'nb_iter': 20}
Optimization Iteration: 61
Running with params: {'nb_iter': 40}
Optimization Iteration: 62
Running with params: {'nb_iter': 30}
Optimization Iteration: 63
Running with params: {'nb_iter': 20}
Optimization Iteration: 64
Running with params: {'nb_iter': 50}
Optimization Iteration: 65
Running with params: {'nb_iter': 30}
Optimization Iteration: 66
Running with params: {'nb_iter': 30}
Optimization Iteration: 67
Running with params: {'nb_iter': 30}
Optimization Iteration: 68
Running with params: {'nb_iter': 30}
Optimization Iteration: 69
Running with params: {'nb_iter': 30}
Optimization Iteration: 70
Running with params: {'nb_iter': 40}
Optimization Iteration: 71
Running with params: {'nb_iter': 40}
Optimization Iteration: 72
Running with params: {'nb_iter': 20}
Optimization Iteration: 73
Running with params: {'nb_iter': 30}
Optimization Iteration: 74
Running with params: {'nb_iter': 40}
Optimization Iteration: 75
Running with params: {'nb_iter': 30}
Optimization Iteration: 76
Running with params: {'nb_iter': 20}
Optimization Iteration: 77
Running with params: {'nb_iter': 30}
Optimization Iteration: 78
Running with params: {'nb_iter': 20}
Optimization Iteration: 79
Running with params: {'nb_iter': 30}
Optimization Iteration: 80
Running with params: {'nb_iter': 40}
Optimization Iteration: 81
Running with params: {'nb_iter': 30}
Optimization Iteration: 82
Running with params: {'nb_iter': 50}
Optimization Iteration: 83
Running with params: {'nb_iter': 20}
Optimization Iteration: 84
Running with params: {'nb_iter': 30}
Optimization Iteration: 85
Running with params: {'nb_iter': 40}
Optimization Iteration: 86
Running with params: {'nb_iter': 20}
Optimization Iteration: 87
Running with params: {'nb_iter': 30}
Optimization Iteration: 88
Running with params: {'nb_iter': 40}
Optimization Iteration: 89
Running with params: {'nb_iter': 20}
Optimization Iteration: 90
Running with params: {'nb_iter': 30}
Optimization Iteration: 91
Running with params: {'nb_iter': 50}
Optimization Iteration: 92
Running with params: {'nb_iter': 30}
Optimization Iteration: 93
Running with params: {'nb_iter': 20}
Optimization Iteration: 94
Running with params: {'nb_iter': 30}
Optimization Iteration: 95
Running with params: {'nb_iter': 40}
Optimization Iteration: 96
Running with params: {'nb_iter': 10}
Optimization Iteration: 97
Running with params: {'nb_iter': 30}
Optimization Iteration: 98
Running with params: {'nb_iter': 20}
Optimization Iteration: 99
Running with params: {'nb_iter': 20}
Optimization Iteration: 100
Running with params: {'nb_iter': 40}
Upto iteration 100: {'nb_iter': 30.0}
Breaking...
{'nb_iter': 30.0}
Best score: 4.343444444444445
Repeated params: 96
Trials: 101
Best param {'nb_iter': 30.0}
Reduction: 28.19961795606494%
Speedup: 1.3927502494180246x
Optimizer time: 64.43218803405762
