Tensor Toolbox Optimization Methods
Most MATLAB optimization methods have different interfaces or none at all. In Tensor Toolbox, we are adopting wrappers for moderate consistency, and these will be called from within other Tensor Toolbox functions. Here we outline the choices, their installation instructions, and then the user-tunable parameters.
A few genearal notes:
- These methods all require explicit initial guesses as well as handles for the function and gradient calculations. These are handled by the calling routines.
- The parameters marked with asterisks should generally not be modified by the user because they will be set by the calling routine.
For more information on the details of these methods, see Developer Information for Tensor Toolbox Optimization Methods.
Contents
lbfgsb: Limited-Memory Quasi-Newton with Bound Constraints
In most methods, setting 'opt' to 'lbfgsb' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method.
Name | Description | Default |
lower | Lower bounds, can be vector or scalar | -Inf |
upper | Upper bounds, can be vector or scalar | +Inf |
maxiters | Max outer iterations | 1000 |
printitn | Printing frequency by iteration (0=no output) | 1 |
m | Limited-memory parameter | 5 |
subiters | Controls maximum calls to function-gradient evalations | 10 |
ftol | Stopping condition based on relative function change | 1e-10 |
gtol | Stopping condition based on gradient norm | 1e-5 |
mdesc (*) | Method description printed out before run | 'L-BFGS-B Optimization' |
xdesc (*) | Variable description | auto-generated |
Installation Instructions. Download and install L-BFGS-B by Stephen Becker. Please see that web page for full details on references, installation, etc. Here we provide cursory instructions for installation:
- Download the zip file https://github.com/stephenbeckr/L-BFGS-B-C/archive/master.zip
- Unzip and goto the Matlab/ subdirectoy with MATLAB
- Type compile_mex
- Add this directory to your saved path!
Detailed notes. The wrapper for this method is tt_lbfgsb in the Tensor Toolbox. Notes regarding mappings to the parameters of Becker's L-BFGS-B code:
- maxIts maps to maxiters and the default is increased from 100 to 1000
- printEvery maps to printitn
- maxTotalIts is set to maxiters*subiters and this effectively changes the default from 5000 to 10000
- factr is set to ftol / eps and this effectively changes the default from 1e7 to 4.5e5
- pgtol is set to gtol
lbfgs: Limited-Memory Quasi-Newton
In most methods, setting 'opt' to 'lbfgs' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method.
Name | Description | Default |
maxiters | Max outer iterations | 1000 |
printitn | Printing frequency by iteration (0=no output) | 1 |
m | Limited-memory parameter | 5 |
subiters | Controls maximum calls to function-gradient evalations | 10 |
ftol | Stopping condition based on relative function change | 1e-10 |
gtol | Stopping condition based on gradient norm | 1e-5 |
mdesc (*) | Method description printed out before run | 'Poblano L-BFGS Optimization' |
xdesc (*) | Variable description | auto-generated |
Installation Instructions. Download and install Poblano Toolbox, v1.2. Please see that web page for full details on references, installation, etc. Here we provide cursory instructions for installation:
- Download the zip file https://github.com/sandialabs/poblano_toolbox/archive/v1.2.zip
- Unzip and goto the poblano_toolbox-1.2/ subdirectoy within MATLAB
- Type install_poblano to save this directory to your path
Detailed notes. The wrapper for this method is tt_lbfgs in the Tensor Toolbox. Notes regarding mappings to the parameters of Poblano's L-BFGS code:
- MaxIters maps to maxiters
- Display maps to printitn
- MaxFuncEvals is set to maxiters*subiters
- RelFuncTol is set to ftol
- StopTol is set to gtol
fminunc: Optimizaton Toolbox Quasi-Newton Method
In most methods, setting 'opt' to 'fminunc' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method. This requires the MATLAB Optimization Toolbox.
Name | Description | Default |
maxiters | Max outer iterations | 1000 |
printitn | Display (0=no output) | 1 |
subiters | Controls maximum calls to function-gradient evalations | 10 |
gtol | Stopping condition based on gradient norm | 1e-5 |
mdesc (*) | Method description printed out before run | 'Quasi-Newton Optimization (via Optimization Toolbox)' |
xdesc (*) | Variable description | auto-generated |
adam: Stochastic Gradient Descent with Momentum
In most methods, setting 'opt' to 'adam' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method.
This is our own implementation of Adam. A failed epoch is one where the function value does not decrease. After a failed epoch, the method either reduces the learning rate (by decay) or exits (once the number of failed epochs exceeds maxfails).
Name | Description | Default |
lower | Lower bounds, can be vector or scalar | -Inf |
subiters | Number of iterations per epoch | 100 |
maxiters | Maximum number of epochs | 100 |
rate | Initial learning rate | 1e-2 |
maxfails | Maximum number of failed epochs | 1 |
decay | Decay of learning rate after failed epoch | 0.1 |
backup | Revert to end of previous epoch after failure | true |
ftol | uit if function value goes below this value | -Inf |
beta1 | Adam parameter | 0.9 |
beta2 | Adam parameter | 0.999 |
epsilon | Adam parameter | 1e-8 |
printitn | Printing frequency by epoch (0=no output) | 1 |
state (*) | State of random number generator | current state |
mdesc (*) | Method description printed out before run | 'Adam Stochastic Optimization' |
xdesc (*) | Variable description | auto-generated |
fdesc (*) | Description of (approximate) function computation | none |
gdesc (*) | Description of stochastic gradient computation | none |
fexact (*) | Boolean if function is computed exactly | true |