TorchOpt

• The implicit gradient and zero-order differentiation APIs become generally available in TorchOpt 0.6.0.
• The project code structure is refactored and reorganized.

• Add unroll pragma for CUDA OPs by @JieRen98 and @XuehaiPan in #112.
• Add Python implementation of accelerated OP and pure-Python wheels by @XuehaiPan in #67.
• Add nan_to_num hook and gradient transformation by @XuehaiPan in #119.
• Add matrix inversion linear solver with neumann series approximation by @Benjamin-eecs and @XuehaiPan in #98.
• Add if condition of number of threads for CPU OPs by @JieRen98 in #105.
• Add implicit MAML omniglot few-shot classification example with OOP APIs by @XuehaiPan in #107.
• Add implicit MAML omniglot few-shot classification example by @Benjamin-eecs in #48.
• Add object-oriented modules support for implicit meta-gradient by @XuehaiPan in #101.
• Bump PyTorch version to 1.13.0 by @XuehaiPan in #104.
• Add zero-order gradient estimation by @JieRen98 in #93.
• Add RPC-based distributed training support and add distributed MAML example by @XuehaiPan in #83.
• Add full type hints by @XuehaiPan in #92.
• Add API documentation and tutorial for implicit gradients by @Benjamin-eecs and @JieRen98 and @XuehaiPan in #73.
• Add wrapper class for functional optimizers and examples of functorch integration by @vmoens and @Benjamin-eecs and @XuehaiPan in #6.
• Implicit differentiation support by @JieRen98 and @waterhorse1 and @XuehaiPan in #41.

• Refactor code organization by @XuehaiPan in #92 and #100.

• Fix implicit MAML omniglot few-shot classification example by @XuehaiPan in #108.
• Align results of distributed examples by @XuehaiPan in #95.
• Fix None in module containers by @XuehaiPan.
• Fix backward errors when using inplace sqrt_ and add_ by @Benjamin-eecs and @JieRen98 and @XuehaiPan.
• Fix LR scheduling by @XuehaiPan in #76.
• Fix the step count tensor (shape=(1,)) can change the shape of the scalar updates (shape=()) by @XuehaiPan in #71.

Full Changelog: https://github.com/metaopt/torchopt/compare/v0.5.0...v0.6.0