Table of contents
6 chapters · 15 sections
Convex Analysis
- 1.1Convex sets and functions
- 1.2Strong convexity and smoothness
Gradient Methods
- 2.1Gradient descent
- 2.2Convergence rates
- 2.3Line search
Stochastic Optimisation
- 3.1Stochastic gradient descent
- 3.2Mini-batches
- 3.3Momentum, RMSProp, Adam
Constrained Optimisation
- 4.1Lagrange multipliers
- 4.2KKT conditions
- 4.3Duality
Second-Order Methods
- 5.1Newton's method
- 5.2Quasi-Newton (BFGS)
Non-convex Landscapes
- 6.1Saddle points
- 6.2Training deep networks