Automatic differentiation collapses the linearized computational graph to compute a Jacobian. We don’t need exact gradients for SGD, so let’s use cheap Monte Carlo estimators instead: arxiv.org/abs/2007.10412
github.com/PrincetonLIPS/…
@denizzokt @AlexBeatson @NMcGreivy @jaduol1