Progress:

• Change names in the backend to better match the new graph representation
• Remove Function superclass from Variable. Replace them with special nodes in the graph
• Implement the new function definition.
  • Make sure it's compatible with legacy definitions.
• Make sure Variables are never unpacked in the Engine (partially done in the first commit)
• Implement some of the built in Functions using the new format (to see if it's actually usable)
• Add a function that returns a list of higher order grads

The tasks below are left for future PRs. The first two can be easily paralellized and can be done by others too.

• Adapt all definitions of built in Functions (using @differentiable_once for now).
• Implement all backward functions using Variables (so they can be differentiated multiple times)
• Add deprecation warnings for old Function format
• Add a switch for saving .creator graph traces

Summary of changes

New function definition format

Note that the old declarations are still supported - most of the core implementations are still not converted.

New format allows to implement jacobian vector products (jvp, L-op) of functions depending on jvp's of other functions (aka. grad of grad, hessian-vector products).

The new declarations look like this:

class MultiplyAdd(Function):
                                                            # 1.
    @staticmethod
    def forward(ctx, input1, scalar, input2):               # 2.
        ctx.scalar = scalar                                 # 3.
        return input1 + scalar * input2

    @staticmethod
    def backward(ctx, grad_output):                         # 4.
        return grad_output, None, ctx.scalar * grad_output  # 5.

Adnotations:

1. Functions no longer can have an __init__ method. Think of them as pairs of pure functions that are formulas specifying how to compute the function and its jvp (Dfn * grad_output).
2. Beacuse of 1., forward can now accept arguments of arbitrary types (used to only accept Variables). Any Variables appearing in args will be unpacked into Tensors. Arguments are not recursively searched. For example, a list of Variables won't be unpacked into a list of Tensors, and they won't be registered as inputs in the graph. Keyword arguments are not supported (need arg ordering to construct the graph).
3. forward gets a ctx as a first argument - this is an object (of unspecified type - not an instance of this class) with an interface identical to self in old style definitions (save_for_backward, mark_non_differentiable, etc.) and is used to pass information to the backward call. For example, this function needs to save a scalar argument. Note that you shouldn't assign input or output tensors to it, however intermediate buffers are ok.
4. grad_output is now a Variable, and the whole backward method needs to be implemented in terms of Variables (they shouldn't be unpacked into tensors, or the derivative graph will be malformed, see notes on @once_differentiable below). ctx will be the same object that was passed to forward.
5. backward should return gradients for all arguments given to forward (even non-Variable arguments, but it should be None in such case). Unnecessary trailing Nones are still accepted (useful when forward has optional arguments).

For comparison, here's how a legacy definition of MultiplyAdd would look like:

class MultiplyAdd(Function):
    def __init__(self, scalar):
        super().__init__()
        self.scalar = scalar

    def forward(self, input1, scalar, input2):
        return input1 + self.scalar * input2

    def backward(self, grad_output):
        return grad_output, self.scalar * grad_output

@once_differentiable

The fact that backward now takes Variables might unnecessarily complicate implementations of custom function that e.g. call into other libs. For that reason, this PR also introduces a @once_differentiable decorator, that can be used to wrap backward. After adding it, backward functions will get a tensor grad_output and will be expected to return a grad input tensor for each tensor argument given in forward (and None for all other args).

class SciPyFunction(Function):
    @staticmethod
    def forward(ctx, input1, input2):
        return scipy.my_function(input1.numpy(), input2.numpy())

    @staticmethod
    @once_differentiable
    def backward(ctx, grad_output):
        return scipy.my_function2(grad_output), scipy.my_function3(grad_output)

torch.autograd.backward

Added create_graph. If True, the graph for vjp will be created (defaults to False), allowing to differentiate the grad computation. Defaults to True if grad_variables contains at least one non-volatile Variable, and False otherwise.
Renamed retain_variables to retain_graph. The old argument will remain supported until v0.3, but will print deprecation warnings. If unspecified, defaults to the value of create_graph.

If grad_variables contains tensors, they are automatically promoted to Variables (volatile unless create_graph is True). Also, None entries in grad_variables are now accepted if their corresponding variables entries are scalar Variables (grad_output filled with 1 is allocated for them). Additionally, if all grad_variables could be None the argument is now optional.

torch.autograd.grad

While Chainer-style API is great for first order grads, it doesn't work nearly as well when computing higher order derivatives. See this example:

x = Variable(torch.randn(2, 2), requires_grad=True)
x.mul(2).sum().backward(create_graph=True)
y = x.grad.mul(2).sum()
# This accumulates grad of grad into x.grad, adding together results of both backward() calls
y.backward() 

For that reason, this PR also implements grad - a functional-style function that computes the vjp, and instead of accumulating it into .grad of all leaves, it returns a list of grads w.r.t. given function inputs (parameters are considered inputs too).

Example:

from torch.autograd import grad

x = Variable(torch.randn(2, 2), requires_grad=True)
x.mul(2).sum().backward(create_graph=True)
y = x.grad.mul(2).sum()
# The line below **doesn't change x.grad**
x_hv = grad(y, x) # grad of y w.r.t. x. This argument would be the grad_output, but y is scalar

Arguments outputs, inputs, grad_outputs arguments can be both sequences of Variables (or Tensors and Nones in case of grad_outputs), or single Variables.

If one doesn't request the grad w.r.t. all leaf Variables, unneeded gradients are not computed, and won't be accumulated into them (by default grad has no side effects). If only_inputs argument is set to False, the whole graph will be differentiated, grads w.r.t. inputs will be returned in a list and not accumulated into .grad, grads w.r.t. all other leaves will be accumulated into their .grad.

.grad semantics

By default the semantics are the same as right now. When not using any of the options implemented in this PR, .grad Variables will be volatile, and incoming grads will be accumulated in-place (both Variable and its .data will be the same objects - while we don't guarantee that, some people depend on that in their scripts, so it's best to support it unless there's no other way).
However, when using derivative graphs, these Variables will need to have their .grad_fn set correctly, and shouldn't be modified in-place (they might have been used in some functions!). For that reason in such cases the .grad attribute will point to a new Variable, with new .data, after each accumulation.

To sum up:

Implementation details

• Variables no longer subclass Function and therefore can no longer appear in the graph. After this PR graphs contain AccumulateGrad nodes instead of Variables.
• Engine now supports per-function callbacks that are called before evaluating the function, and if they return false the apply function won't be called (all gradients will default to null, which is an equivalent of 0), and its next_functions won't be added to the ready queue (unless they are already waiting for execution and this was their last dependency).