AutoDiff

Updated Wiki: Home

alexshtf — Sat, 11 May 2013 06:24:00 GMT

Project Description
A library that provides fast, accurate and automatic differentiation (computes derivative / gradient) of mathematical functions.

Getting AutoDiff
Using NuGet:

What is it for?
AutoDiff provides a simple and intuitive API for computing function gradients/derivatives along with a fast state-of-the-art algorithm for performing the computation. Such computations are mainly useful in numeric optimization scenarios.

Code example

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Documentation
In the Documentation tab right now we have some basic tutorials, and some others are under construction. We also have an article on CodeProject. In addition, the binary distribution contains XML comments for all public methods and a help file you can view with your favorite help viewer. And finally, the source control contains some code examples in addition to the library's code.

Motivation
There are many open and commercial .NET libraries that have numeric optimization as one of their features (for example, Microsoft Solver Foundation, AlgLib, Microsoft Research SHO Extreme Optimization, CenterSpace NMath) . Most of them require the user to be able to evaluate the function and the function's gradient. This library tries to save the work in manually developing the function's gradient and coding it.
Once the developer defines his/her function, the AutoDiff library can automatically evaluate and differentiate this function at any point. This allows easy development and prototyping of applications based on numeric optimization.

Features

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Used by

Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, Daniel Cohen-Or Geosemantic Snapping for Sketch-Based Modeling Eurographics 2013 proceedings
Hendrik Skubch, Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing
AlicaEngine - A cooperative planning engine for robotics. You can see it in action in this video
HeuristicsLab - a framework for heuristic and evolutionary algorithms that is developed by members of the Heuristic and Evolutionary Algorithms Laboratory (HEAL)

Updated Wiki: Home

alexshtf — Mon, 04 Feb 2013 09:49:28 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Used by

Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, Daniel Cohen-Or Geosemantic Snapping for Sketch-Based Modeling Proceedings of Eurographics 2013
Hendrik Skubch, Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing
AlicaEngine - A cooperative planning engine for robotics. You can see it in action in this video
HeuristicsLab - a framework for heuristic and evolutionary algorithms that is developed by members of the Heuristic and Evolutionary Algorithms Laboratory (HEAL)

Updated Wiki: Home

alexshtf — Mon, 04 Feb 2013 09:48:38 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Users

Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, Daniel Cohen-Or Geosemantic Snapping for Sketch-Based Modeling Proceedings of Eurographics 2013
Hendrik Skubch, Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing
AlicaEngine - A cooperative planning engine for robotics. You can see it in action in this video
HeuristicsLab - a framework for heuristic and evolutionary algorithms that is developed by members of the Heuristic and Evolutionary Algorithms Laboratory (HEAL)

Updated Wiki: Home

alexshtf — Mon, 26 Nov 2012 11:11:29 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Users

Hendrik Skubch, Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing
AlicaEngine - A cooperative planning engine for robotics. You can see it in action in this video
HeuristicsLab - a framework for heuristic and evolutionary algorithms that is developed by members of the Heuristic and Evolutionary Algorithms Laboratory (HEAL)

Updated Wiki: Home

alexshtf — Fri, 23 Nov 2012 20:08:13 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Users

Hendrik Skubch, Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing
AlicaEngine - A cooperative planning engine for robotics. You can see it in action in this video

Updated Wiki: Home

alexshtf — Fri, 23 Nov 2012 10:26:40 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Users

Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing

Updated Wiki: Home

alexshtf — Fri, 23 Nov 2012 09:40:50 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Users

Solving non-linear arithmetic constraints in soft realtime environments Proceedings of the 27th Annual ACM Symposium on Applied Computing

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

Updated Wiki: Home

alexshtf — Fri, 23 Nov 2012 09:34:49 GMT

using AutoDiff;

class Program
{
    public static void Main(string[] args)
    {
            // define variables
            var x = new Variable();
            var y = new Variable();
            var z = new Variable();

            // define our function
            var func = (x + y) * TermBuilder.Exp(z + x * y);

            // prepare arrays needed for evaluation/differentiation
            Variable[] vars = { x, y, z };
            double[] values = {1, 2, -3 };

            // evaluate func at (1, 2, -3)
            double value = func.Evaluate(vars, values);

            // calculate the gradient at (1, 2, -3)
            double[] gradient = func.Differentiate(vars, values);

            // print results
            Console.WriteLine("The value at (1, 2, -3) is " + value);
            Console.WriteLine("The gradient at (1, 2, -3) is ({0}, {1}, {2})", gradient[0], gradient[1], gradient[2]);
    }
}

Users

Academic research
- Solving non-linear arithmetic constraints in soft realtime environments

Fast! See 0.5 vs 0.3 benchmark and 0.3 benchmark.
Composition of functions using arithmetic operators, Exp, Log, Power and user-defined unary and binary functions.
Function gradient evaluation at specified points
Function value evaluation at specified points
Uses Code Contracts for specifying valid parameters and return values
Computes gradients using Reverse-Mode AD algorithm in linear time!
- Yes, it's faster than numeric approximation for multivariate functions
- You get both high accuracy and speed!

New Post: Double integration

alexshtf — Tue, 13 Nov 2012 16:22:44 GMT

Well, now this feature is supported, I know it is quite late, but it might help :)

Source code checked in, #1dc444607846

alexshtf — Mon, 08 Oct 2012 18:09:57 GMT

Added AutoDiff.NMath extension

Source code checked in, #3b65828053ce

alexshtf — Mon, 08 Oct 2012 12:50:56 GMT

Removed "perftest" project. It does not represent performance testing well. We will write a new one based on NUnit

Source code checked in, #9ca3d816b0df

alexshtf — Mon, 08 Oct 2012 12:49:03 GMT

Folder structure changes - moved the core library to the "core" folder

Source code checked in, #f5043e53d780

alexshtf — Mon, 08 Oct 2012 12:39:55 GMT

Created a solution folder

Commented Feature: Try to accelerate computation using IL generation [1574]

davidacoder — Wed, 26 Sep 2012 18:42:55 GMT

Use Reflection.Emit API to construct a specific set of instructions for a given function such that future differentiations will be faster than with current Visitor based implementations
Comments: Hm, interesting, and a bit surprising. I've worked quite a bit on dynamically generating code lately, if you want I could have a look at this as well, maybe you could send me the code of your experiment?

Closed Feature: Try to accelerate computation using IL generation [1574]

alexshtf — Wed, 26 Sep 2012 14:34:40 GMT

Commented Feature: Try to accelerate computation using IL generation [1574]

alexshtf — Wed, 26 Sep 2012 14:34:15 GMT

Use Reflection.Emit API to construct a specific set of instructions for a given function such that future differentiations will be faster than with current Visitor based implementations
Comments: An experiment shows that custom IL generation actually works SLOWER both when using DynamicMethod or Expression.Compile. Unless proven otherwise, it seems that manually generating IL does not accelerate the gradient computation but slows it down.

Created Feature: Try to accelerate computation using IL generation [1574]

alexshtf — Wed, 26 Sep 2012 14:33:08 GMT

Use Reflection.Emit API to construct a specific set of instructions for a given function such that future differentiations will be faster than with current Visitor based implementations

Source code checked in, #f7101e4e460e

Alex Shtof — Fri, 14 Sep 2012 00:27:52 GMT

Code cleanup - Remove redundant class (existed before performance optimizations prior to v1.0) - Fix contract classes to be abstract - Make a field private

Source code checked in, #5d581d91062d

Alex Shtof — Fri, 14 Sep 2012 00:05:09 GMT

Removed redundant solution files

Source code checked in, #116cf0ea528a

Alex Shtof — Fri, 14 Sep 2012 00:01:28 GMT

Fixed solution configurations