Scipy optimization algorithms? (for minimizing neural network cost function) - python

Question

I wrote a neural network object in python that has a cost function and determines its gradients via back-propogation. I see a bunch of optimization functions here, but I have no idea how to implement them. I'm also having a hard time finding any example code to learn from.

Clearly I need to somehow tell it what parameters I'm trying to change, the cost function I'm trying to minimize, and then the gradient calculated by backprop. How do I tell, say, fmin_cg what's what?

bonus question: where can I learn about the differences in uses of the various algorithms?

===== OK, update =====

This is what I have atm:

def train(self, x, y_vals, iters = 400):
    t0 = concatenate((self.hid_t.reshape(-1), self.out_t.reshape(-1)), 1)
    self.forward_prop(x, t0)

    c = lambda v: self.cost(x, y_vals, v)
    g = lambda v: self.back_prop(y_vals, v)

    t_best = fmin_cg(c, t0, g, disp=True, maxiter=iters)
    self.hid_t = reshape(t_best[:,:(hid_n * (in_n+1))], (hid_n, in_n+1))
    self.out_t = reshape(t_best[:,(hid_n * (in_n+1)):], (out_n, hid_n+1))

And, this is the error it's throwing:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netset.py", line 27, in <module>
    net.train(x,y)
  File "neuralnet.py", line 60, in train
    t_best = fmin_cg(c, t0, g, disp=True, maxiter=iters)
  File "/usr/local/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 952, in fmin_cg
    res = _minimize_cg(f, x0, args, fprime, callback=callback, **opts)
  File "/usr/local/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 1017, in _minimize_cg
    deltak = numpy.dot(gfk, gfk)
ValueError: matrices are not aligned

...Halp!

score 1 · Accepted Answer · edited May 23 '17 at 10:28

I never used fmin_cg. I guess v is your weight vector. I did not find an error in your code, when I read the documentation. But I searched for your error and I found this: matrices are not aligned Error: Python SciPy fmin_bfgs

In addition, I think it is not garantueed that g(v) is always calculated after c(v). Thus, your backpropagation function should forward propagate x once again:

c = lambda v: self.cost(x, y_vals, v)
g = lambda v: self.cost(x, y_vals, v); self.back_prop(y_vals, v)
#             ------------------------

Or you can just pass one function that returns the cost function and the gradient as a tuple to avoid two forward propagations as Issam Laradji mentioned.

Good articles about optimization algorithms for artificial neural networks are:

I can recommend Levenberg-Marquardt. This algorithms works really well. Unfortunately every iteration step has cubic complexity O(n^3).

This is really cost inefficient, why is that forward propagation should be performed twice? In matlab's fmin_cg, you only need one function that returns both cost and gradient as a tuple. — IssamLaradji, Jul 02 '13 at 16:32

score 0 · Answer 2 · answered Jul 06 '12 at 22:55

0

To Address the Bonus Question only: Coursera had a great class on Machine learning, I'd check it out to see if it's still up and if it is, go through the key concepts to get a feel for each implementation of machine learning (if that makes sense) and then from there go more in depth.

answered Jul 06 '12 at 22:55

Aaron Tp

353
1
3
12

I just finished the class last week :) Unfortunately he doesn't go into much detail about the optimization algorithms. The course is fantastic though. – mavix Jul 06 '12 at 23:02
I know...that's why I only addressed the Bonus Question xP sorry mate – Aaron Tp Jul 06 '12 at 23:05

Scipy optimization algorithms? (for minimizing neural network cost function) - python

2 Answers2