Multi-Layered Neural Network
Published November 10th, 2006 in ITP, blog, neural, p5, programming, teaching_
So after a fierce battle with my own neurons, I am ready to release part II of my Processing series: “Neural Network! Huah! What is it good for? (Sing it again, now.)”
This example implements a multi-layered neural network that learns via “back propogation.” It’s specifically trained to solve XOR. In other words, there are two inputs and the desired result is input1 XOR input2.
0,1 –> 1
1,0 –> 1
0,0 –> 0
1,1 –> 0
The structure looks something like this:
However, I think there might be a flaw in my back propogation learning algorithm. For whatever reason, with the above neural structure, I can only successfully train my network (starting with random connection weights between -1 and 1) approximately 60% of the time. For the other 40%, the network gets stuck and can’t find the proper solution. If I add two more neurons to the hidden layer, like so. . .
. . . it trains flawlessly, finding a reasonable solution space after a few thousand training iterations 100% of the time (or at least as far as I can reasonably test.) What am I missing?
Anyway, a more involved tutorial about the theory, concepts, algorithms, and code behind neural networks is forthcoming. . . at some point. . . after I invent that machine that makes time that is . .
If you are downloading the source, note that the code for the nn.jar package is contained in /xor/code/src/nn. Because I’m using a large number of classes in the design of the network, I didn’t want to restrict myself to Processing tabs.
Your network with two hidden nodes should train to a error of
Train to an error of. . . what?? The suspense is killing me. . .
I don’t think you’ve done anything wrong, one of the problems with backpropagation is that it sometimes converges on local minima (this is one of the reasons “momentum” is used during weight change). I get similar percentages of ~66% and ~98%: http://rpi.edu/~mcdonk/code/xor/ (My code allows for a more generic network structure but isn’t quite as nicely packaged
)
Thanks for the comments Kyle. After doing some further reading I came to similar conclusions. I’m going to make this example a bit more generic (maybe even turn it into a library) and investigate incorporating “momentum” into the weight changes. . .
It think Aaron Steed is in the middle of a backpropagation library already, actually: http://www.robotacid.com/PBeta/AILibrary/ (A more general connectionist library might be cool though, there are some other architectures people would appreciate…)
I love how gratuitously pretty you’ve made this.
I’ll be curious to see what else you come up with on the Neural Networks In Processing front (especially if you do make a library!) - I expect to be doing some experimenting in this area myself, for academic reasons (see http://www.mimeomai.f2s.com/oolong/ac/ if you’re curious).