Explain to me Neural Networks

[Tannhauser]

I have tried to get a grasp of what neural networks actually do and what they are. I have a background in mathematics, statistics and programming, including stuff like linear optimization and graph theory. It seems to me that NNs borrow from these fields, but I find it very hard to understand what problems NNs actually solve and how they work. People in this field seem to find it hard to write about it without namedropping a million different fancy-sounding concepts in every sentence.

I would be impressed if someone could explain what NNs do and how they work, while following the following guidelines:

- No namedropping of concepts without defining clearly what they are
- Explain the basic mechanism of NNs while refraining from using abstract terms like "hidden layers" or similar
- Explain what problems they solve which cannot be solved with other, "classical" methods like linear optimization, regression etc
- In what ways are they similar to the classical methods

 

[ProxyAmenRa]

Neural networks are methods of mathematical modelling whereby the relationship between independent variables and the dependent don't have to be explicitly specified. They are composed of neurons, neuron layers and weights. Weights are simply a fraction multiplier that's use to determine the significance of an input into a neuron. A neuron is merely a function such as tan(x) where x is the summation of the inputs multiplied by the weights. A neuron layer is simply a set of neurons. Each neuron received the output of the neurons of the previous layer, visually represented by weight lines. A neural network may have one dependent variable output or multiple depending on the number of neurons in the last layer. The weights throughout the neural network are altered according to a training algorithm. The most common training algorithm (back propagation) reduces the square of the error.

I have used neural networks to create predictive models, classification engines, pattern recognition, control systems, etc. Predictive modelling is similar to non-linear regression. Classification is similar to logistics regression. Pattern recognition is similar to optimising heuristics combined with feature extraction. Control systems is similar to writing a control system according from first principles.

I have found that neural networks perform generally worse than classical techniques. They are very poor at extrapolating.

If you want to get your hands dirty, feed forward back propagation is as easy as it gets. It is quite simple to program. You can test it by attempting to fit data created from an arbitrarily created multivariate function. Once that works, add Gaussian white noise to the output of your arbitrarily created function and test again.

 

[Ex-User (9086)]

It's good with problems where it's easier to define what a good outcome would be, rather than point in any specific direction. Add noise, some kind of genetic algorithm for network fitness comparison and iterate or have it learn over millions of cycles.

Training can adjust weights or add new nodes (neurons) to the mix, it works well with many heuristics. If it's learning by evolution, for example, it randomly generates a starting population runs all cases and the least hopeful variants are culled.

 

[Haim]

I have tried to get a grasp of what neural networks actually do and what they are. I have a background in mathematics, statistics and programming, including stuff like linear optimization and graph theory. It seems to me that NNs borrow from these fields, but I find it very hard to understand what problems NNs actually solve and how they work. People in this field seem to find it hard to write about it without namedropping a million different fancy-sounding concepts in every sentence.

I would be impressed if someone could explain what NNs do and how they work, while following the following guidelines:

- No namedropping of concepts without defining clearly what they are
- Explain the basic mechanism of NNs while refraining from using abstract terms like "hidden layers" or similar
- Explain what problems they solve which cannot be solved with other, "classical" methods like linear optimization, regression etc
- In what ways are they similar to the classical methods

Basically to produce an approximate function(trained neural network) from a data of outputs and inputs.The "outputs and inputs" can be just how much the neural network is doing well.
Say you want recognize if their is a dog in the picture, first you need to train the network, you need a series of photos which have a percentage of similarity to dogs, things which are similar to dogs such as cats,wolves should be in the range 1%-99%.
The network is adjusted so it's outputs(similarity to dog) giving the inputs(pictures) are close to your training data.The way it doing it is statistical, making statistic of what output usually produced by that input, it's mapping inputs to outputs.
It is kind of regression, it can work on very complex,highly nonlinear data.

It's good for when the algorithm of the function you want to make works too much complex or/and unclear for classical methods.Say the way we recognise objects, the way we move every muscle to walk, voice recognition.
Or when you want to find a pattern in complex data, for example the player's next move in a go game.Of course there are more crazy stuff people are doing with NN.
That one makes "handwriting" picture from text.
http://www.cs.toronto.edu/~graves/h...rt_all_labels.nc,1082+554&bias=0.15&samples=3

 

[Architect]

Nobody has mentioned Deep Neural Networks which is the preeminent technology, after they achieved some breakthroughs. A DNN is what we've been looking for with ANN's (artificial neural nets) for 30 years, which is one which has more than a few layers and so can be made general purpose. So far you really had to work on the nets to get them to learn, basically by picking the feature set (e.g. helping a cat NN recognize cats, but tweaking it to recognize 'features' like pointy ears, fur, etc). Now you can just throw a lot of data at a DNN and off it goes.

The trick has been the explosive growth of GPU's from the gaming world, and realizing that a RBM (reduced Boltzmann machine - look it up if you're interested) is a way to run a NN on a GPU. This year NVIDIA is releasing the Pascal architecture targeted at DNN's - 32 GB VRAM, 16nm FinFET and probably twice as many cores as last year. I've been running DNN's for the last few years and am planning on a 10 Tflop DNN trainer later this year.

Tannhauser, the reason there's so much jargon and such is you can't get away from it. Once you try to understand what's going on after looking at the typical DAG (directed acrylic graph) pictures you always see, it just gets technical.

If you have a few bucks get a NVIDIA Jetson TX1 ($300 academic price) and run some NN's on it.

 

[Tannhauser]

I am impressed with your knowledge, guys. Very interesting stuff. And thanks to Proxy for a great basic and clear explanation -- it actually made it a lot clearer.