Neural Networks started as an academic discipline over 50 years ago with the publication by McCulloch and Pitts of their famous result that any logical problem can be solved by a suitable network composed of so-called binary decision nodes. These are processors which make the simplest possible decision, that is, whether or not to respond to a weighted sum of their inputs, at each time step. The details of the decision are also of the simplest, being whether or not the weighted sum is positive or not. There could not be simpler processors, yet the results of McCulloch and Pitts showed that such a network has as much logical power as is ever needed to solve any difficult logical problem. This result helped usher in the age of the digital computer. Moreover, the similarity of the simple processors to the living cells—the neurons—of the brain has also has led to increasingly more precise models of brain processes.
During the decade after the introduction of binary decision neurons, it was suggested that the manner in which they influenced each other—the so-called connection weights that determine the weighting in the 'linear weighted sum' of inputs—could be modified. In particular, automatic training methods were developed so that neural nets could solve a whole range of classification and function mapping problems.
These methods have now been expanded so as to be increasingly effective. They have also been developed so as to avoid some of the difficulties recognized as facing them in the 1960s and 1970s. In particular, there was the difficulty of determining the degree of credit or blame to be assigned to a neuron for its response to a particular input if the neuron only handed its response to another one but not to the outside, a so-called hidden neuron. This credit assignment problem was solved by the method of error-back-propagation, introduced by Paul Werbos in 1974, and well publicised by Rumelhart and McClelland in the mid-1980s, and which then became almost synonymous with neural networks in some application areas.
The growth of the subject of neural networks since then has been not only in the development of ever greater expertise in the application of this training technique, but more especially in the creation of new training methods so as to solve a wider range of problems more efficiently. At the same time, there has been an ever greater understanding of the underlying manner in which