The backpropagation algorithm was used to train the multi layer perception MLP

Backpropagation

Backpropagation

Learning Algorithm

x₁

x_n

The backpropagation algorithm was used to train the multi layer perception MLP

MLP used to describe any general Feedforward (no recurrent connections) Neural Network FNN

Architecture of BP Nets

• Multi-layer, feed-forward networks have the following characteristics:

-They must have at least one hidden layer

– Hidden units must be non-linear units (usually with sigmoid activation functions)

– Fully connected between units in two consecutive layers, but no connection between units within one layer.

– For a net with only one hidden layer, each hidden unit receives input from all input units and sends output to all output units

– Number of output units need not equal number of input units

Other Feedforward Networks

• Madaline

– Multiple adalines (of a sort) as hidden nodes

• Adaptive multi-layer networks

– Dynamically change the network size (# of hidden nodes)

• Networks of radial basis function (RBF)

– e.g., Gaussian function

– Perform better than sigmoid function (e.g.,

Introduction to Backpropagation

• In 1969 a method for learning in multi-layer network, Backpropagation (or generalized delta rule) , was

invented by Bryson and Ho.

• It is best-known example of a training algorithm. Uses training data to adjust weights and thresholds of neurons so as to minimize the networks errors of prediction.

• Slower than gradient descent . • Easiest algorithm to understand

• How many hidden layers and hidden units per layer?

– Theoretically, one hidden layer (possibly with many hidden units) is sufficient for any L2 functions

– There is no theoretical results on minimum necessary # of hidden units (either problem dependent or independent)

– Practical rule :

• n = # of input units; p = # of hidden units • For binary/bipolar data: p = 2n

• For real data: p >> 2n

– Multiple hidden layers with fewer units may be

Training a BackPropagation Net

• Feedforward training of input patterns

– each input node receives a signal, which is broadcast to all of the hidden units

– each hidden unit computes its activation which is broadcast to all output nodes

• Back propagation of errors

– each output node compares its activation with the desired output

– based on these differences, the error is propagated back to all previous nodes Delta Rule

• Adjustment of weights

Three

Three

-

-

layer back

layer back

-

-

propagation neural network

propagation neural network

Generalized delta rule

• Delta rule only works for the output

layer.

Description of Training BP Net:

Feedforward Stage

1. Initialize weights with small, random values 2. While stopping condition is not true

– for each training pair (input/output):

• each input unit broadcasts its value to all hidden units

• each hidden unit sums its input signals &

applies activation function to compute its output signal

• each hidden unit sends its signal to the output units

Training BP Net:

Backpropagation stage

3. Each output computes its error term, its own weight correction term and its bias(threshold) correction term & sends it to layer below

Training a Back Prop Net:

Adjusting the Weights

5. Each output unit updates its weights and bias

6. Each hidden unit updates its weights and bias

– Each training cycle is called an epoch. The weights are updated in each cycle

– It is not analytically possible to determine

where the global minimum is. Eventually the algorithm stops in a low point, which may

How long should you train?

• Goal: balance between correct responses for training patterns & correct responses for new patterns (memorization v. generalization)

• In general, network is trained until it reaches an acceptable error rate (e.g. 95%)

Graphical description of of training multi-layer

neural network using

_BP

algorithm

• To teach the neural network we need training data set. The training data set consists of input signals (x₁ and x₂ ) assigned with corresponding target

(desired output) z.

• The network training is an iterative process. In each iteration weights coefficients of nodes are modified using new data from training data set.

• After this stage we can determine output signals values for each neuron in each network layer.

• Pictures below illustrate how signal is propagating through the network, Symbols w(x_m)n represent weights of connections between network input x_m

• Propagation of signals through the hidden layer. Symbols w_mn represent weights of connections between output of neuron m

• Propagation of signals through the output layer.

• In the next algorithm step the output signal of the network y is compared with the desired output value (the target), which is found in

• It is impossible to compute error signal for internal neurons directly, because output values of these neurons are unknown. For many years the effective method for training multiplayer networks has been unknown.

• Only in the middle eighties the backpropagation algorithm has been worked out. The idea is to propagate error signal δ

• The weights' coefficients w_mn used to propagate errors back are equal to this used during computing output value. Only the direction of data flow is changed (signals are propagated from output to inputs one after the

• When the error signal for each neuron is

computed, the weights

coefficients of each neuron input node may be modified. In formulas below

df(e)/de

represents derivative of neuron

activation

function (which weights are

Coefficient η affects network teaching speed. There are a few techniques to select this parameter. The first method is to start teaching process with large value of the parameter. While weights coefficients are being

established the parameter is being decreased gradually.

The second, more complicated, method starts teaching with small parameter value. During the teaching process the parameter is being

Training Algorithm 1

• Step 0: Initialize the weights to small random values

• Step 1: Feed the training sample through the network and determine the final output

• Step 2: Compute the error for each output unit, for unit k it is:

δ_k = (t_k – y_k)f’(y_in_k)

Required output

Actual output

Training Algorithm 2

• Step 3: Calculate the weight correction

term for each output unit, for unit k it is:

∆

w

_jk

=

ηδ

_k

z

_j

A small constant

Training Algorithm 3

• Step 4: Propagate the delta terms (errors)

back through the weights of the hidden

units where the delta input for the j

th

hidden unit is:

δ

_in

_j

=

Σ

δ

_k

w

_jk

k=1 m

The delta term for the jth _{hidden unit is:}

δ_j = δ_in_jf’(z_in_j) where

Training Algorithm 4

• Step 5: Calculate the weight correction term for the hidden units:

• Step 6: Update the weights:

• Step 7: Test for stopping (maximum cylces, small changes, etc)

∆

w

_ij

=

ηδ

_j

x

_i

Options

• There are a number of options in the

design of a backprop system

– Initial weights – best to set the initial weights (and all other free parameters) to random

numbers inside a small range of values (say: – 0.5 to 0.5)

– Number of cycles – tend to be quite large for backprop systems

Example

• The XOR function could not be solved by

a single layer perceptron network

• The function is:

X Y F

0 0 0

0 1 1

1 0 1

Initial Weights

•

R

andomly assign small weight values:

x y

Feedfoward – 1

st

Pass

Activation function f:

Final Result

•

A

fter about 500 iterations:

x y