Methodology

4. NEURONAL TRANSFER LEARNING

Figure 4.1: Two layers from AN NS transferred toAN NT

means that the associated weights of the neurons in those layers are transferred to theAN N_T. The figure4.1shows the transfer of two hidden layers, viz. the 2nd and 3rd, from anAN N_S to anAN N_T. All the weights in theAN N_T, except those cor- responding to the transferred layers, are initialized randomly. During the training process of the AN N_T, the weights associated with the transferred neurons remain frozen (unchanged), while all the others change as per the back-propagation rule.

4.2.1 Immuno-inspired Idiotypic network based Transfer

This work proposes an Idiotypic Network (IN) inspired transfer of neurons from AN NS to AN NT, where only some of the neurons in the layers are transferred.

As explained earlier in section 4.1, in an IN, the antibodies form a network by stimulating or suppressing the others based on conditions. This work metaphorizes the neurons as antibodies that form an Idiotypic-like network. The neurons in each layer constitute a population and form an Idiotypic Network which is local to that layer. If the number of neurons in a layer is large, then these are divided into populations of neurons, with an IN formed within each population.

Every neuron in the AN N_S is associated with a T emperature which is akin to the Concentration [111] of an antibody in the IN. At the end of an epoch in the training phase, the neuron which has lowest loss is stimulated by the others within its population. Likewise, this neuron suppresses the others. These suppres- sions and stimulations cause a neuron’s temperature to increase (or decrease) with stimulations (or suppressions) much like in the biological Idiotypic Network. Thus, at the end of the training, some neurons in every layer will have higher temper- atures than the others in their respective populations. The weights of these high

4.2. METHODOLOGY

Figure 4.2: Stimulations and Suppressions of the neurons in a population in AN N_S

temperature, Hot neurons, are then transferred to the AN NT and frozen so that back-propagation does not have any effect on them. Thus, instead of transferring the whole layer toAN NT, only theHot neurons are transferred. The weights of the other non-transferred neurons in theAN N_T are now re-tuned to achieve convergence and learning.

Ascertaining Hot Neurons in AN NT

At every epoch, a neuron is stimulated if that neuron has the lowest loss in that population; otherwise, it is suppressed. This mechanism is shown in fig. 4.2. These suppressions and stimulations result in a change in the temperature associated to the neurons. To model the change in the temperature of the neurons, this work uses a variant of Farmer’s equation [35].

Let the set L contain all the layers in AN N_S and Lⁱ ∈ L be the i^th layer in L. Let Nⁱ be the set of neurons in this layerLⁱ, with nnumber of neurons in Nⁱ. Every neuron N_kⁱ ∈ Nⁱ, is associated with a temperature, Θⁱ_k. These n neurons constitute a population which form an Idiotypic Network local to the layer Lⁱ. In each epoch, letN_jⁱbe the neuron with the lowest loss in this population. The amount of stimulation received by N_jⁱ from all other neurons in this population is given by:

4. NEURONAL TRANSFER LEARNING

τ_jⁱ = (1− |Ψⁱ_j|)∗ω_jⁱ (4.1) ω_jⁱ =

n

X

k=1

(β∗Θⁱ_k) wherek̸=j

where τ_jⁱ is the stimulation received by the neuron N_jⁱ, Ψⁱ_j is the net loss at that neuron (|Ψⁱ_j|being the absolute value), β is a positive constant (0> β≥1) to scale down the temperature values, ω_jⁱ is the sum of the temperatures of all other neurons except N_jⁱ and n is the number of neurons in that population. For every epoch, the suppression received by all other neurons in the population is given by:

ιⁱ_l =β∗(|Ψⁱ_l|)∗(Θⁱ_l/Ωⁱ_l) where∀l∈Nⁱ and l̸=j (4.2) Ωⁱ_l =

n

X

k=1

Θⁱ_k

whereιⁱ_l is the suppression received by the neuronN_lⁱ, Ψⁱ_lis the net loss at that neuron (|Ψⁱ_l|being the absolute value) and β is a positive constant (0> β ≥1) to scale down the temperature values and Ωⁱ_l is the sum of the temperatures of all the neurons. The changes in temperature of the neuron with lowest loss, N_jⁱ, and that of all the other neurons are governed by the following equations:

Θⁱ_j = Θⁱ_j+τ_jⁱ (4.3)

Θⁱ_l= Θⁱ_l−ιⁱ_l where∀l∈Nⁱ andl̸=j (4.4)

Algorithm 3 portrays the method of identification of the Hot neurons in the training process of AN N_S. The Θ values of all the neurons are initialized before the training process begins. In every epoch, both the forward and the backward propagations are performed in the conventional manner and the losses for every neuron in each epoch are recorded. For every layer, the neuron which has incurred

4.2. METHODOLOGY

Algorithm 3:Algorithm to ascertain Hot Neurons

1 Initialise Θ values

2 Lⁱ ←set of layers from which neurons will be selected

3 for every epoch do

4 f orward prop()

5 backward prop()

6 forevery Lⁱ ∈L do

7 Nⁱ ←Set of all the neurons in Lⁱ

8 j ←Index of the Neuron with lowest loss

9 Calculate τ_jⁱ and Stimulate N_jⁱ

10 for every N_lⁱ∈Nⁱ where l̸=j do

11 Calculate ιⁱ_l and Suppress N_lⁱ

12 Update Θ values of all neurons

13 forevery Lⁱ ∈L do

14 Determine ϑⁱ

the lowest loss is stimulated by the rest of the neurons in this layer. The stimulations received by this neuron are calculated based on τ_jⁱ, Ψⁱ_j and ωⁱ_j as per the equation 4.1. All other neurons in this layer are then suppressed by this neuron usingιⁱ_l and Ωⁱ_l as per the equation4.2. The value of Θ associated to the neuron N_jⁱ and also all other neurons are updated using the equations 4.3and 4.4, respectively.

Once the training is complete, the setϑⁱis determined, comprising the top best Hot neurons within the layer i, with higher Θ values.

Each of theHot neurons in this set has its respective associated set of neuronal weights. These weights are the ones that are transferred to the AN NT in the TL process, so as to reduce the overall loss of training, consequently increasing the efficacy of the target model.

Transferring of Hot Neurons to AN N_T

After theHotneurons in theAN N_Sare ascertained during the training process, they are transferred to theAN NT before training the latter. The weights associated with these transferred Hot neurons are frozen (remain unchanged) in the AN N_T while those of the others are initialized randomly and allowed to be re-tuned based on the propagation. Figure 4.3 depicts the transfer of the red coloured Hot neurons from AN NS toAN NT wherein they remain frozen (blue).

4. NEURONAL TRANSFER LEARNING

Figure 4.3: Idiotypic Network-based transfer ofHot Neurons (Red) from AN NS

toAN N_T where their weights remain frozen (blue)

After the transfer and freezing of the weights of theHot neurons in theAN N_T, it is trained using the target dataset,DT.