| Neural Network Toolbox | |
Calculate Jacobian performance vector
Syntax
[je,jj,normje] = calcjejj(net,Pd,BZ,IWZ,LWZ,N,Ac,El,Q,TS,MR)
Description
This function calculates two values (related to the Jacobian of a network) required to calculate the network's Hessian, in a memory efficient way.
Two values needed to calculate the Hessian of a network are J*E (Jacobian times errors) and J'J (Jacobian squared). However the Jacobian J can take up a lot of memory. This function calculates J*E and J'J by dividing up training vectors into groups, calculating partial Jacobians Ji and its associated values Ji*Ei and Ji'Ji, then summing the partial values into the full J*E and J'J values.
This allows the J*E and J'J values to be calculated with a series of smaller Ji matrices, instead of a larger J matrix.
[je,jj,normgX] = calcjejj(net,PD,BZ,IWZ,LWZ,N,Ac,El,Q,TS,MR) takes,
net - Neural network.
PD - Delayed inputs.
BZ - Concurrent biases.
IWZ - Weighted inputs.
LWZ - Weighted layer outputs.
N - Net inputs.
Ac - Combined layer outputs.
El - Layer errors.
Q - Concurrent size.
TS - Time steps.
MR - Memory reduction factor
je - Jacobian times errors.
jj - Jacobian transposed time the Jacobian.normgX
normgX - Norm of gradient
Examples
Here we create a linear network with a single input element ranging from 0 to 1, two neurons, and a tap delay on the input with taps at zero, two, and four time steps. The network is also given a recurrent connection from layer 1 to itself with tap delays of [1 2].
net = newlin([0 1],2);
net.layerConnect(1,1) = 1;
net.layerWeights{1,1}.delays = [1 2];
Here is a single (Q = 1) input sequence P with five time steps (TS = 5), and the four initial input delay conditions Pi, combined inputs Pc, and delayed inputs Pd.
P = {0 0.1 0.3 0.6 0.4};
Pi = {0.2 0.3 0.4 0.1};
Pc = [Pi P];
Pd = calcpd(net,5,1,Pc);
Here the two initial layer delay conditions for each of the two neurons, and the layer targets for the two neurons over five time steps are defined.
Ai = {[0.5; 0.1] [0.6; 0.5]};
Tl = {[0.1;0.2] [0.3;0.1], [0.5;0.6] [0.8;0.9], [0.5;0.1]};
Here the network's weight and bias values are extracted, and the network's performance and other signals are calculated.
[perf,El,Ac,N,BZ,IWZ,LWZ] = calcperf(net,X,Pd,Tl,Ai,1,5);
Finally we can use calcgx to calculate the Jacobian times error, Jacobian squared, and the norm of the Jocobian times error using a memory reduction of 2.
[je,jj,normje] = calcjejj(net,Pd,BZ,IWZ,LWZ,N,Ac,El,1,5,2);
The results should be the same whatever the memory reduction used. Here a memory reduction of 3 is used.
[je,jj,normje] = calcjejj(net,Pd,BZ,IWZ,LWZ,N,Ac,El,1,5,3);
See Also
| | calcgx | calcjx | |