GClasses
GClasses::GNeuralNet Class Reference

A feed-forward artificial neural network, or multi-layer perceptron. More...

#include <GNeuralNet.h>

Inheritance diagram for GClasses::GNeuralNet:
GClasses::GIncrementalLearner GClasses::GSupervisedLearner GClasses::GTransducer

Public Member Functions

 GNeuralNet ()
 
 GNeuralNet (GDomNode *pNode, GLearnerLoader &ll)
 Load from a text-format. More...
 
virtual ~GNeuralNet ()
 
void addLayer (GNeuralNetLayer *pLayer, size_t position=INVALID_INDEX)
 Adds pLayer to the network at the specified position. (The default position is at the end in feed-forward order.) Takes ownership of pLayer. If the number of inputs and/or outputs do not align with the previous and/or next layers, then any layers with FLEXIBLE_SIZE inputs or FLEXIBLE_SIZE outputs will be resized to accomodate. If both layers have fixed sizes that do not align, then an exception will be thrown. More...
 
void align (const GNeuralNet &that)
 Swaps nodes in hidden layers of this neural network to align with those in that neural network, as determined using bipartite matching. (This might be done, for example, before averaging weights together.) More...
 
void autoTune (GMatrix &features, GMatrix &labels)
 Uses cross-validation to find a set of parameters that works well with the provided data. That is, this method will add a good number of hidden layers, pick a good momentum value, etc. More...
 
void backpropagate (const double *pTarget, size_t startLayer=INVALID_INDEX)
 This method assumes that the error term is already set at every unit in the output layer. It uses back-propagation to compute the error term at every hidden unit. (It does not update any weights.) More...
 
void backpropagateSingleOutput (size_t outputNode, double target, size_t startLayer=INVALID_INDEX)
 Backpropagates error from a single output node over all of the hidden layers. (Assumes the error term is already set on the specified output node.) More...
 
void bleedWeightsL1 (double beta)
 Adjust the magnitudes of the incoming and outgoing connections by amount beta, such that sum of the absolute values of the weights remains constant. If beta is 0, no bleeding will occur. If beta is 1, total bleeding will occur, such that the sum of the absolute values of the input and output weights are the same. Typically, only a small amount of bleeding is desirable, so values close to 0 (like 0.00001) are used. More...
 
void bleedWeightsL2 (double beta)
 Adjust the magnitudes of the incoming and outgoing connections by amount beta, such that sum-squared-weights remains constant. If beta is 0, no bleeding will occur. If beta is 1, total bleeding will occur, such that the sum of the squares of the input and output weights are the same. Typically, only a small amount of bleeding is desirable, so values close to 0 (like 0.00001) are used. More...
 
virtual bool canImplicitlyHandleMissingFeatures ()
 See the comment for GTransducer::canImplicitlyHandleMissingFeatures. More...
 
virtual bool canImplicitlyHandleNominalFeatures ()
 See the comment for GTransducer::canImplicitlyHandleNominalFeatures. More...
 
virtual bool canImplicitlyHandleNominalLabels ()
 See the comment for GTransducer::canImplicitlyHandleNominalLabels. More...
 
virtual void clear ()
 See the comment for GSupervisedLearner::clear. More...
 
GMatrixcompressFeatures (GMatrix &features)
 Performs principal component analysis (without reducing dimensionality) on the features to shift the variance of the data to the first few columns. Adjusts the weights on the input layer accordingly, such that the network output remains the same. Returns the transformed feature matrix. More...
 
void containIntrinsics (GMatrix &intrinsics)
 Finds the column in the intrinsic matrix with the largest deviation, then centers the matrix at the origin and renormalizes so the largest deviation is 1. Also renormalizes the input layer so these changes will have no effect. More...
 
void contractWeights (double factor, bool contractBiases)
 Contract all the weights in this network by the specified factor. More...
 
void copyPrediction (double *pOut)
 This method assumes forwardProp has been called. It copies the predicted vector into pOut. More...
 
void copyStructure (GNeuralNet *pOther)
 Makes this neural network into a deep copy of pOther, including layers, nodes, settings and weights. More...
 
void copyWeights (GNeuralNet *pOther)
 Copy the weights from pOther. It is assumed (but not checked) that pOther already has the same network structure as this neural network. This method is faster than copyStructure. More...
 
size_t countWeights () const
 Counts the number of weights in the network. (This value is not cached, so you should cache it rather than frequently call this method.) More...
 
size_t countWeights (size_t layer) const
 
void descendGradient (const double *pFeatures, double learningRate, double momentum)
 This method assumes that the error term is already set for every network unit (by a call to backpropagate). It adjusts weights to descend the gradient of the error surface with respect to the weights. More...
 
void descendGradientSingleOutput (size_t outputNeuron, const double *pFeatures, double learningRate, double momentum)
 This method assumes that the error term has been set for a single output network unit, and all units that feed into it transitively (by a call to backpropagateSingleOutput). It adjusts weights to descend the gradient of the error surface with respect to the weights. More...
 
void diminishWeights (double amount, bool regularizeBiases=true, size_t startLayer=0, size_t layerCount=INVALID_INDEX)
 Diminishes all weights in the network by the specified amount. This can be used to implemnet L1 regularization, which promotes sparse representations. That is, it makes many of the weights approach zero. More...
 
void forwardProp (const double *pInputs, size_t maxLayers=INVALID_INDEX)
 Evaluates a feature vector. (The results will be in the nodes of the output layer.) The maxLayers parameter can limit how far into the network values are propagated. More...
 
double forwardPropSingleOutput (const double *pInputs, size_t output)
 This is the same as forwardProp, except it only propagates to a single output node. It returns the value that this node outputs. If bypassInputWeights is true, then pInputs is assumed to have the same size as the first layer, and it is fed into the net of this layer, instead of the inputs. More...
 
void gradientOfInputs (double *pOutGradient)
 This method assumes that the error term is already set for every network unit. It calculates the gradient with respect to the inputs. That is, it points in the direction of changing inputs that makes the error bigger. (Note that this calculation depends on the weights, so be sure to call this method before you call descendGradient. Also, note that descendGradient depends on the input features, so be sure not to update them until after you call descendGradient.) More...
 
void gradientOfInputsSingleOutput (size_t outputNeuron, double *pOutGradient)
 This method assumes that the error term is already set for every network unit. It calculates the gradient with respect to the inputs. That is, it points in the direction of changing inputs that makes the error bigger. This method assumes that error is computed for only one output neuron, which is specified. (Note that this calculation depends on the weights, so be sure to call this method before you call descendGradientSingleOutput.) Also, note that descendGradientSingleOutput depends on the input features, so be sure not to update them until after you call descendGradientSingleOutput.) More...
 
double improvementThresh ()
 Returns the threshold ratio for improvement. More...
 
GMatrixinternalTraininGMatrix ()
 Gets the internal training data set. More...
 
GMatrixinternalValidationData ()
 Gets the internal validation data set. More...
 
void invertNode (size_t layer, size_t node)
 Inverts the weights of the specified node, and adjusts the weights in the next layer (if there is one) such that this will have no effect on the output of the network. (Assumes this model is already trained.) More...
 
GNeuralNetLayerlayer (size_t n)
 Returns a reference to the specified layer. More...
 
size_t layerCount () const
 Returns the number of layers in this neural network. These include the hidden layers and the output layer. (The input vector does not count as a layer.) More...
 
double learningRate () const
 Returns the current learning rate. More...
 
virtual void maxNorm (double max)
 Scales weights if necessary such that the manitude of the weights (not including the bias) feeding into each unit are <= max. More...
 
double momentum () const
 Returns the current momentum value. More...
 
GNeuralNetLayeroutputLayer ()
 Returns a reference to the last layer. More...
 
void perturbAllWeights (double deviation)
 Perturbs all weights in the network by a random normal offset with the specified deviation. More...
 
virtual void predict (const double *pIn, double *pOut)
 See the comment for GSupervisedLearner::predict. More...
 
virtual void predictDistribution (const double *pIn, GPrediction *pOut)
 See the comment for GSupervisedLearner::predictDistribution. More...
 
void pretrainWithAutoencoders (const GMatrix &features, size_t maxLayers=INVALID_INDEX)
 Pretrains the network using the method of stacked autoencoders. This method performs the following steps: 1- Start with the first layer. 2- Create an autoencoder using the current layer as the encoder and a temporary layer as the decoder. 3- Train the autoencoder with the features. 4- Discard the decoder. 5- Map the features through the encoder to obtain a set of features for training the next layer, and go to step 2 until all (or maxLayers) layers have been pretrained in this manner. More...
 
void printWeights (std::ostream &stream)
 Prints weights in a human-readable format. More...
 
GNeuralNetLayerreleaseLayer (size_t index)
 Drops the layer at the specified index. Returns a pointer to the layer. You are then responsible to delete it. (This doesn't resize the remaining layers to fit with each other, so the caller is responsible to repair any such issues before using the neural network again.) More...
 
void scaleWeights (double factor, bool scaleBiases=true, size_t startLayer=0, size_t layerCount=INVALID_INDEX)
 Multiplies all weights in the network by the specified factor. This can be used to implement L2 regularization, which prevents weight saturation. More...
 
void scaleWeightsSingleOutput (size_t output, double lambda)
 Just like scaleWeights, except it only scales the weights in one of the output units. More...
 
virtual GDomNodeserialize (GDom *pDoc) const
 Saves the model to a text file. More...
 
void setImprovementThresh (double d)
 Specifies the threshold ratio for improvement that must be made since the last validation check for training to continue. (For example, if the mean squared error at the previous validation check was 50, and the mean squared error at the current validation check is 49, then training will stop if d is > 0.02.) More...
 
void setLearningRate (double d)
 Set the learning rate. More...
 
void setMomentum (double d)
 Momentum has the effect of speeding convergence and helping the gradient descent algorithm move past some local minimums. More...
 
void setUseInputBias (bool b)
 Specify whether to use an input bias. (The default is false.) This feature is used with generative-backpropagation, which adjusts inputs to create latent features. More...
 
void setValidationPortion (double d)
 Set the portion of the data that will be used for validation. If the value is 0, then all of the data is used for both training and validation. More...
 
void setWeights (const double *pWeights)
 Sets all the weights from an array of doubles. The number of doubles in the array can be determined by calling countWeights(). More...
 
void setWeights (const double *pWeights, size_t layer)
 
void setWindowSize (size_t n)
 Sets the number of epochs that will be performed before each time the network is tested again with the validation set to determine if we have a better best-set of weights, and whether or not it's achieved the termination condition yet. (An epochs is defined as a single pass through all rows in the training set.) More...
 
double sumSquaredPredictionError (const double *pTarget)
 This method assumes forwardProp has been called. It computes the sum squared prediction error with the specified target vector. More...
 
virtual bool supportedFeatureRange (double *pOutMin, double *pOutMax)
 See the comment for GTransducer::supportedFeatureRange. More...
 
virtual bool supportedLabelRange (double *pOutMin, double *pOutMax)
 See the comment for GTransducer::supportedFeatureRange. More...
 
void swapNodes (size_t layer, size_t a, size_t b)
 Swaps two nodes in the specified layer. If layer specifies one of the hidden layers, then this will have no net effect on the output of the network. (Assumes this model is already trained.) More...
 
virtual void trainIncremental (const double *pIn, const double *pOut)
 See the comment for GIncrementalLearner::trainIncremental. More...
 
void trainIncrementalWithDropConnect (const double *pIn, const double *pOut, double probOfDrop)
 Presents a pattern for training. Applies dropConnect to the weights. Note that when training with dropConnect is complete, you should call scaleWeights(1.0 - probOfDrop, false) to compensate for the scaling effect dropConnect has on the weights. More...
 
void trainIncrementalWithDropout (const double *pIn, const double *pOut, double probOfDrop)
 Presents a pattern for training. Applies dropout to the activations of hidden layers. Note that when training with dropout is complete, you should call scaleWeights(1.0 - probOfDrop, false, 1) to compensate for the scaling effect dropout has on the weights. More...
 
virtual void trainSparse (GSparseMatrix &features, GMatrix &labels)
 See the comment for GIncrementalLearner::trainSparse Assumes all attributes are continuous. More...
 
size_t trainWithValidation (const GMatrix &trainFeatures, const GMatrix &trainLabels, const GMatrix &validateFeatures, const GMatrix &validateLabels)
 Train the network until the termination condition is met. Returns the number of epochs required to train it. More...
 
bool useInputBias () const
 Returns whether this neural network utilizes an input bias. More...
 
void weights (double *pOutWeights) const
 Serializes the network weights into an array of doubles. The number of doubles in the array can be determined by calling countWeights(). More...
 
void weights (double *pOutWeights, size_t layer) const
 
size_t windowSize ()
 Returns the number of epochs to perform before the validation data is evaluated to see if training should stop. More...
 
- Public Member Functions inherited from GClasses::GIncrementalLearner
 GIncrementalLearner ()
 General-purpose constructor. More...
 
 GIncrementalLearner (GDomNode *pNode, GLearnerLoader &ll)
 Deserialization constructor. More...
 
virtual ~GIncrementalLearner ()
 Destructor. More...
 
void beginIncrementalLearning (const GRelation &featureRel, const GRelation &labelRel)
 You must call this method before you call trainIncremental. More...
 
virtual bool canTrainIncrementally ()
 Returns true. More...
 
virtual bool isFilter ()
 Only the GFilter class should return true to this method. More...
 
- Public Member Functions inherited from GClasses::GSupervisedLearner
 GSupervisedLearner ()
 General-purpose constructor. More...
 
 GSupervisedLearner (GDomNode *pNode, GLearnerLoader &ll)
 Deserialization constructor. More...
 
virtual ~GSupervisedLearner ()
 Destructor. More...
 
void basicTest (double minAccuracy1, double minAccuracy2, double deviation=1e-6, bool printAccuracy=false, double warnRange=0.035)
 This is a helper method used by the unit tests of several model learners. More...
 
virtual bool canGeneralize ()
 Returns true because fully supervised learners have an internal model that allows them to generalize previously unseen rows. More...
 
void confusion (GMatrix &features, GMatrix &labels, std::vector< GMatrix * > &stats)
 Generates a confusion matrix containing the total counts of the number of times each value was expected and predicted. (Rows represent target values, and columns represent predicted values.) stats should be an empty vector. This method will resize stats to the number of dimensions in the label vector. The caller is responsible to delete all of the matrices that it puts in this vector. For continuous labels, the value will be NULL. More...
 
void precisionRecall (double *pOutPrecision, size_t nPrecisionSize, GMatrix &features, GMatrix &labels, size_t label, size_t nReps)
 label specifies which output to measure. (It should be 0 if there is only one label dimension.) The measurement will be performed "nReps" times and results averaged together nPrecisionSize specifies the number of points at which the function is sampled pOutPrecision should be an array big enough to hold nPrecisionSize elements for every possible label value. (If the attribute is continuous, it should just be big enough to hold nPrecisionSize elements.) If bLocal is true, it computes the local precision instead of the global precision. More...
 
const GRelationrelFeatures ()
 Returns a reference to the feature relation (meta-data about the input attributes). More...
 
const GRelationrelLabels ()
 Returns a reference to the label relation (meta-data about the output attributes). More...
 
double sumSquaredError (const GMatrix &features, const GMatrix &labels)
 Computes the sum-squared-error for predicting the labels from the features. For categorical labels, Hamming distance is used. More...
 
void train (const GMatrix &features, const GMatrix &labels)
 Call this method to train the model. More...
 
virtual double trainAndTest (const GMatrix &trainFeatures, const GMatrix &trainLabels, const GMatrix &testFeatures, const GMatrix &testLabels)
 Trains and tests this learner. Returns sum-squared-error. More...
 
- Public Member Functions inherited from GClasses::GTransducer
 GTransducer ()
 General-purpose constructor. More...
 
 GTransducer (const GTransducer &that)
 Copy-constructor. Throws an exception to prevent models from being copied by value. More...
 
virtual ~GTransducer ()
 
virtual bool canImplicitlyHandleContinuousFeatures ()
 Returns true iff this algorithm can implicitly handle continuous features. If it cannot, then the GDiscretize transform will be used to convert continuous features to nominal values before passing them to it. More...
 
virtual bool canImplicitlyHandleContinuousLabels ()
 Returns true iff this algorithm can implicitly handle continuous labels (a.k.a. regression). If it cannot, then the GDiscretize transform will be used during training to convert nominal labels to continuous values, and to convert nominal predictions back to continuous labels. More...
 
double crossValidate (const GMatrix &features, const GMatrix &labels, size_t nFolds, RepValidateCallback pCB=NULL, size_t nRep=0, void *pThis=NULL)
 Perform n-fold cross validation on pData. Returns sum-squared error. Uses trainAndTest for each fold. pCB is an optional callback method for reporting intermediate stats. It can be NULL if you don't want intermediate reporting. nRep is just the rep number that will be passed to the callback. pThis is just a pointer that will be passed to the callback for you to use however you want. It doesn't affect this method. More...
 
GTransduceroperator= (const GTransducer &other)
 Throws an exception to prevent models from being copied by value. More...
 
GRandrand ()
 Returns a reference to the random number generator associated with this object. For example, you could use it to change the random seed, to make this algorithm behave differently. This might be important, for example, in an ensemble of learners. More...
 
double repValidate (const GMatrix &features, const GMatrix &labels, size_t reps, size_t nFolds, RepValidateCallback pCB=NULL, void *pThis=NULL)
 Perform cross validation "nReps" times and return the average score. pCB is an optional callback method for reporting intermediate stats It can be NULL if you don't want intermediate reporting. pThis is just a pointer that will be passed to the callback for you to use however you want. It doesn't affect this method. More...
 
GMatrixtransduce (const GMatrix &features1, const GMatrix &labels1, const GMatrix &features2)
 Predicts a set of labels to correspond with features2, such that these labels will be consistent with the patterns exhibited by features1 and labels1. More...
 
void transductiveConfusionMatrix (const GMatrix &trainFeatures, const GMatrix &trainLabels, const GMatrix &testFeatures, const GMatrix &testLabels, std::vector< GMatrix * > &stats)
 Makes a confusion matrix for a transduction algorithm. More...
 

Static Public Member Functions

static GNeuralNetfourier (GMatrix &series, double period=1.0)
 Generate a neural network that is initialized with the Fourier transform to reconstruct the given time-series data. The number of rows in the given time-series data is expected to be a power of 2. The resulting neural network will accept one input, representing time. The outputs will match the number of columns in the given time-series data. The series is assumed to represent one period of time in a repeating cycle. The duration of this period is specified as the parameter, period. The returned network has already had beginIncrementalLearning called. More...
 
static void test ()
 Performs unit tests for this class. Throws an exception if there is a failure. More...
 
- Static Public Member Functions inherited from GClasses::GSupervisedLearner
static void test ()
 Runs some unit tests related to supervised learning. Throws an exception if any problems are found. More...
 

Protected Member Functions

virtual void beginIncrementalLearningInner (const GRelation &featureRel, const GRelation &labelRel)
 See the comment for GIncrementalLearner::beginIncrementalLearningInner. More...
 
virtual void trainInner (const GMatrix &features, const GMatrix &labels)
 See the comment for GSupervisedLearner::trainInner. More...
 
double validationSquaredError (const GMatrix &features, const GMatrix &labels)
 Measures the sum squared error against the specified dataset. More...
 
- Protected Member Functions inherited from GClasses::GSupervisedLearner
GDomNodebaseDomNode (GDom *pDoc, const char *szClassName) const
 Child classes should use this in their implementation of serialize. More...
 
size_t precisionRecallContinuous (GPrediction *pOutput, double *pFunc, GMatrix &trainFeatures, GMatrix &trainLabels, GMatrix &testFeatures, GMatrix &testLabels, size_t label)
 This is a helper method used by precisionRecall. More...
 
size_t precisionRecallNominal (GPrediction *pOutput, double *pFunc, GMatrix &trainFeatures, GMatrix &trainLabels, GMatrix &testFeatures, GMatrix &testLabels, size_t label, int value)
 This is a helper method used by precisionRecall. More...
 
void setupFilters (const GMatrix &features, const GMatrix &labels)
 This method determines which data filters (normalize, discretize, and/or nominal-to-cat) are needed and trains them. More...
 
virtual GMatrixtransduceInner (const GMatrix &features1, const GMatrix &labels1, const GMatrix &features2)
 See GTransducer::transduce. More...
 

Protected Attributes

size_t m_epochsPerValidationCheck
 
std::vector< GNeuralNetLayer * > m_layers
 
double m_learningRate
 
double m_minImprovement
 
double m_momentum
 
bool m_useInputBias
 
double m_validationPortion
 
- Protected Attributes inherited from GClasses::GSupervisedLearner
GRelationm_pRelFeatures
 
GRelationm_pRelLabels
 
- Protected Attributes inherited from GClasses::GTransducer
GRand m_rand
 

Detailed Description

A feed-forward artificial neural network, or multi-layer perceptron.

Constructor & Destructor Documentation

GClasses::GNeuralNet::GNeuralNet ( )
GClasses::GNeuralNet::GNeuralNet ( GDomNode pNode,
GLearnerLoader ll 
)

Load from a text-format.

virtual GClasses::GNeuralNet::~GNeuralNet ( )
virtual

Member Function Documentation

void GClasses::GNeuralNet::addLayer ( GNeuralNetLayer pLayer,
size_t  position = INVALID_INDEX 
)

Adds pLayer to the network at the specified position. (The default position is at the end in feed-forward order.) Takes ownership of pLayer. If the number of inputs and/or outputs do not align with the previous and/or next layers, then any layers with FLEXIBLE_SIZE inputs or FLEXIBLE_SIZE outputs will be resized to accomodate. If both layers have fixed sizes that do not align, then an exception will be thrown.

void GClasses::GNeuralNet::align ( const GNeuralNet that)

Swaps nodes in hidden layers of this neural network to align with those in that neural network, as determined using bipartite matching. (This might be done, for example, before averaging weights together.)

void GClasses::GNeuralNet::autoTune ( GMatrix features,
GMatrix labels 
)

Uses cross-validation to find a set of parameters that works well with the provided data. That is, this method will add a good number of hidden layers, pick a good momentum value, etc.

void GClasses::GNeuralNet::backpropagate ( const double *  pTarget,
size_t  startLayer = INVALID_INDEX 
)

This method assumes that the error term is already set at every unit in the output layer. It uses back-propagation to compute the error term at every hidden unit. (It does not update any weights.)

void GClasses::GNeuralNet::backpropagateSingleOutput ( size_t  outputNode,
double  target,
size_t  startLayer = INVALID_INDEX 
)

Backpropagates error from a single output node over all of the hidden layers. (Assumes the error term is already set on the specified output node.)

virtual void GClasses::GNeuralNet::beginIncrementalLearningInner ( const GRelation featureRel,
const GRelation labelRel 
)
protectedvirtual
void GClasses::GNeuralNet::bleedWeightsL1 ( double  beta)

Adjust the magnitudes of the incoming and outgoing connections by amount beta, such that sum of the absolute values of the weights remains constant. If beta is 0, no bleeding will occur. If beta is 1, total bleeding will occur, such that the sum of the absolute values of the input and output weights are the same. Typically, only a small amount of bleeding is desirable, so values close to 0 (like 0.00001) are used.

void GClasses::GNeuralNet::bleedWeightsL2 ( double  beta)

Adjust the magnitudes of the incoming and outgoing connections by amount beta, such that sum-squared-weights remains constant. If beta is 0, no bleeding will occur. If beta is 1, total bleeding will occur, such that the sum of the squares of the input and output weights are the same. Typically, only a small amount of bleeding is desirable, so values close to 0 (like 0.00001) are used.

virtual bool GClasses::GNeuralNet::canImplicitlyHandleMissingFeatures ( )
inlinevirtual
virtual bool GClasses::GNeuralNet::canImplicitlyHandleNominalFeatures ( )
inlinevirtual
virtual bool GClasses::GNeuralNet::canImplicitlyHandleNominalLabels ( )
inlinevirtual

See the comment for GTransducer::canImplicitlyHandleNominalLabels.

Reimplemented from GClasses::GTransducer.

virtual void GClasses::GNeuralNet::clear ( )
virtual

See the comment for GSupervisedLearner::clear.

Implements GClasses::GSupervisedLearner.

GMatrix* GClasses::GNeuralNet::compressFeatures ( GMatrix features)

Performs principal component analysis (without reducing dimensionality) on the features to shift the variance of the data to the first few columns. Adjusts the weights on the input layer accordingly, such that the network output remains the same. Returns the transformed feature matrix.

void GClasses::GNeuralNet::containIntrinsics ( GMatrix intrinsics)

Finds the column in the intrinsic matrix with the largest deviation, then centers the matrix at the origin and renormalizes so the largest deviation is 1. Also renormalizes the input layer so these changes will have no effect.

void GClasses::GNeuralNet::contractWeights ( double  factor,
bool  contractBiases 
)

Contract all the weights in this network by the specified factor.

void GClasses::GNeuralNet::copyPrediction ( double *  pOut)

This method assumes forwardProp has been called. It copies the predicted vector into pOut.

void GClasses::GNeuralNet::copyStructure ( GNeuralNet pOther)

Makes this neural network into a deep copy of pOther, including layers, nodes, settings and weights.

void GClasses::GNeuralNet::copyWeights ( GNeuralNet pOther)

Copy the weights from pOther. It is assumed (but not checked) that pOther already has the same network structure as this neural network. This method is faster than copyStructure.

size_t GClasses::GNeuralNet::countWeights ( ) const

Counts the number of weights in the network. (This value is not cached, so you should cache it rather than frequently call this method.)

size_t GClasses::GNeuralNet::countWeights ( size_t  layer) const
void GClasses::GNeuralNet::descendGradient ( const double *  pFeatures,
double  learningRate,
double  momentum 
)

This method assumes that the error term is already set for every network unit (by a call to backpropagate). It adjusts weights to descend the gradient of the error surface with respect to the weights.

void GClasses::GNeuralNet::descendGradientSingleOutput ( size_t  outputNeuron,
const double *  pFeatures,
double  learningRate,
double  momentum 
)

This method assumes that the error term has been set for a single output network unit, and all units that feed into it transitively (by a call to backpropagateSingleOutput). It adjusts weights to descend the gradient of the error surface with respect to the weights.

void GClasses::GNeuralNet::diminishWeights ( double  amount,
bool  regularizeBiases = true,
size_t  startLayer = 0,
size_t  layerCount = INVALID_INDEX 
)

Diminishes all weights in the network by the specified amount. This can be used to implemnet L1 regularization, which promotes sparse representations. That is, it makes many of the weights approach zero.

void GClasses::GNeuralNet::forwardProp ( const double *  pInputs,
size_t  maxLayers = INVALID_INDEX 
)

Evaluates a feature vector. (The results will be in the nodes of the output layer.) The maxLayers parameter can limit how far into the network values are propagated.

double GClasses::GNeuralNet::forwardPropSingleOutput ( const double *  pInputs,
size_t  output 
)

This is the same as forwardProp, except it only propagates to a single output node. It returns the value that this node outputs. If bypassInputWeights is true, then pInputs is assumed to have the same size as the first layer, and it is fed into the net of this layer, instead of the inputs.

static GNeuralNet* GClasses::GNeuralNet::fourier ( GMatrix series,
double  period = 1.0 
)
static

Generate a neural network that is initialized with the Fourier transform to reconstruct the given time-series data. The number of rows in the given time-series data is expected to be a power of 2. The resulting neural network will accept one input, representing time. The outputs will match the number of columns in the given time-series data. The series is assumed to represent one period of time in a repeating cycle. The duration of this period is specified as the parameter, period. The returned network has already had beginIncrementalLearning called.

void GClasses::GNeuralNet::gradientOfInputs ( double *  pOutGradient)

This method assumes that the error term is already set for every network unit. It calculates the gradient with respect to the inputs. That is, it points in the direction of changing inputs that makes the error bigger. (Note that this calculation depends on the weights, so be sure to call this method before you call descendGradient. Also, note that descendGradient depends on the input features, so be sure not to update them until after you call descendGradient.)

void GClasses::GNeuralNet::gradientOfInputsSingleOutput ( size_t  outputNeuron,
double *  pOutGradient 
)

This method assumes that the error term is already set for every network unit. It calculates the gradient with respect to the inputs. That is, it points in the direction of changing inputs that makes the error bigger. This method assumes that error is computed for only one output neuron, which is specified. (Note that this calculation depends on the weights, so be sure to call this method before you call descendGradientSingleOutput.) Also, note that descendGradientSingleOutput depends on the input features, so be sure not to update them until after you call descendGradientSingleOutput.)

double GClasses::GNeuralNet::improvementThresh ( )
inline

Returns the threshold ratio for improvement.

GMatrix* GClasses::GNeuralNet::internalTraininGMatrix ( )

Gets the internal training data set.

GMatrix* GClasses::GNeuralNet::internalValidationData ( )

Gets the internal validation data set.

void GClasses::GNeuralNet::invertNode ( size_t  layer,
size_t  node 
)

Inverts the weights of the specified node, and adjusts the weights in the next layer (if there is one) such that this will have no effect on the output of the network. (Assumes this model is already trained.)

GNeuralNetLayer& GClasses::GNeuralNet::layer ( size_t  n)
inline

Returns a reference to the specified layer.

size_t GClasses::GNeuralNet::layerCount ( ) const
inline

Returns the number of layers in this neural network. These include the hidden layers and the output layer. (The input vector does not count as a layer.)

double GClasses::GNeuralNet::learningRate ( ) const
inline

Returns the current learning rate.

virtual void GClasses::GNeuralNet::maxNorm ( double  max)
virtual

Scales weights if necessary such that the manitude of the weights (not including the bias) feeding into each unit are <= max.

double GClasses::GNeuralNet::momentum ( ) const
inline

Returns the current momentum value.

GNeuralNetLayer& GClasses::GNeuralNet::outputLayer ( )
inline

Returns a reference to the last layer.

void GClasses::GNeuralNet::perturbAllWeights ( double  deviation)

Perturbs all weights in the network by a random normal offset with the specified deviation.

virtual void GClasses::GNeuralNet::predict ( const double *  pIn,
double *  pOut 
)
virtual
virtual void GClasses::GNeuralNet::predictDistribution ( const double *  pIn,
GPrediction pOut 
)
virtual
void GClasses::GNeuralNet::pretrainWithAutoencoders ( const GMatrix features,
size_t  maxLayers = INVALID_INDEX 
)

Pretrains the network using the method of stacked autoencoders. This method performs the following steps: 1- Start with the first layer. 2- Create an autoencoder using the current layer as the encoder and a temporary layer as the decoder. 3- Train the autoencoder with the features. 4- Discard the decoder. 5- Map the features through the encoder to obtain a set of features for training the next layer, and go to step 2 until all (or maxLayers) layers have been pretrained in this manner.

void GClasses::GNeuralNet::printWeights ( std::ostream &  stream)

Prints weights in a human-readable format.

GNeuralNetLayer* GClasses::GNeuralNet::releaseLayer ( size_t  index)

Drops the layer at the specified index. Returns a pointer to the layer. You are then responsible to delete it. (This doesn't resize the remaining layers to fit with each other, so the caller is responsible to repair any such issues before using the neural network again.)

void GClasses::GNeuralNet::scaleWeights ( double  factor,
bool  scaleBiases = true,
size_t  startLayer = 0,
size_t  layerCount = INVALID_INDEX 
)

Multiplies all weights in the network by the specified factor. This can be used to implement L2 regularization, which prevents weight saturation.

void GClasses::GNeuralNet::scaleWeightsSingleOutput ( size_t  output,
double  lambda 
)

Just like scaleWeights, except it only scales the weights in one of the output units.

virtual GDomNode* GClasses::GNeuralNet::serialize ( GDom pDoc) const
virtual

Saves the model to a text file.

Implements GClasses::GSupervisedLearner.

void GClasses::GNeuralNet::setImprovementThresh ( double  d)
inline

Specifies the threshold ratio for improvement that must be made since the last validation check for training to continue. (For example, if the mean squared error at the previous validation check was 50, and the mean squared error at the current validation check is 49, then training will stop if d is > 0.02.)

void GClasses::GNeuralNet::setLearningRate ( double  d)
inline

Set the learning rate.

void GClasses::GNeuralNet::setMomentum ( double  d)
inline

Momentum has the effect of speeding convergence and helping the gradient descent algorithm move past some local minimums.

void GClasses::GNeuralNet::setUseInputBias ( bool  b)
inline

Specify whether to use an input bias. (The default is false.) This feature is used with generative-backpropagation, which adjusts inputs to create latent features.

void GClasses::GNeuralNet::setValidationPortion ( double  d)
inline

Set the portion of the data that will be used for validation. If the value is 0, then all of the data is used for both training and validation.

void GClasses::GNeuralNet::setWeights ( const double *  pWeights)

Sets all the weights from an array of doubles. The number of doubles in the array can be determined by calling countWeights().

void GClasses::GNeuralNet::setWeights ( const double *  pWeights,
size_t  layer 
)
void GClasses::GNeuralNet::setWindowSize ( size_t  n)
inline

Sets the number of epochs that will be performed before each time the network is tested again with the validation set to determine if we have a better best-set of weights, and whether or not it's achieved the termination condition yet. (An epochs is defined as a single pass through all rows in the training set.)

double GClasses::GNeuralNet::sumSquaredPredictionError ( const double *  pTarget)

This method assumes forwardProp has been called. It computes the sum squared prediction error with the specified target vector.

virtual bool GClasses::GNeuralNet::supportedFeatureRange ( double *  pOutMin,
double *  pOutMax 
)
virtual

See the comment for GTransducer::supportedFeatureRange.

Reimplemented from GClasses::GTransducer.

virtual bool GClasses::GNeuralNet::supportedLabelRange ( double *  pOutMin,
double *  pOutMax 
)
virtual

See the comment for GTransducer::supportedFeatureRange.

Reimplemented from GClasses::GTransducer.

void GClasses::GNeuralNet::swapNodes ( size_t  layer,
size_t  a,
size_t  b 
)

Swaps two nodes in the specified layer. If layer specifies one of the hidden layers, then this will have no net effect on the output of the network. (Assumes this model is already trained.)

static void GClasses::GNeuralNet::test ( )
static

Performs unit tests for this class. Throws an exception if there is a failure.

virtual void GClasses::GNeuralNet::trainIncremental ( const double *  pIn,
const double *  pOut 
)
virtual
void GClasses::GNeuralNet::trainIncrementalWithDropConnect ( const double *  pIn,
const double *  pOut,
double  probOfDrop 
)

Presents a pattern for training. Applies dropConnect to the weights. Note that when training with dropConnect is complete, you should call scaleWeights(1.0 - probOfDrop, false) to compensate for the scaling effect dropConnect has on the weights.

void GClasses::GNeuralNet::trainIncrementalWithDropout ( const double *  pIn,
const double *  pOut,
double  probOfDrop 
)

Presents a pattern for training. Applies dropout to the activations of hidden layers. Note that when training with dropout is complete, you should call scaleWeights(1.0 - probOfDrop, false, 1) to compensate for the scaling effect dropout has on the weights.

virtual void GClasses::GNeuralNet::trainInner ( const GMatrix features,
const GMatrix labels 
)
protectedvirtual
virtual void GClasses::GNeuralNet::trainSparse ( GSparseMatrix features,
GMatrix labels 
)
virtual

See the comment for GIncrementalLearner::trainSparse Assumes all attributes are continuous.

Implements GClasses::GIncrementalLearner.

size_t GClasses::GNeuralNet::trainWithValidation ( const GMatrix trainFeatures,
const GMatrix trainLabels,
const GMatrix validateFeatures,
const GMatrix validateLabels 
)

Train the network until the termination condition is met. Returns the number of epochs required to train it.

bool GClasses::GNeuralNet::useInputBias ( ) const
inline

Returns whether this neural network utilizes an input bias.

double GClasses::GNeuralNet::validationSquaredError ( const GMatrix features,
const GMatrix labels 
)
protected

Measures the sum squared error against the specified dataset.

void GClasses::GNeuralNet::weights ( double *  pOutWeights) const

Serializes the network weights into an array of doubles. The number of doubles in the array can be determined by calling countWeights().

void GClasses::GNeuralNet::weights ( double *  pOutWeights,
size_t  layer 
) const
size_t GClasses::GNeuralNet::windowSize ( )
inline

Returns the number of epochs to perform before the validation data is evaluated to see if training should stop.

Member Data Documentation

size_t GClasses::GNeuralNet::m_epochsPerValidationCheck
protected
std::vector<GNeuralNetLayer*> GClasses::GNeuralNet::m_layers
protected
double GClasses::GNeuralNet::m_learningRate
protected
double GClasses::GNeuralNet::m_minImprovement
protected
double GClasses::GNeuralNet::m_momentum
protected
bool GClasses::GNeuralNet::m_useInputBias
protected
double GClasses::GNeuralNet::m_validationPortion
protected