Add RGB intensity regularization

[hma02]

Add RGB regularization on image pixels to make the Neural Network invariant to changes in the intensity and color of the illumination. First, run python get_cov_matrix.py to generate the RGB covariance matrix file and RGB mean file based on all training images. Then use the new proc_load.py during training, which will load the covariance matrix and add RGB regularization to each image before cropping.

Here's a sample result of the the RGB covariance matrix generated from 1.2 million training images:

[[RR RG RB], [GR GG GB], [BR BG BB]]=
[[ 5092.20548764 4433.19480634 3994.61330229]
[ 4433.19480634 4832.88253809 4575.09578822]
[ 3994.61330229 4575.09578822 5332.67691264]]

This is related to issue #15

[gwding]

indentation doesn't seem right

[gwding]

as a general guideline, take a look at python pep 8 and see the auto-checking/formatting tool in your text editor, to keep the code format nice.

[gwding]

could you also report the accuracy after adding the RGB augmentation?

[hma02]

I just finished one testing of the RGB regularization. The result shows that when using it in training and not using it in validation,
validation error 43.575721 %
top 5 validation error 20.512821 %
validation loss 1.884180

When using it in both training and validation,
validation error 43.657853 %
top 5 validation error 20.594952 %
validation loss 1.890634
This is when the Gaussian random variable alpha is drawn with standard deviation = 0.01.

The alpha with standard deviation = 0.1 is being trained. So far it seems not converge as good as standard deviation = 0.01.

[gwding]

@hma02 any updates here? it seems I missed your last comment. so that means the augmentation doesn't bring any improvement then?

[hma02]

@gwding Sorry. I just checked the job. It seems the RGB augmentation doesn't improve much. Using alpha=0.1 even makes the error higher:

validation error 55.723157 %
top 5 validation error 31.794872 %
validation loss 2.570946

I'm also confused. Maybe I did something wrong somewhere.

[gwding]

@hma02 any updates on this? is it still under development?