WordNet: A Toy Example for Deep Learning

This project is based on the second programming assignment of Hinton's Coursera course Neural Networks for Machine Learning. The original code is written in MATLAB/Octave. Since I find this example elegent for showing how deep learning works, I rewrite the code with C from scratch and call it WordNet (without Hinton's permission). For such a teaching purpose, the code doesn't have a GPU mode or a good performance in terms of memory or CPU usage, and currently I have no plan for optimization.

Basically, WordNet reads three consecutive words and predict the fourth word. The layout of WordNet is defined as

For more technical details, please read Slides.pdf.

Building the project is quite simple since it only relies on the standard C libs. If your building tool chain is properly configured, you may simply modify the following lines in src/makefile with your compiler and header include path:

CC = clang
INCLUDES = -I/usr/local/opt/llvm/include/c++/v1 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include

and type

make forward load=model9-3000.bin

to interact with a console UI doing inference with the pre-trained model model9-3000.bin. If everything goes well, you'll see

> make forward load=model9-3000.bin 
./wordnet forward model9-3000.bin
# Load all data
# Load model: model9-3000.bin
## Model Info
   Mini-batch size          =      100
   Layer 1 Neurons          =       50
   Layer 2 Neurons          =      200
   Training epochs          =        9
   Early stop @ iteration   =     3000
   Momentum                 = 0.900000
   Learning rate            = 0.100000
   Verify per iteration     = 2147483647
   Raw training data rows   =   372550
   Raw validation data rows =    46568
   Raw test data rows       =    46568
   Raw data columns         =        4
   Input dimension          =        3
   Vocabulary size          =      250
##------Interactive UI------##
[... here lists the vocabulary.]
|Input first 3 words > have a good 
have a good 
*Top 5 = 1.time(0.332076) 2.day(0.102091) 3.game(0.059815) 4.team(0.057552) 5.year(0.041762) 
|Choose a number (default = 1)>

Of course, this project also covers codes for training:

Usage:
  ./wordnet info model.bin               | Show info of pretrained model.
  ./wordnet train model.bin              | Train from scratch and save model.
  ./wordnet train pretrain.bin model.bin | Read pretrained data, finetune it, & save model.
  ./wordnet forward pretrain.bin         | Read pretrained data and do inferences.
Or
  make info load=model.bin
  make train save=model.bin
  make train load=pretrain.bin save=model.bin
  make forward load=pretrain.bin

Enjoy!