# Copyright 2015 Yin Zheng, Yu-Jin Zhang, Hugo Larochelle. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification, are
# permitted provided that the following conditions are met:
#
#    1. Redistributions of source code must retain the above copyright notice, this list of
#       conditions and the following disclaimer.
#
#    2. Redistributions in binary form must reproduce the above copyright notice, this list
#       of conditions and the following disclaimer in the documentation and/or other materials
#       provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY Yin Zheng, Yu-Jin Zhang, Hugo Larochelle ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
# Yin Zheng, Yu-Jin Zhang, Hugo Larochelle OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are those of the
# authors and should not be interpreted as representing official policies, either expressed
# or implied, of Yin Zheng, Yu-Jin Zhang, Hugo Larochelle.


		
'''
@reference: A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data, IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI)

			Project page: https://sites.google.com/site/zhengyin1126/home/supdeepdocnade

@Authors:     Yin Zheng, Received Ph.D from Tsinghua University, 2015. Homepage: https://sites.google.com/site/zhengyin1126/
              Yu-Jin Zhang, Tsinghua University,
              Hugo Larochelle, University of Sherbrooke and Twitter. Homepage: http://www.dmi.usherb.ca/~larocheh/index_en.html

@contact: Yin Zheng, [email protected]
          
'''


We provide:
1. SupDeepDocNADE.py : Classes of DeepDocNADE and SupDeepDocNADE, which is used by run_pretrain_DeepDocNADE.py and run_SupDeepDocNADE.py
2. run_pretrain_DeepDocNADE.py : Python script to pretrain DeepDocNADE model on MIR Flickr Unlabeled data
3. run_SupDeepDocNADE.py : Python script to train and test SupDeepDocNADE mdoel.
4. gen_dataset_labeled.py: Python script to generate the labeled dateset files needed by the model
5. gen_dataset_unlabeled.py: Python script to generate the unlabeled dateset files needed by the model
6. MIR_Flickr_Theano_lab.py: Python script to load labeled dataset and provide interface to SupDeepDocNADE model
7. MIR_Flickr_Theano_Unlab.py: Python script to load unlabeled dataset and provide interface to DeepDocNADE model

===================================================================================================================== 	

0. Install liblinear (http://www.csie.ntu.edu.tw/~cjlin/liblinear/) and put liblinear into the PYTHONPATH
	NOTE: make sure that you can use liblinear in python.  To test whether you config liblinear successfully, you could
	use "from liblinearutil import *" in python.

1. Generate the dataset:
	1) Download the dataset from Nitsh's homepage: http://www.cs.toronto.edu/~nitish/multimodal/index.html
	2) Run "python gen_dataset_labeled.py" to generate the labeled dataset. 
		a) Read the comments in the script about how to use it.
		b) create a file IN THE SAME PATH OF THE LABELED DATASET named "meta.txt" with the following lines in the meta.txt
			voc_size:2000
			n_regions:14
			text_voc_size:2000
			global_feat_size:1857
			n_classes:38
		c) Create a file IN THE SAME PATH OF THE LABELED DATASET named "sizes.txt" with the following content:
			10000
			5000
			10000
	
	3) Run "python gen_dataset_unlabeled.py" to generate the unlabeled dataset. 
		a) Read the comments in the script about how to use it.
		b) create a file IN THE SAME PATH OF THE UNLABELED DATASET with the name meta.txt, the content is:
			voc_size:2000
			n_regions:14
			text_voc_size:2000
			global_feat_size:1857
			dataset_split:50
		c) create a file IN THE SAME PATH OF THE UNLABELED DATASET named "sizes.txt" with the following content:
			975000
			
		
			
2. Run run_pretrain_DeepDocNADE.py to pretrain DeepDocNADE model on unlabeled data:
		python run_pretrain_DeepDocNADE.py n_pretrain pre_learning_rate hidden_size activation_function  dropout_rate model_file_dir unlab_dataset_dir batch_size anno_weight  polyakexp_weight model_init
	

	The parameters of the script is as follows:
	
		n_pretrain : number of iterations
		pre_learning_rate : learning rate of the pretraining 
		hidden_size : the hidden size of the model, e.g. 2048_2048_2048 is a 3 hidden layers model with 2048 units each layer
		activation_function : the activation function of the hidden layers, "sigmoid, relu or tanh" 
		dropout_rate: the dropout rate for each hidden layer, e.g. "0.5_0.5_0.5" means dropout rate is 0.5 for each layer
		model_file_dir: path to save the pretrained model
		unlab_dataset_dir: the path to the unlabeled dataset 
		batch_size: the batch size 
		anno_weight: annotation weight 
		polyakexp_weight: polyak weight
		model_init: path to the saved pretrain model, which is used to continue_training based on the trained model. It could be NULL if no model saved before.
	
	One example of the scipt is: 
		python run_pretrain_DeepDocNADE.py 6000 0.03 2048_2048_2048 reclin 0.5_0.5_0.5 PATH_TO_SAVE_THE_MODEL PATH_TO_UNLABELED_DATA 500 12000 std 0.9995 PATH_TO_SAVED_MODEL


3. Run run_SupDeepDocNADE.py to train SupDeepDocNADE model on labeled data, which could be trained from scratch or initialize from pretrained model on unlabeled data :
		python run_SupDeepDocNADE.py folder_ID use_pretrain max_iter look_ahead hidden_size learning_rate unsup_weight activation_function Linear_minC, Linear_maxC, dropout_rate uniresult_dir Pretrain_model_name lab_dataset_dir batch_size anno_weight polyakexp_weight
	
	The parameters are as follows:
		folder_ID : ID of the dataset ( 1 to 5 )
		use_pretrain: Whether use pretrained model or training from randomly initialized parameters (True or False)
		max_iter: number of max iterations
		look_ahead: early stop if number of iterations without improvement exceed the number of look ahead
		hidden_size: the hidden size of the model, e.g. 2048_2048_2048 is a 3 hidden layers model with 2048 units each layer
		learning_rate: learning rate of training process
		unsup_weight: the weight of the unsupervised part 
		activation_function: the activation function of the hidden layers, "sigmoid, relu or tanh" 
		Linear_minC: the minimum value of C for linear SVM (in log_2 space) 
		Linear_maxC: the max value of C for linear SVM (in log_2 space) 
		dropout_rate: the dropout rate for each hidden layer, e.g. "0.5_0.5_0.5" means dropout rate is 0.5 for each layer
		uniresult_dir: the path to save the results
		Pretrain_model_name: the path and the name of the pretrained model if you set use_pretrain True. Otherwise, any string
		lab_dataset_dir: the path to the labeled dataset
		batch_size : the mini-batch size
		anno_weight: the weight of the annotation words
		polyakexp_weight: polyak weight
	
	One example of the script is: 
		
		python run_SupDeepDocNADE.py 1 True 20000 300 2048_2048_2048 0.01 0.25 reclin -17 10 0.5_0.5_0.5 PATH_TO_SAVE_RESULTS PATH_NAME_OF_PRETRAINED_MODEL PATH_TO_LABELED_DATA 500 12000  0.9995