shared_list does not have data_set in forward block with TIMIT tutorial

[hajime9652]

------------------------------ Epoch 23 / 23 ------------------------------
 
----- Summary epoch 23 / 23
Training on ['TIMIT_tr']
Loss = 0.932 | err = 0.298 
-----
Validating on TIMIT_dev
Loss = 1.811 | err = 0.468 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 574

 
Testing TIMIT_test chunk = 1 / 1
shared list []
shared list [None, None, None, {'mfcc': ['mfcc', 'exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0_mfcc.lst', 'apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5_0827_test/data/test/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5_0827_test/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |', '5', '5']}, {}, {'MLP_layers1': ['architecture1', 'MLP_layers1', 0]}, {'input': None, 'ref': None}]
output folder exp/TIMIT_MLP_basic
data_set_dict <class 'dict'>
data_set_dict {'input': None, 'ref': None}
Traceback (most recent call last):
  File "run_exp.py", line 340, in <module>
    data_set_inp, data_set_ref = convert_numpy_to_torch(data_set_dict, save_gpumem, use_cuda)
  File "/home/sysadmin/pytorch-kaldi/core.py", line 46, in convert_numpy_to_torch
    data_set_inp=torch.from_numpy(data_set_dict['input']).float()
TypeError: expected np.ndarray (got NoneType)

[hajime9652]

# --------FORWARD--------#
for forward_data in forward_data_lst:

         # Compute the number of chunks
         N_ck_forward=compute_n_chunks(out_folder,forward_data,ep,N_ep_str_format,'forward')
         N_ck_str_format='0'+str(max(math.ceil(np.log10(N_ck_forward)),1))+'d'

         processes = list()
         info_files = list()
         for ck in range(N_ck_forward):

            if not is_production:
                print('Testing %s chunk = %i / %i' %(forward_data,ck+1, N_ck_forward))
            else:
                print('Forwarding %s chunk = %i / %i' %(forward_data,ck+1, N_ck_forward))

            # output file
            info_file=out_folder+'/exp_files/forward_'+forward_data+'_ep'+format(ep, N_ep_str_format)+'_ck'+format(ck, N_ck_str_format)+'.info'
            config_chunk_file=out_folder+'/exp_files/forward_'+forward_data+'_ep'+format(ep, N_ep_str_format)+'_ck'+format(ck, N_ck_str_format)+'.cfg'


            # Do forward if the chunk was not already processed
            if not(os.path.exists(info_file)):

                # Doing forward

                # getting the next chunk 
                next_config_file=cfg_file_list[op_counter]

                # run chunk processing                    
                if _run_forwarding_in_subprocesses(config):
                    shared_list = list()
                    print("shared list",shared_list)
                    output_folder = config['exp']['out_folder']
                    save_gpumem = strtobool(config['exp']['save_gpumem'])
                    use_cuda=strtobool(config['exp']['use_cuda'])
                    p = read_next_chunk_into_shared_list_with_subprocess(read_lab_fea, shared_list, config_chunk_file, is_production, output_folder, wait_for_process=True)
                    data_name, data_end_index_fea, data_end_index_lab, fea_dict, lab_dict, arch_dict, data_set_dict = extract_data_from_shared_list(shared_list)
                    print("shared list", shared_list)
                    print("output folder",output_folder)
                    print("data_set_dict",type(data_set_dict))
                    print("data_set_dict",data_set_dict)
                    data_set_inp, data_set_ref = convert_numpy_to_torch(data_set_dict, save_gpumem, use_cuda)

[hajime9652]

When is shared_list overwrite?
and How to bring the correct data_set?

[TParcollet]

Hi ! Isn't it simply a problem with the path of the test dataset in the config file ?

[mravanelli]

Yes, it looks like that!

…

On Thu, 29 Aug 2019 at 04:48, Parcollet Titouan ***@***.***> wrote: Hi ! Isn't it simply a problem with the path of the test dataset in the config file ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#157?email_source=notifications&email_token=AEA2ZVUTZHOAI7NUZZN4R4DQG6EM3A5CNFSM4IRZSXT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5NX6NI#issuecomment-526090037>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEA2ZVUQKVVQBV35NWHQSSLQG6EM3ANCNFSM4IRZSXTQ> .

[hajime9652]

I will check again.

[hajime9652]

I'm still in trouble.

ERROR MSG

------------------------------ Epoch 23 / 23 ------------------------------
 
----- Summary epoch 23 / 23
Training on ['TIMIT_tr']
Loss = 0.932 | err = 0.298 
-----
Validating on TIMIT_dev
Loss = 1.812 | err = 0.468 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 489

 
Testing TIMIT_test chunk = 1 / 1
config chunk file exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0.cfg
shared list [None, None, None, {'mfcc': ['mfcc', 'exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0_mfcc.lst', 'apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5/data/test/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |', '5', '5']}, {}, {'MLP_layers1': ['architecture1', 'MLP_layers1', 0]}, {'input': None, 'ref': None}]
Traceback (most recent call last):
  File "run_exp.py", line 338, in <module>
    data_set_inp, data_set_ref = convert_numpy_to_torch(data_set_dict, save_gpumem, use_cuda)
  File "/home/sysadmin/pytorch-kaldi/core.py", line 46, in convert_numpy_to_torch
    data_set_inp=torch.from_numpy(data_set_dict['input']).float()
TypeError: expected np.ndarray (got NoneType)

cfg

[dataset1]
data_name = TIMIT_tr
fea = fea_name=mfcc
        fea_lst=/home/sysadmin/kaldi/egs/timit/s5/data/train/feats.scp
        fea_opts=apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5/data/train/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
        cw_left=5
        cw_right=5
        

lab = lab_name=lab_cd
        lab_folder=/home/sysadmin/kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali
        lab_opts=ali-to-pdf
        lab_count_file=auto
        lab_data_folder=/home/sysadmin/kaldi/egs/timit/s5/data/train/
        lab_graph=/home/sysadmin/kaldi/egs/timit/s5/exp/tri3/graph
        

n_chunks = 5

[dataset2]
data_name = TIMIT_dev
fea = fea_name=mfcc
        fea_lst=/home/sysadmin/kaldi/egs/timit/s5/data/dev/feats.scp
        fea_opts=apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5/data/dev/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
        cw_left=5
        cw_right=5
        

lab = lab_name=lab_cd
        lab_folder=/home/sysadmin/kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali_dev
        lab_opts=ali-to-pdf
        lab_count_file=auto
        lab_data_folder=/home/sysadmin/kaldi/egs/timit/s5/data/dev/
        lab_graph=/home/sysadmin/kaldi/egs/timit/s5/exp/tri3/graph
        

n_chunks = 1

[dataset3]
data_name = TIMIT_test
fea = fea_name=mfcc
        fea_lst=/home/sysadmin/kaldi/egs/timit/s5/data/test/feats.scp
        fea_opts=apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5/data/test/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
        cw_left=5
        cw_right=5
        

lab = lab_name=lab_cd
        lab_folder=/home/sysadmin/kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali_test
        lab_opts=ali-to-pdf
        lab_count_file=auto
        lab_data_folder=/home/sysadmin/kaldi/egs/timit/s5/data/test/
        lab_graph=/home/sysadmin/kaldi/egs/timit/s5/exp/tri3/graph
        

n_chunks = 1

[hajime9652]

data_name, data_end_index_fea, data_end_index_lab, lab_dict, data_set_dict is None.
Especially why can not read lab_dict?

shared list [None, None, None, {'mfcc': ['mfcc', 'exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0_mfcc.lst', 'apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5/data/test/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |', '5', '5']}, {}, {'MLP_layers1': ['architecture1', 'MLP_layers1', 0]}, {'input': None, 'ref': None}]

lab_folder

$ ls /home/sysadmin/kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali_test
ali.1.gz  ali.2.gz  ali.3.gz  ali.4.gz  final.mdl  log  num_jobs  phones.txt  tree

exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0.cfg

[cfg_proto]
cfg_proto = proto/global.proto
cfg_proto_chunk = proto/global_chunk.proto

[exp]
cmd = 
run_nn_script = run_nn
out_folder = exp/TIMIT_MLP_basic
seed = 1257
use_cuda = False
multi_gpu = False
save_gpumem = False
n_epochs_tr = 24
production = False
to_do = forward
out_info = exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0.info

[batches]
batch_size_train = 128
max_seq_length_train = 1000
batch_size_valid = 128
max_seq_length_valid = 1000

[architecture1]
arch_name = MLP_layers1
arch_proto = proto/MLP.proto
arch_library = neural_networks
arch_class = MLP
arch_pretrain_file = exp/TIMIT_MLP_basic/exp_files/train_TIMIT_tr_ep23_ck4_architecture1.pkl
arch_freeze = False
arch_seq_model = False
dnn_lay = 1024,1024,1024,1024,1896
dnn_drop = 0.15,0.15,0.15,0.15,0.0
dnn_use_laynorm_inp = False
dnn_use_batchnorm_inp = False
dnn_use_batchnorm = True,True,True,True,False
dnn_use_laynorm = False,False,False,False,False
dnn_act = relu,relu,relu,relu,softmax
arch_lr = 0.08
arch_halving_factor = 0.5
arch_improvement_threshold = 0.001
arch_opt = sgd
opt_momentum = 0.0
opt_weight_decay = 0.0
opt_dampening = 0.0
opt_nesterov = False

[model]
model_proto = proto/model.proto
model = out_dnn1=compute(MLP_layers1,mfcc)
        loss_final=cost_nll(out_dnn1,lab_cd)
        err_final=cost_err(out_dnn1,lab_cd)

[forward]
forward_out = out_dnn1
normalize_posteriors = True
normalize_with_counts_from = exp/TIMIT_MLP_basic/exp_files/forward_out_dnn1_lab_cd.count
save_out_file = False
require_decoding = True

[data_chunk]
fea = fea_name=mfcc
        fea_lst=exp/TIMIT_MLP_basic/exp_files/forward_TIMIT_test_ep23_ck0_mfcc.lst
        fea_opts=apply-cmvn --utt2spk=ark:/home/sysadmin/kaldi/egs/timit/s5/data/test/utt2spk  ark:/home/sysadmin/kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
        cw_left=5
        cw_right=5
lab = lab_name=lab_cd
        lab_folder=/home/sysadmin/kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali_test
        lab_opts=ali-to-pdf
        lab_count_file=auto
        lab_data_folder=/home/sysadmin/kaldi/egs/timit/s5/data/test/
        lab_graph=/home/sysadmin/kaldi/egs/timit/s5/exp/tri3/graph

[spencerkirn]

Did you find a solution to this? I am having the exact same issue. Double checked all paths in my cfg file and the same error is occurring.

Note: I am using PyTorch-Kaldi on WSL without CUDA (still no CUDA support on WSL) not sure if this would make a difference.

[mravanelli]

It looks like and error in reading features and labels with kaldi. To debug, you can try to "manually" read the features in this way: 1- select one ark file in the /mnt/mscteach_home/s1870525/dissertation/PruninNeuralNetworksSpeech/s5/data/test_dev93/feats.scp (e.g, quick_test/fbank/raw_fbank_dev.1.ark) 2- run copy-feats ark:your_ark_file.ark ark,t:- . If everything works you should see a lot of numbers is standard output. If it doesn't work, try to take a look into the error. 3- If it works, you can add the options and you can write: copy-feats ark:your_ark.ark ark:- | apply-cmvn --utt2spk=ark:/mnt/mscteach_home/s1870525/dissertation/PruninNeuralNetworksSpeech/s5/data/test_dev93/utt2spk ark:/mnt/mscteach_home/s1870525/dissertation/PruninNeuralNetworksSpeech/s5/data/test_dev93/data/cmvn_test_dev93.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark,t:- If it doesn't work, take a look into the error message. You can also try to take a look into the log.log file you find into the output folder. Please, let me know if you are able to solve the data loading issue...

…

On Wed, 2 Oct 2019 at 09:29, spencerkirn ***@***.***> wrote: Did you find a solution to this? I am having the exact same issue. Double checked all paths in my cfg file and the same error is occurring. Note: I am using PyTorch-Kaldi on WSL without CUDA (still no CUDA support on WSL) not sure if this would make a difference. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#157?email_source=notifications&email_token=AEA2ZVTDF3BUVASYTBN5OV3QMSO3LA5CNFSM4IRZSXT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAEXV4A#issuecomment-537492208>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEA2ZVU5Z7HOD7RZI763UTDQMSO3LANCNFSM4IRZSXTQ> .

[spencerkirn]

Thank you for the quick reply. I apologize if these are basic questions, I am new to using Kaldi and this toolkit. So I ran copy-feats ark:/home/spencer/kaldi/egs/timit/s5/mfcc/raw_mfcc_dev.1.ark ark,t:- and it ran just like you said it should, with a lot of numbers output to the terminal. So after that I ran copy-feats ark:/home/spencer/kaldi/egs/timit/s5/mfcc/raw_mfcc_dev.1.ark ark:- | apply-cmvn --utt2spk=ark:/home/spencer/kaldi/egs/timit/data/dev/utt2spk ark:/home/spencer/kaldi/egs/timit/s5/data/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark, t:- and got the attached error. One thing I noticed is that there is no cmvn_dev.ark in my data folder (no .ark files at all in that folder) is that meant to be the output or should there be a .ark file there? Seems like the error is centered around that file.

[mravanelli]

Does /home/spencer/kaldi/egs/timit/s5/data/cmvn_dev.ark exists? Mirco

…

On Thu, 3 Oct 2019 at 09:24, spencerkirn ***@***.***> wrote: Thank you for the quick reply. I apologize if these are basic questions, I am new to using Kaldi and this toolkit. So I ran copy-feats ark:/home/spencer/kaldi/egs/timit/s5/mfcc/raw_mfcc_dev.1.ark ark,t:- and it ran just like you said it should, with a lot of numbers output to the terminal. So after that I ran copy-feats ark:/home/spencer/kaldi/egs/timit/s5/mfcc/raw_mfcc_dev.1.ark ark:- | apply-cmvn --utt2spk=ark:/home/spencer/kaldi/egs/timit/data/dev/utt2spk ark:/home/spencer/kaldi/egs/timit/s5/data/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark, t:- and got the attached error. One thing I noticed is that there is no cmvn_dev.ark in my data folder (no .ark files at all in that folder) is that meant to be the output or should there be a .ark file there? Seems like the error is centered around that file. [image: TIMITError] <https://user-images.githubusercontent.com/49201733/66129779-8fc35180-e5be-11e9-8b3c-d0ea6a826948.PNG> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#157?email_source=notifications&email_token=AEA2ZVRWMRDPA6HIA2ECK3TQMXXCJA5CNFSM4IRZSXT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIGC4A#issuecomment-537944432>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEA2ZVSLWWHPNLZPMECNK23QMXXCJANCNFSM4IRZSXTQ> .

[spencerkirn]

No like I said there are not .ark files in that folder (or subfolders). I thought this might be an output folder, but it looks like the issue is in the creation of those files.

[mravanelli]

This cmvn file is created by kaldi during the feature extraction phase and it performs mean and variance normalization. You should probably have the cmvn file somewhere else like in data/dev/cmvn* or mfcc/cmv* Mirco

…

On Oct 3, 2019 09:36, "spencerkirn" ***@***.***> wrote: No like I said there are not .ark files in that folder (or subfolders). I thought this might be an output folder, but it looks like the issue is in the creation of those files. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#157?email_source=notifications&email_token=AEA2ZVREGRGKV5LM674GLATQMXYODA5CNFSM4IRZSXT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIHFCY#issuecomment-537948811>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEA2ZVQZEBDVBCBQJ4SVEDDQMXYODANCNFSM4IRZSXTQ> .

[spencerkirn]

Yea I had the wrong path for cmvn file, but when I run copy-feats ark:/home/spencer/kaldi/egs/timit/s5/mfcc/raw_mfcc_test.1.ark ark,t:- | apply-cmvn --utt2spk=ark:/home/spencer/kaldi/egs/timit/s5/data/test/utt2spk ark:/home/spencer/kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- now I get a Kaldi Fatal error