Repackaging training data

[kristinbranson]

Is it possible to avoid repackaging all the training data every time one hits train?

[allenleetc]

Yes should be. I thought we had some re-use/caching in there but probably not as optimized as it could be. Will check on it (caught a bug and am out sick atm).

What are you changing/toggling between successive Trains, just hyperparameters? Or are you adding labels etc?

[kristinbranson]

I don't think I changed anything. Just had training crash the first time I tried it, and was trying it a second time. But, something could have gone very wrong in between those times. In more detail:
From when Roian last trained the tracker, I added a few labels and changed the maximum number of animals to 5 (was 2). Then, I hit train. It did ask me a lot of questions that I didn't totally understand about wanting to align stuff and flipping stuff and I think I said to go ahead and do whatever parameter changes it wanted. It regenerated all images for training from scratch. It crashed during training (please see info on Slack). I tried to train again after that. There should be a history of what trackers were stored in the printInfo I put on Slack.