You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the model raises an error if not all buckets can be filled. When training this is hardly a problem, and likewise when parsing a sequence of suitably large files. However, when the list of files to parse contains a mix of large files and small files, this causes problems--in order to parse the large files quickly without large memory consumption, you need to sort it into multiple buckets, but in order to parse files with only one or two sentences you can't use more than one or two buckets.
In order to handle a mix of large and small files, the system needs a way of setting up empty buckets.
The text was updated successfully, but these errors were encountered:
Currently, the model raises an error if not all buckets can be filled. When training this is hardly a problem, and likewise when parsing a sequence of suitably large files. However, when the list of files to parse contains a mix of large files and small files, this causes problems--in order to parse the large files quickly without large memory consumption, you need to sort it into multiple buckets, but in order to parse files with only one or two sentences you can't use more than one or two buckets.
In order to handle a mix of large and small files, the system needs a way of setting up empty buckets.
The text was updated successfully, but these errors were encountered: