Checking Size of Discogs Datadumps

[ijabz]

Checking Size of Discogs Datadumps to see if the new datadumps are valid, also see https://android-developer.ro/projects/the_ogger_club/discogs/data_dumps/count/artists/

Main differences between the 20240701 dump and the 20241201 and 20250101 dumps is the later dumps no longer contain image info, and unknked artists no longer have a value for id beforehand it was set to zero

Artist and Release table counts have increased in later dumps but not by as much as expected.

Artist count in 202501 is smaller than 202412 even though release count is greater and the same grep fizes were applied to both dumps.

20240701

10351307 artist.csv
4379583 artist_alias.csv
2524651 artist_image.csv
4409972 artist_namevariation.csv
1896194 artist_url.csv
2072004 group_member.csv
2962934 label.csv
573615 label_image.csv
349301 label_url.csv
2335074 master.csv
2856745 master_artist.csv
3088743 master_genre.csv
8021194 master_image.csv
3591634 master_style.csv
32591720 master_video.csv
45959745 release.csv
84406768 release_artist.csv
29913721 release_company.csv
17814473 release_format.csv
23120611 release_genre.csv
29673180 release_identifier.csv
51767290 release_image.csv
20146959 release_label.csv
25876614 release_style.csv
161025798 release_track.csv
113340558 release_track_artist.csv
266611731 release_video.csv

Database Counts

discogs.artist;
9187524

discogs.release;
17402068

discogs.release_artist;
84406767

discogs.release_track_artist;
113340556

discogs.release_track;
161025765

20241201

Csv Files Before Fixes
wc -l
10362512 artist.csv
4401636 artist_alias.csv
1 artist_image.csv
4479541 artist_namevariation.csv
1902052 artist_url.csv
2100891 group_member.csv
3026807 label.csv
1 label_image.csv
356840 label_url.csv
2377785 master.csv
2911535 master_artist.csv
3147898 master_genre.csv
1 master_image.csv
3666572 master_style.csv
35436333 master_video.csv
47113536 release.csv
86369236 release_artist.csv
30864871 release_company.csv
18204253 release_format.csv
23633892 release_genre.csv
30563955 release_identifier.csv
1 release_image.csv
20598734 release_label.csv
26481471 release_style.csv
164560073 release_track.csv
116040412 release_track_artist.csv
63771608 release_video.csv
702372447 total

Csv Files after Fixes
wc -l *.csv
10362511 artist.csv
4401636 artist_alias.csv
1 artist_image.csv
4479541 artist_namevariation.csv
1902052 artist_url.csv
2100891 group_member.csv
3026807 label.csv
1 label_image.csv
356840 label_url.csv
2377785 master.csv
2911407 master_artist.csv
3147898 master_genre.csv
1 master_image.csv
3666572 master_style.csv
35436333 master_video.csv
47113536 release.csv
85643096 release_artist.csv
30862475 release_company.csv
18204253 release_format.csv
23633892 release_genre.csv
30563955 release_identifier.csv
1 release_image.csv
20598734 release_label.csv
26481471 release_style.csv
164560073 release_track.csv
115071423 release_track_artist.csv
63771608 release_video.csv
700674793 total

Database Count

discogs.artist;
9197873

discogs.release;
17780660

discogs.release_artist;
85643095

discogs.release_track_artist;
115071421

discogs.release_track;
164560039

20250101

** CsvFiles before fixes**

10359398 artist.csv
4404696 artist_alias.csv
1 artist_image.csv
4496106 artist_namevariation.csv
1901920 artist_url.csv
2107228 group_member.csv
3032306 label.csv
1 label_image.csv
357573 label_url.csv
2388271 master.csv
2924727 master_artist.csv
3162515 master_genre.csv
1 master_image.csv
3685264 master_style.csv
35878616 master_video.csv
47361156 release.csv
86773174 release_artist.csv
31071930 release_company.csv
18283990 release_format.csv
23739953 release_genre.csv
30745341 release_identifier.csv
1 release_image.csv
20691679 release_label.csv
26608544 release_style.csv
165327677 release_track.csv
116628041 release_track_artist.csv
64523640 release_video.csv

CsvFiles (after fixes to remove invalid entries that prevent import into db)
wc -l
10359397 artist.csv
4404696 artist_alias.csv
1 artist_image.csv
4496106 artist_namevariation.csv
1901920 artist_url.csv
2107228 group_member.csv
3032305 label.csv
1 label_image.csv
357573 label_url.csv
2388271 master.csv
2924599 master_artist.csv
3162515 master_genre.csv
1 master_image.csv
3685264 master_style.csv
35878616 master_video.csv
47361156 release.csv
86041226 release_artist.csv
31069464 release_company.csv
18283990 release_format.csv
23739953 release_genre.csv
30745341 release_identifier.csv
1 release_image.csv
20691679 release_label.csv
26608544 release_style.csv
165327677 release_track.csv
115649401 release_track_artist.csv
64523640 release_video.csv

discogs.artist;
9194912

discogs.release;
17857796

discogs.release_artist;
86041225

discogs.release_track_artist
115649399

discogs.release_track;
165327643