Skip to content

Mailing list migration

Jump to bottom Edit New page
Eric Larson edited this page Dec 10, 2020 · 17 revisions

Following:

1. In Ubuntu host

git clone https://github.com/discourse/discourse.git
cd discourse
d/boot_dev --init
d/bundle install
d/rails db:migrate RAILS_ENV=development
d/shell

2. In docker shell

# follow the import/list_name hierarchy from
# https://meta.discourse.org/t/importing-mailing-lists-mbox-listserv-google-groups-emails/79773#1-5-prepare-files

cd ~
mkdir -p import/data
wget -r -l1 --no-parent --no-directories "https://mail.nmr.mgh.harvard.edu/pipermail//mne_analysis/" -P ./import/data/mne_analysis -A "*-*.txt.gz"
gzip -d ./import/data/mne_analysis/*.txt.gz
wget https://gist.githubusercontent.com/larsoner/940cd6c7100b87c4c5668cb0bc540afb/raw/9e78513620d11355ad0e10f4a2470996c26ebc8c/mailmanToMBox.py
python3 mailmanToMBox.py import/data/mne_analysis/
rm import/data/mne_analysis/*.txt

# check results
cat import/data/mne_analysis/*.mbox > all.mbox
sudo apt install -y procmail
mkdir -p split
export FILENO=0000
formail -ds sh -c 'cat > split/msg.$FILENO' < all.mbox
rm -rf split all.mbox

# settings
wget https://raw.githubusercontent.com/discourse/discourse/master/script/import_scripts/mbox/settings.yml -O import/settings.yml

All good so far!

Error!

Then we get an error in the shell trying to do:

ruby /src/script/import_scripts/mbox.rb ~/import/settings.yml

discourse@discourse:~$ ruby /src/script/import_scripts/mbox.rb ~/import/settings.yml
Traceback (most recent call last):
	31: from /src/script/import_scripts/mbox.rb:9:in `<main>'
	30: from /src/script/import_scripts/mbox.rb:10:in `<module:ImportScripts>'
	29: from /src/script/import_scripts/mbox.rb:11:in `<module:Mbox>'
	28: from /src/script/import_scripts/mbox.rb:11:in `require_relative'
	27: from /src/script/import_scripts/mbox/importer.rb:3:in `<top (required)>'
	26: from /src/script/import_scripts/mbox/importer.rb:3:in `require_relative'
	25: from /src/script/import_scripts/base.rb:16:in `<top (required)>'
	24: from /src/script/import_scripts/base.rb:16:in `require_relative'
	23: from /src/config/environment.rb:4:in `<top (required)>'
	22: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	21: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	20: from /src/config/application.rb:70:in `<top (required)>'
	19: from /src/config/application.rb:71:in `<module:Discourse>'
	18: from /src/config/application.rb:87:in `<class:Application>'
	17: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:280:in `require_dependency'
	16: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:86:in `depend_on'
	15: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:367:in `depend_on'
	14: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:48:in `require_or_load'
	13: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:17:in `allow_bootsnap_retry'
	12: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:49:in `block in require_or_load'
	11: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:389:in `require_or_load'
	10: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:40:in `load_interlock'
	 9: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:13:in `loading'
	 8: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
	 7: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
	 6: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:40:in `block in load_interlock'
	 5: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:406:in `block in require_or_load'
	 4: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:509:in `load_file'
	 3: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:701:in `new_constants_in'
	 2: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:510:in `block in load_file'
	 1: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:77:in `load'
/usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:77:in `load': No such file to load -- lib/discourse.rb (LoadError)
	34: from /src/script/import_scripts/mbox.rb:9:in `<main>'
	33: from /src/script/import_scripts/mbox.rb:10:in `<module:ImportScripts>'
	32: from /src/script/import_scripts/mbox.rb:11:in `<module:Mbox>'
	31: from /src/script/import_scripts/mbox.rb:11:in `require_relative'
	30: from /src/script/import_scripts/mbox/importer.rb:3:in `<top (required)>'
	29: from /src/script/import_scripts/mbox/importer.rb:3:in `require_relative'
	28: from /src/script/import_scripts/base.rb:16:in `<top (required)>'
	27: from /src/script/import_scripts/base.rb:16:in `require_relative'
	26: from /src/config/environment.rb:4:in `<top (required)>'
	25: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	24: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	23: from /src/config/application.rb:70:in `<top (required)>'
	22: from /src/config/application.rb:71:in `<module:Discourse>'
	21: from /src/config/application.rb:87:in `<class:Application>'
	20: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:280:in `require_dependency'
	19: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:85:in `depend_on'
	18: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:93:in `rescue in depend_on'
	17: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:9:in `without_bootsnap_cache'
	16: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:93:in `block in depend_on'
	15: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:367:in `depend_on'
	14: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:48:in `require_or_load'
	13: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:17:in `allow_bootsnap_retry'
	12: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/active_support.rb:49:in `block in require_or_load'
	11: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:389:in `require_or_load'
	10: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:40:in `load_interlock'
	 9: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:13:in `loading'
	 8: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
	 7: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
	 6: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:40:in `block in load_interlock'
	 5: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:406:in `block in require_or_load'
	 4: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:509:in `load_file'
	 3: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:701:in `new_constants_in'
	 2: from /usr/local/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:510:in `block in load_file'
	 1: from /usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:77:in `load'
/usr/local/lib/ruby/gems/2.7.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:77:in `load': No such file to load -- lib/discourse.rb (LoadError)

Similar error when using a command even more like the one from the YAML:

bundle exec ruby /src/script/import_scripts/mbox.rb /home/discourse/import/settings.yml

Traceback (most recent call last):
	11: from /src/script/import_scripts/mbox.rb:9:in `<main>'
	10: from /src/script/import_scripts/mbox.rb:10:in `<module:ImportScripts>'
	 9: from /src/script/import_scripts/mbox.rb:11:in `<module:Mbox>'
	 8: from /src/script/import_scripts/mbox.rb:11:in `require_relative'
	 7: from /src/script/import_scripts/mbox/importer.rb:3:in `<top (required)>'
	 6: from /src/script/import_scripts/mbox/importer.rb:3:in `require_relative'
	 5: from /src/script/import_scripts/base.rb:16:in `<top (required)>'
	 4: from /src/script/import_scripts/base.rb:16:in `require_relative'
	 3: from /src/config/environment.rb:4:in `<top (required)>'
	 2: from /src/config/environment.rb:4:in `require'
	 1: from /src/config/application.rb:17:in `<top (required)>'
/src/config/application.rb:17:in `require': cannot load such file -- active_record/railtie (LoadError)
# 3. In Ubuntu host (hopefully) 
d/unicorn
google-chrome http://0.0.0.0:9292

Done!

Add a custom footer
Add a custom sidebar
Clone this wiki locally