Skip to content
This repository has been archived by the owner on Aug 30, 2017. It is now read-only.

Set up the Data Broker

Jump to bottom
Sameera Abeykoon edited this page Jul 23, 2014 · 41 revisions

Developers: Arman Arkilic, Daron Chabot

.._local_mac: Local Installation: Mac OSX #

Local Installation: Debian 7.5 "Wheezy"

  1. Install mongodb 1::2.0.6-1.1
    1. sudo apt-get install mongodb
  2. Check to see if pymongo 2.7.1 is already installed with conda list pymongo or pip list pymongo
    1. Note that mongoengine-0.8.7 requires pymongo>=2.5
    2. If you have at least this version, great! Otherwise, continue with this list
    3. Install pymongo 2.7.1 or use the direct link
    4. cd ~/dev/other_src/
    5. tar -xf ~/Downloads/pymongo-2.7.1.tar.gz
    6. pip install ./pymongo-2.7.1/
  3. Install Mongoengine 0.8.7
    1. Download from the above link
      1. Note that if you're at the beamline computers you'll probably need to download mongoengine with a web browser and then use sftp to put it on the channel archiver.
    2. cd ~/dev/other_src/
    3. tar -xf ~/Downloads/mongoengine-0.8.7.tar.gz
    4. pip install ./mongoengine-0.8.7
  4. Make the directory to house the database
    1. mkdir /var/tmp/mongo_data
  5. Clone and install the metadatStore repository (the data broker)
    1. cd ~/dev/other_src/
    2. git clone https://github.com/arkilic/metadataStore.git
    3. pip install ./metadataStore
    4. copy the dataBroker.conf file to your home directory: cp metadataStore/dataBroker.conf ~/dataBroker.conf
    5. edit the dataBroker.conf file so that the host is localhost
  6. Start the mongo database server in one terminal
    1. sudo mongod --dbpath /var/tmp/mongo_data
  7. Go to usage

Local Installation: Windows 7 Professional

  1. Install mongodb
    1. mongodb-win32-x86_64-2008plus-2.6.3-signed.msi
    2. Download from here
    3. Run the installer
  2. Configure mongodb
    1. Follow these instructions
  3. The rest is probably fairly similar to the Debian 7.5 Install, but I'm not sure as I haven't done the rest yet...

Channel Archiver Installation (CSX Example)

  1. ssh to the channel archiver

    1. ssh xf23id-ca

  2. Install mongodb 1::2.0.6-1.1

    1. sudo apt-get install mongodb

  3. Check to see if pymongo 2.7.1 is already installed with conda list pymongo or pip list pymongo

    1. Note 1: mongoengine-0.8.7 requires pymongo>=2.5
    2. Note 2: You'll probably need to download pymongo with a web browser and then use sftp to put it on the channel archiver since wget and pip don't seem to be working due to http proxy issues
    3. If you have at least this version, great! Otherwise, continue with this list
    4. Install pymongo 2.7.1 or use the direct link
    5. cd ~/dev/other_src/
    6. tar -xf ~/Downloads/pymongo-2.7.1.tar.gz
    7. pip install ./pymongo-2.7.1.tar.gz

  4. Install Mongoengine 0.8.7

    1. Download from the above link
      1. Note that if you're at the beamline computers you'll probably need to download pymongo with a web browser and then use sftp to put it on the channel archiver.
    2. cd ~/dev/other_src/
    3. tar -xf ~/Downloads/mongoengine-0.8.7.tar.gz
    4. pip install ./mongoengine-0.8.7

  5. Make the directory to house the database

    1. mkdir /DATA
    2. mkdir /DATA/broker

  6. Start the mongodb

    1. sudo mongod --dbpath /DATA/broker

    2. If you run into an error that looks like this:

      edill@xf23id-ca:~/mongo_setup$ sudo mongod --dbpath /DATA/broker
Fri Jul 11 09:29:26 [initandlisten] MongoDB starting : pid=3314 port=27017 dbpath=/DATA/broker 64-bit host=xf23id-ca
Fri Jul 11 09:29:26 [initandlisten]
Fri Jul 11 09:29:26 [initandlisten] ** WARNING: You are running on a NUMA machine.
Fri Jul 11 09:29:26 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Fri Jul 11 09:29:26 [initandlisten] **              numactl --interleave=all mongod [other options]
Fri Jul 11 09:29:26 [initandlisten]
Fri Jul 11 09:29:26 [initandlisten] db version v2.0.6, pdfile version 4.5
Fri Jul 11 09:29:26 [initandlisten] git version: nogitversion
Fri Jul 11 09:29:26 [initandlisten] build info: Linux z6 3.8-trunk-amd64 #1 SMP Debian 3.8.3-1~experimental.1 x86_64 BOOST_LIB_VERSION=1_49
Fri Jul 11 09:29:26 [initandlisten] options: { dbpath: "/DATA/broker" }
Fri Jul 11 09:29:26 [initandlisten] journal dir=/DATA/broker/journal
Fri Jul 11 09:29:26 [initandlisten] recover : no journal files present, no recovery needed
Fri Jul 11 09:29:26 [initandlisten] ERROR: listen(): bind() failed errno:98 Address already in use for socket: 0.0.0.0:27017
Fri Jul 11 09:29:26 [initandlisten] ERROR:   addr already in use
Fri Jul 11 09:29:26 [websvr] ERROR: listen(): bind() failed errno:98 Address already in use for socket: 0.0.0.0:28017
Fri Jul 11 09:29:26 [websvr] ERROR:   addr already in use
Fri Jul 11 09:29:26 [initandlisten] now exiting
Fri Jul 11 09:29:26 dbexit:
Fri Jul 11 09:29:26 [initandlisten] shutdown: going to close listening sockets...
Fri Jul 11 09:29:26 [initandlisten] shutdown: going to flush diaglog...
Fri Jul 11 09:29:26 [initandlisten] shutdown: going to close sockets...
Fri Jul 11 09:29:26 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jul 11 09:29:26 [initandlisten] shutdown: lock for final commit...
Fri Jul 11 09:29:26 [initandlisten] shutdown: final commit...
Fri Jul 11 09:29:26 [initandlisten] shutdown: closing all files...
Fri Jul 11 09:29:26 [initandlisten] closeAllFiles() finished
Fri Jul 11 09:29:26 [initandlisten] journalCleanup...
Fri Jul 11 09:29:26 [initandlisten] removeJournalFiles
Fri Jul 11 09:29:26 [initandlisten] shutdown: removing fs lock...
Fri Jul 11 09:29:26 dbexit: really exiting now

    3. Find the pid of the currently running mongod:

      edill@xf23id-ca:~/mongo_setup$ ps -e | grep mongo
  300 ?        00:00:09 mongod

    4. In this case, the pid is 300

      1. CHECK WITH THE BEAMLINE SCIENTIST FIRST TO MAKE SURE IT IS OK TO TERMINATE THIS DB INSTANCE!
      2. Then kill it with kill -9 300
      3. Restart mongo
        1. sudo mongod --dbpath /DATA/broker

  7. Return to your local computer (i.e., not the ssh target)

  8. Clone and install the metadatStore repository (the data broker)

    1. cd ~/dev/other_src/

    2. git clone https://github.com/arkilic/metadataStore.git

    3. pip install ./metadataStore

    4. copy the dataBroker.conf file to your home directory:

      cp metadataStore/dataBroker.conf ~/dataBroker.conf

    5. edit the dataBroker.conf file so that the host is xf23id-ca

  9. Go to usage to test the data broker

Usage

  1. Copy sampleCode.py from ~/path_to_metadataStore_repository/sampleCode.py to some other folder so that it runs from the pip installed version, not the local repo

    1. cp ~/dev/my_src/metadataStore/sampleCode.py ~/

  2. Start an ipython session in another terminal

    1. %run ~/sampleCode.py

    2. You should see the database server terminal make some updates, something like:

      Thu Jul 10 17:33:34 [initandlisten] connection accepted from 127.0.0.1:41132 #1
Thu Jul 10 17:33:34 [FileAllocator] allocating new datafile /var/tmp/mongo_data/metaDataStore.ns, filling with zeroes...
Thu Jul 10 17:33:34 [FileAllocator] creating directory /var/tmp/mongo_data/_tmp
Thu Jul 10 17:33:34 [FileAllocator] done allocating datafile /var/tmp/mongo_data/metaDataStore.ns, size: 16MB,  took 0.016 secs
Thu Jul 10 17:33:34 [FileAllocator] allocating new datafile /var/tmp/mongo_data/metaDataStore.0, filling with zeroes...
Thu Jul 10 17:33:34 [FileAllocator] done allocating datafile /var/tmp/mongo_data/metaDataStore.0, size: 64MB,  took 0.487 secs

  3. Still within the ipython interpreter

    1. Test the search functionality

    2. from metadataStore.userapi.commands import search

    3. search(owner='arkilic') should result in something like the following:

      In [6]: search(owner='arkilic')
None
Out[6]:
[{u'_id': 1903,
  u'beamline_id': u'csx',
  u'custom': {},
  u'end_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 408000),
  u'owner': u'arkilic',
  u'start_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 408000),
  u'update_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 408000)},
 {u'_id': 1905,
  u'beamline_id': u'csx',
  u'custom': {},
  u'end_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 988000),
  u'owner': u'arkilic',
  u'start_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 988000),
  u'update_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 988000)}]
Clone this wiki locally