Skip to content
This repository has been archived by the owner on Aug 30, 2017. It is now read-only.

Set up the Data Broker

Jump to bottom
arkilic edited this page Oct 3, 2014 · 41 revisions

Developers: Arman Arkilic, Eric Dill

Directory

  1. Install pymongo
  2. Install metadataStore
  3. Set up on mac
  4. Set up on win7
  5. Set up on debianwheezy
  6. Set up on ChannelArchiverCSX
  7. metadataStore usage

Mac

  1. Use anaconda to manage python libraries
    1. Download anaconda
    2. Install anaconda into your home directory:
      1. bash Anaconda-[version].sh
    3. Check that your python version is running from ~/anaconda/bin/python
    4. If which python does not return /home/username/anaconda/bin/, do not go to the next step until it does!!
  2. Install pymongo
  3. Make the directory to house the database
    1. mkdir /var/tmp/mongo_data
  4. Install metadataStore
  5. Start the mongo database server in one terminal
    1. sudo mongod --dbpath /var/tmp/mongo_data
  6. Go to usage

DebianWheezy

  1. Install mongodb 1::2.0.6-1.1
    1. sudo apt-get install mongodb
  2. Install pymongo
  3. Make the directory to house the database
    1. mkdir /var/tmp/mongo_data
  4. Install metadataStore
  5. Start the mongo database server in one terminal
    1. sudo mongod --dbpath /var/tmp/mongo_data
  6. Go to usage

Win7

  1. Download mongodb

    1. 64-bit install
    2. Run the installer mongodb-win32-[architecture]-[version] (e.g.: mongodb-win32-x86_64-2008plus-2.6.4-signed)
    3. Install Typical configuration
    4. Add C:Program FilesMongoDB 2.6 Standardbin to your system path
      1. press win + pause to open Control Panel\All Control Panel Items\System
      2. open Advanced system settings (item is on the left sidebar)
      3. open the Advanced tab
      4. open Environment Variables
      5. find Path in the System variables section of the window that just popped up
      6. click on Path and then Edit
      7. go to the end of the editable line with the label Variable value:
      8. add the following text: ;"C:\Program Files\MongoDB 2.6 Standard\bin";
      9. click ok a number of times to close all popup windows

  2. Configure mongodb

    1. Follow these instructions

    2. create a location to house the mongo data

      1. I'm using c:\var\tmp\mongo_data

    3. start the mongo db in a terminal with elevated permissions (see next bullet): mongod --dbpath c:\var\tmp\mongo_data and you should get something like this:

      C:\dev\mongodb>mongod --dbpath \var\tmp\mongo_data
2014-08-30T10:35:29.004-0400 [initandlisten] MongoDB starting : pid=8344 port=27017
   dbpath=\dev\mongodb\mongo_data 64-bit host=Eric-i5-desktop
2014-08-30T10:35:29.005-0400 [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2014-08-30T10:35:29.006-0400 [initandlisten] db version v2.6.4
2014-08-30T10:35:29.006-0400 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2014-08-30T10:35:29.006-0400 [initandlisten] build info: windows sys.getwindowsversion(major=6,
   minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2014-08-30T10:35:29.006-0400 [initandlisten] allocator: system
2014-08-30T10:35:29.007-0400 [initandlisten] options: { storage: { dbPath: "\dev\mongodb\mongo_data" }
   }
2014-08-30T10:35:29.010-0400 [initandlisten] journal dir=\dev\mongodb\mongo_data\journal
2014-08-30T10:35:29.011-0400 [initandlisten] recover : no journal files present, no recovery needed
2014-08-30T10:35:29.029-0400 [FileAllocator] allocating new datafile \dev\mongodb\mongo_data\local.ns,
   filling with zeroes...
2014-08-30T10:35:29.030-0400 [FileAllocator] creating directory \dev\mongodb\mongo_data\_tmp
2014-08-30T10:35:29.063-0400 [FileAllocator] done allocating datafile \dev\mongodb\mongo_data\local.ns,
   size:16MB,  took 0.032 secs
2014-08-30T10:35:29.065-0400 [FileAllocator] allocating new datafile \dev\mongodb\mongo_data\local.0,
   filling with zeroes...
2014-08-30T10:35:29.171-0400 [FileAllocator] done allocating datafile \dev\mongodb\mongo_data\local.0,
   size: 64MB,  took 0.104 secs
2014-08-30T10:35:29.173-0400 [initandlisten] build index on: local.startup_log properties: { v: 1, key:
   { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2014-08-30T10:35:29.173-0400 [initandlisten]     added index to empty collection
2014-08-30T10:35:29.174-0400 [initandlisten] command local.$cmd command: create { create:
   "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 144ms
2014-08-30T10:35:29.174-0400 [initandlisten] waiting for connections on port 27017

    4. elevated terminal permissions on windows:

      1. open the start menu
      2. type cmd
      3. right click on cmd and click on Run as administrator

    5. install pymongo

    6. install metadataStore

    7. run metadataStore\examples\sample_code_raw.py to ensure that the metadataStore is functional

  3. go to usage

ChannelArchiverCSX

  1. ssh to the channel archiver

    1. ssh xf23id-ca

  2. Install mongodb 1::2.0.6-1.1

    1. sudo apt-get install mongodb

  3. Check to see if pymongo 2.7.1 is already installed with conda list pymongo or pip list pymongo

    1. Note 1: mongoengine-0.8.7 requires pymongo>=2.5
    2. Note 2: You'll probably need to download pymongo with a web browser and then use sftp to put it on the channel archiver since wget and pip don't seem to be working due to http proxy issues
    3. If you have at least this version, great! Otherwise, continue with this list
    4. Install pymongo 2.7.1 or use the direct link
    5. cd ~/dev/other_src/
    6. tar -xf ~/Downloads/pymongo-2.7.1.tar.gz
    7. pip install ./pymongo-2.7.1.tar.gz

  4. Install Mongoengine 0.8.7

    1. Download from the above link
      1. Note that if you're at the beamline computers you'll probably need to download pymongo with a web browser and then use sftp to put it on the channel archiver.
    2. cd ~/dev/other_src/
    3. tar -xf ~/Downloads/mongoengine-0.8.7.tar.gz
    4. pip install ./mongoengine-0.8.7

  5. Make the directory to house the database

    1. mkdir /DATA
    2. mkdir /DATA/broker

  6. Start the mongodb

    1. sudo mongod --dbpath /DATA/broker

    2. If you run into an error that looks like this:

      edill@xf23id-ca:~/mongo_setup$ sudo mongod --dbpath /DATA/broker
Fri Jul 11 09:29:26 [initandlisten] MongoDB starting : pid=3314 port=27017 dbpath=/DATA/broker 64-bit host=xf23id-ca
Fri Jul 11 09:29:26 [initandlisten]
Fri Jul 11 09:29:26 [initandlisten] ** WARNING: You are running on a NUMA machine.
Fri Jul 11 09:29:26 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Fri Jul 11 09:29:26 [initandlisten] **              numactl --interleave=all mongod [other options]
Fri Jul 11 09:29:26 [initandlisten]
Fri Jul 11 09:29:26 [initandlisten] db version v2.0.6, pdfile version 4.5
Fri Jul 11 09:29:26 [initandlisten] git version: nogitversion
Fri Jul 11 09:29:26 [initandlisten] build info: Linux z6 3.8-trunk-amd64 #1 SMP Debian 3.8.3-1~experimental.1 x86_64 BOOST_LIB_VERSION=1_49
Fri Jul 11 09:29:26 [initandlisten] options: { dbpath: "/DATA/broker" }
Fri Jul 11 09:29:26 [initandlisten] journal dir=/DATA/broker/journal
Fri Jul 11 09:29:26 [initandlisten] recover : no journal files present, no recovery needed
Fri Jul 11 09:29:26 [initandlisten] ERROR: listen(): bind() failed errno:98 Address already in use for socket: 0.0.0.0:27017
Fri Jul 11 09:29:26 [initandlisten] ERROR:   addr already in use
Fri Jul 11 09:29:26 [websvr] ERROR: listen(): bind() failed errno:98 Address already in use for socket: 0.0.0.0:28017
Fri Jul 11 09:29:26 [websvr] ERROR:   addr already in use
Fri Jul 11 09:29:26 [initandlisten] now exiting
Fri Jul 11 09:29:26 dbexit:
Fri Jul 11 09:29:26 [initandlisten] shutdown: going to close listening sockets...
Fri Jul 11 09:29:26 [initandlisten] shutdown: going to flush diaglog...
Fri Jul 11 09:29:26 [initandlisten] shutdown: going to close sockets...
Fri Jul 11 09:29:26 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jul 11 09:29:26 [initandlisten] shutdown: lock for final commit...
Fri Jul 11 09:29:26 [initandlisten] shutdown: final commit...
Fri Jul 11 09:29:26 [initandlisten] shutdown: closing all files...
Fri Jul 11 09:29:26 [initandlisten] closeAllFiles() finished
Fri Jul 11 09:29:26 [initandlisten] journalCleanup...
Fri Jul 11 09:29:26 [initandlisten] removeJournalFiles
Fri Jul 11 09:29:26 [initandlisten] shutdown: removing fs lock...
Fri Jul 11 09:29:26 dbexit: really exiting now

    3. Find the pid of the currently running mongod:

      edill@xf23id-ca:~/mongo_setup$ ps -e | grep mongo
  300 ?        00:00:09 mongod

    4. In this case, the pid is 300

      1. CHECK WITH THE BEAMLINE SCIENTIST FIRST TO MAKE SURE IT IS OK TO TERMINATE THIS DB INSTANCE!
      2. Then kill it with kill -9 300
      3. Restart mongo
        1. sudo mongod --dbpath /DATA/broker

  7. Return to your local computer (i.e., not the ssh target)

  8. Install metadataStore

    1. edit the dataBroker.conf file so that the host is xf23id-broker

  9. Go to usage to test the data broker

Usage

  1. Copy sampleCode.py from ~/path_to_metadataStore_repository/sampleCode.py to some other folder so that it runs from the pip installed version, not the local repo

    1. cp ~/dev/my_src/metadataStore/sampleCode.py ~/

  2. Start an ipython session in another terminal

    1. %run ~/sampleCode.py

    2. You should see the database server terminal make some updates, something like:

      Thu Jul 10 17:33:34 [initandlisten] connection accepted from 127.0.0.1:41132 #1
Thu Jul 10 17:33:34 [FileAllocator] allocating new datafile /var/tmp/mongo_data/metaDataStore.ns, filling with zeroes...
Thu Jul 10 17:33:34 [FileAllocator] creating directory /var/tmp/mongo_data/_tmp
Thu Jul 10 17:33:34 [FileAllocator] done allocating datafile /var/tmp/mongo_data/metaDataStore.ns, size: 16MB,  took 0.016 secs
Thu Jul 10 17:33:34 [FileAllocator] allocating new datafile /var/tmp/mongo_data/metaDataStore.0, filling with zeroes...
Thu Jul 10 17:33:34 [FileAllocator] done allocating datafile /var/tmp/mongo_data/metaDataStore.0, size: 64MB,  took 0.487 secs

  3. Still within the ipython interpreter

    1. Test the search functionality

    2. from metadataStore.userapi.commands import search

    3. search(owner='arkilic') should result in something like the following:

      In [6]: search(owner='arkilic')
None
Out[6]:
[{u'_id': 1903,
  u'beamline_id': u'csx',
  u'custom': {},
  u'end_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 408000),
  u'owner': u'arkilic',
  u'start_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 408000),
  u'update_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 408000)},
 {u'_id': 1905,
  u'beamline_id': u'csx',
  u'custom': {},
  u'end_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 988000),
  u'owner': u'arkilic',
  u'start_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 988000),
  u'update_time': datetime.datetime(2014, 7, 10, 21, 33, 34, 988000)}]

pymongo

  1. Check to see if pymongo 2.7.1 is already installed with conda list pymongo or pip list pymongo
    1. Note that mongoengine-0.8.7 requires pymongo>=2.5
    2. If you have at least this version, great! Otherwise, continue with this list
    3. Install pymongo 2.7.1 or use the direct link
    4. cd ~/dev/other_src/
    5. tar -xf ~/Downloads/pymongo-2.7.1.tar.gz
    6. pip install ./pymongo-2.7.1/
    7. Or just pip install pymongo

metadataStore

  1. Clone and install the metadataStore repository (the data broker)
    1. cd ~/dev/other_src/
    2. git clone https://github.com/arkilic/metadataStore.git
    3. cd metadataStore
    4. python setup.py develop
    5. the metadataStore defaults to using localhost as the mongo db host
Clone this wiki locally