===========================================================
Ocean Observatories Initiative Cyberinfrastructure (OOI CI)
Integrated Observatory Network (ION)

ioncore-python - Python Capability Container and Core Modules

(C) UC Regents, 2010-2011 (see LICENSE.txt)
===========================================================

This project provides a Python process execution framework (capability
container, CC) with core, auxiliary and testing functions for deploying,
running and testing capability container processes within a standard local and
distributed execution environment.

The primary purpose of the ION system, with ioncore-python at its core, is to
provide a data distribution network based on pub-sub messaging with instrument
integration, based on underlying pervasive infrastructure services (security,
policy and governance, persistence) and computational resource provisioning.

The Capability Container provides a standard life-cycle model for CC processes,
embedded within the ION system bootstrapping process and defined launch
plans (see cloudinit.d). The Capability Container provides an interface to the
Exchange, a message-based communication system, to enable inter-process
communication following the OOI Common Message Format. It also provides
access to ION object definitions and provides the Distributed State Framework
for transporting, persisting and manipulating structured objects of defined
type.

For more information, please see:
http://www.oceanobservatories.org/spaces/display/syseng/CIAD+COI+SV+Python+Capability+Container
https://confluence.oceanobservatories.org/display/syseng/Architecture+and+Design

Status
======

The current version of ioncore-python is preliminary, preparing for Release 1
of the OOI ION targeted for May 2011. Versions designated 0.x are released
project internally in order to enable dependent packages to include
ioncore-python in stable versions.

When getting code from GitHub, please note that until Release 1, all development
and integration occurs in the "develop" branch. Released ioncore-python packages
are available via buildout scripts or from the OOI package server:
    http://ooici.net/packages/

Dependencies
============

ioncore-python is compatible with Python 2.5 only.
This project has several dependencies on libraries and packages. Most of these
dependencies are resolved automatically using the provided buildout scripts.

IMPORTANT:

1. When using buildout to build your project, you should only call wrapper scripts
generated by buildout in the bin/ directory.
    * bin/trial  (not trial)
    * bin/twistd (not twistd)
    * bin/mypython (not python)
    * etc...

2. You need to start a new virtualenv for buildout to work
correctly.  Buildout will skip installing twistd/trial into bin/ directory
if there's already twisted installed in the old virtualenv.

Installation
============

Example setup to work on ioncore-python:
----------------------------------------

mkvirtualenv --no-site-packages --python=/usr/bin/python2.5 your_env_name
cdvirtualenv
mkdir some_dir_to_keep_it_all_in
cd some_dir_to_keep_it_all_in
git clone [email protected]:ooici/ioncore-python.git
cd ioncore-python
python bootstrap.py
ant clean
bin/buildout -c development.cfg
bin/trial ion

Example setup to work on ioncore-python and ion-object-definitions:
-------------------------------------------------------------------

mkvirtualenv --no-site-packages --python=/usr/bin/python2.5 your_env_name
cdvirtualenv
mkdir some_dir_to_keep_it_all_in
cd some_dir_to_keep_it_all_in
git clone [email protected]:ooici/ioncore-python.git
git clone [email protected]:ooici/ion-object-definitions.git
cd ioncore-python
cd ../ion-object-definitions
(Refer to ion-object-definitions README.txt for build instructions)
cd ../ioncore-python
python bootstrap.py
ant clean
bin/buildout -c dev-integration.cfg
bin/trial ion


Current dependencies:
====================
M2Crypto note: Needs swig installed on the O.S.


Usage
=====

(all subsequent steps assume you are in the ioncore-python/ root dir)

Start empty Python Capability Container shell with:
::
    bin/twistd -n cc -h amoeba.ucsd.edu
    bin/twistd -n cc   # to run with localhost
    # To set a sysname, i.e. a "cluster name" for all containers in a cluster
    bin/twistd -n cc -h amoeba.ucsd.edu -a sysname=mycluster

(to end a capability container shell, press Ctrl-D Ctrl-C)

Start system by executing within buildout altered Python shell:
bin/mypython:
><>
    from ion.core import bootstrap
    bootstrap.start()

Alternatively (better) from executing a script:
::
    bin/twistd -n cc -h amoeba.ucsd.edu res/scripts/bootstrap.py
    bin/twistd -n cc -h amoeba.ucsd.edu res/scripts/newcc.py


Testing
=======

Run trial test cases (recursively)
::
    bin/trial ion
    bin/trial ion.core
    bin/trial ion.services.coi.test.test_resource_registry

A good learning example is the HelloService
::
    bin/trial ion.play.test.test_hello

Or in the cc shell.
bin/twistd -n cc -h amoeba.ucsd.edu 
><>
    from ion.play import hello_service
    spawn(hello_service)
    send(1, {'op':'hello','content':'Hello you there!'})

    from ion.play.hello_service import HelloServiceClient
    hc = HelloServiceClient()
    hc.hello()


Ant Targets
=============================

To clean your working directories, run
::  ant clean

To clean buildout generated directories
::  ant clean-buildouti

To compile all code to see if there are Python compile errors anywhere:
::  ant compile


Local Configuration
===================

To override default configuration and logging config files (and in order to
prevent changes to the global default), please make local changes in:
    res/config/ionlocal.config   (eg. to set a local message broker)
    res/logging/loglevelslocal.cfg   (eg. to increase/reduce log verbosity)


---------------------------------------------------------------------------
Change log:
===========

2011-04-09:
- Removed support for bootstrap.declare_messaging and
  IonTestCase._declare_messaging.
- Removed all scripts in res/scripts. Use app and rel files instead.
- Added interaction (message) observer app. Start with
  bin/twistd -n cc res/apps/observer.app

2011-04-05:
- Call _so_transition in StateObject action function to transition state.
- Release (rel) files in res/deploy now supported as start argument. Release
  files can provide app specific config and start args.
- Apps can accept kwargs in app file and via the apparg_<appname> container
  command line arg.

2011-04-02:
- App files support an entry "config" of type dict with ion.config override
  values
- Apps support loading of dependent apps from res/apps by using the
  "applications" entry of type list.

2011-04-01:
- Introduced Conversations framework. Standard conversation types: rpc (request,
  inform-result/failure), request (request, agree/refuse, inform-result/failure),
  generic (any message is processed). Also conversation-less one-off messages.
- Refactored Process base class to use conversations. Made rpc_send retrofit
  RPC conversation; reply_err now sends a conversation failure (either rpc or
  request); other reply methods also supported.
- Conversations log messages; they are garbage collected in final state.
- Process now supports graceful terminate from active state; Receiver can
  gracefully handle deactivating the consumer followed by wait for all
  processing completion.
- Bug fixes to process terminate and life cycle handling.
- Bug fixes to StateObject error handling and reporting (the moment of the state
  change current->next was not exactly clear in the handler code; now it is
  either before or after)
- FSM now handles catch events and post state change action methods

2011-01-28:
- Switched to binary sha1 keys in all objects.
- Added @ITV decorator to skip itv tests when running 'trial ion'

2011-01-26:
- Moved scripts in bin/ dir to scripts/ dir.

2011-01-24:
- Removed IRODS as backend storage configureation option

2011-01-21:
- ION is now using google protocol buffer objects in the resource object model
- Bumped version number to 0.4.0 consistent with the setup.py file.

2010-10-28:
- Set RPC default timeout to 15 secs (see ion.config).
  Use a different secs value in in rpc_send(..., ..., timeout=5)

2010-10-05:
- REFACTORING OF BASE CLASSES CONTINUED
- Changed ion.core.base_process.BaseProcess to ion.core.process.process.Process
- Changed ion.services.base_service.BaseService to
  ion.core.process.service_process.ServiceProcess
- Modified all dependent classes

2010-10-04:
- MASSIVE REFACTORING IN BASE CLASSES
- Refactored the former magnet code into more object oriented style.
- Requires Carrot 0.10.11. Carrot before does not handle all deferred
  operations correctly.
- Refactored the Receiver use. There are now subclasses for Receivers that
  manage and declare the specific types of AMQP resources, such as worker and
  fanout. No more declare_messaging necessary.
- Refactoried the capability container classes.
- Added a FSM based StateObject. Many manager/controller level objects now make
  use of states. States and operations INIT -> initialize() -> READY ->
  activate() -> ACTIVE -> terminate() -> TERMINATED (and more, with errors)
- BaseProcess (and subclasses), Receiver, ProcessDesc, Container etc are all
  BasicLifecyleObjects.
- BaseProcess now waits to activate the receiver until in ACTIVE state. Before,
  code can do RPC, but not receive messages on the process id
- Massively enhanced the capability container API. Delegated the actual
  implementation to manager classes: proc, exchange, app manager
- Refactored the way processes are spawned. Refactored ProcessDesc to use the
  new container API. Processes are by default immediately initialized and
  activated. The op_init message has been eliminated.
- Renamed ProtocolFactory to ProcessFactory; changed in each process module
- Message headers now contain status code for every message. 'OK is the default
  and 'ERROR' is set on error
- BaseProcess.rpc_send now raises a ReceivedError in case the RPC comes back
  with status='ERROR'
- Changed reply_ok and reply_err: a dict content value will not be modified
- Fixed imports and tests throughout the code base
- Added OTP style apps and app files as primary way to start up processes
  in the container. See res/apps/*.app files and ion.core.pack

2010-09-20:
- Added start scripts in bin/
- Use ant install to install Python dependencies (calls python setup.py install)
- Removed dependency on magnet. Included all relevant magnet code in ion.core
  packages cc and messaging.
  Start with: twistd -n cc
- Included all CEI services and base classes in code base

2010-08-29:
- Changed all logging instances for loggers to log, to avoid name clashes.

2010-08-14:
- BaseProcess: added backend receiver, used for sending out any messages
  from self.send and self.rpc_send. This keeps the message queue for the process
  frontend separate from the process backend, e.g. for RPC during a message
  processing.
- Changed BaseProcess logging to make message send and receive easier to spot.

2010-08-06:
- BaseProcess.spawn() now calls init() automatically. No need to call init()
  on a process anymore manually. For testing only. Note: The correct way to
  spawn a process is through a parent process with spawn_child()
- Modified and fixed the BaseProcess states, when receiving messages
- MAJOR update to BaseProcess message dispatching and subsequent error handling.
  On error, reply_err messages are sent back, if reply-to header set.

2010-08-03:
- Added ant build.xml file to LCAarch root dir. Start with ant.
  Supports ant clean, which removes all *.pyc from ion path.

2010-07-23:
- Refactored the Registry Services to inherit from a common base class. This
  will allow easier implementation of the many registries in the OOICI. The
  Resource Registry and Service Registry now have basic registration of resource
  descriptions and services.
- The Ion message is now encoded usig the MsgPack library to allow for binary
  transport of message content. The JSON library was mangleing string content.
  This is a temporary fix for a wider problem with the encoding structure which
  needs to be addressed in construction.

2010-06-07:
- Redefined logging import to set as module level logger. Each module has now
  a different logger that can be configured separately. Don't configure
  manually though.
- Added possibility to modify logging levels by module/package, with support
  for package hierarchy. See res/logging/loglevels.cfg for standard entries.
  Do not modify. Add file res/logging/loglevelslocal.cfg for local override.
  Default logging level for all ion code: WARNING

2010-06-02:
- BaseProcess self members renamed to conform to PEP8
- Added process shutdown to BaseProcess
- Added container UNIX shell argument -a processes=<path to filename> used
  by newcc.py script, with a list of processes to startup in standard format

2010-05-25:
- Made Cassandra backend parameterizable with keyspace/colfamily and added
  SuperColumn support.
- Modified the IStore interface to support a create_store factory method. This
  method can yield and return a deferred. Modified and fixed IStore impls.
  Changed delete to remove to be more compliant with standard collections.

2010-05-22:
- Added timeout to BaseProcess.rpc_send. Use with kwarg timeout=<secs>
- CC-Agent detects missing known containers and removes them from the list
- Enhanced CC-Agent operations and CC shell helpers
- Added sequence numbers for messages
- Added glue functions BaseProcess.reply_ok and reply_err and changes some
  RPC style service operations.

2010-05-20:
- The system now looks for a local config file ionlocal.config and if exists,
  overrides entries in ion.config.
- Test cases use the config file to determine the broker host to use. If local
  config override exists, a different broker (e.g. localhost) can be given.
- Added BaseProcessClient and changed BaseServiceClient and all clients and
  all test cases (again).
- Added container shell helpers under 'cc', such as cc.spawn('hello')

2010-05-16:
- Removed support for BaseProcess.send_message and reply_message. Always use
  send, reply and rpc_send now.
- Any BaseProcess instance can now spawn_child() other processes.
- Removed RpcClient class, because every process can do rpc_send()
- Service processes now also listen to their service name's queue. The service
  name is determined from the service declaration. Two processes will listen
  to the same queue and take messages round robin from the queue.
- Startup arguments evaluated, for instance to start with system name set:
  twistd -n magnet -a sysname=mysys
  twistd -n magnet -a "{'sysname':'mysys'}"
- Added capability container agent process. Start with:
  twistd -n magnet res/scripts/newcc.py
  Agents announce themselves to others in the same system and can spawn procs.
- Name scope 'local' for messaging names means now really local to one container.
  Use scope 'system' for names unique for each bootstrapped system. Do not use
  global names, because they will clash.
- Less verbose trace output for process init messages and changes to other
  trace output as well.
- Changed BaseServiceClient and tests. Initializer arguments different.
- Using master branch of magnet now

2010-05-10:
- Based on entries in config files, service process modules are sought and
  loaded in order to collect the service process declarations. This enables
  a registration of services with versions and dependencies on startup

2010-05-06:
- Refactored the BaseProcess class.
  - You can now do RPC directly from the process, via self.rpc_send without
    the need for an RpcClient. Works even with other messages coming in at the
    same time (using the conv-id)
  - Added aliases: self.send, self.reply (use these now)
  - Process instance can be spawned via self.spawn()
- Provided an easier to use BaseServiceClient, which uses a default service
  name lookup. Accepts BaseProcess instance as argument to use for sending/
  receiving service calls.

TODO:

- TBD: Timeout as defined state transition of conversation
- TBD: Conversation garbage collection and final/error state detection - tombstone
- TBD: Conversation FSM cloning (instead of always new creation)
- TBD: Interaction observer and generic receiver (for all operations on broker)
- TBD: magnet.topic routing key naming (proc.main.#, proc.back.#, service.# etc)
- TBD: proc.back.send-contid.sendpid.main.reccontid.recpid naming and bindings

Apps:
- Declare which apps to load in a test case
- Easier to use app-supervisor
- Generic app with configured file
- Logs in apps
- App dependencies
- Distributed apps
Conversations:
- send inside FSM
- timeout handing inside FSM
- timeout transition
- request as process default
- distinguish failure and result
- non-blocking rpc
- refactor common pieces of rpc and request out
- guard FSM vs observer FSM vs interceptor
- State repo for conversations
- Tombstones and timeouts
- FSM keeps list of recent messages (for logging)
- FSM caching, cloning
- msg_send() FSM exceptions and abort of send. Return code of FSM handler
Interactions/messaging:
- Generic receiver, no dispatch
- Process observer keeps list of recent messages
- Routing key for processes
- Public to private relay (or guard/interceptor) for processes as service
- Conversation observer
Misc:
- Better FSM semantics: pre, post actions,
- Correct handling of error conditions, timeouts, tombstones
- Exceptions in FSM. Automatism of error transition?
Testing:
- optional process shutdown
- msg observer usable / queriable in test
Exchange Spaces:
- separate process/service from data from events
- use VHOST instead of sysname

.