-
Notifications
You must be signed in to change notification settings - Fork 2
Home
John Cameron
12 July 2013
FIXwiki is a wiki whose core content is taken from the FIX Protocol Specification. Wiki pages are generated automatically from the FIX repository.
This document describes the Java and PHP code which is used to generate FIXwiki. It is intended for developers who are responsible for maintaining that code base.
You should start by taking a look at the FIXwiki website http://fixwiki.org/ for high level information about FIXwiki. Particularly the following sections:
All code, data files, and this documentation are available from FPL's Subversion repository at https://svn.fixprotocol.org/svn/fixwiki.
FIXwiki uses MediaWiki for the basic wiki software. There are numerous wiki implementations. However, MediaWiki was chosen because it is the software used for the world's largest and most active wiki, Wikipedia. This gave confidence that the software, whatever shortcomings it might have, would be highly scalable and highly reliable.
FIXwiki's design was strongly influenced by the book "MediaWiki" (Daniel J Barrett, published by O'Reilly). The FIXwiki developer will find that book a very useful reference.
Mediawiki is built using PHP. PHP may not be everyone's favourite language. However it is Mediawiki's native language so the FIXwiki developer will need to acquire some basic knowledge of it.
MediaWiki sits on top of a database. Theoretically that database does not have to be MySQL, but MySQL is what is used by Wikipedia, so FIXwiki also uses MySQL on the assumption that it would be the most reliable partner for MediaWiki. Some attempts were made a while back to move FIXwiki to PostgreSQL and a whole bunch of problems arose. The project was eventually abandoned. So MySQL does indeed appear to be the best choice for MediaWiki. Fortunately, the operation of the underlying database is largely hidden, so the FIXwiki developer requires minimal knowledge of MySQL.
You can freely download the latest MediaWiki software from http://www.mediawiki.org/wiki/Download.
Unpack the MediaWiki distribution into a directory called "w" accessible to a web browser. The web root (DocumentRoot in Apache terms) should be the directory which contains "w".
See the INSTALL file of the unpacked distribution for installation instructions. Basically you just point your browser to the directory you unpacked into and then follow the instructions. Generally you can choose the default values. This is how you should fill in other values.
- Database name - fixwikidb. Not really necessary - the default name of wikidb is OK but it might be handy to have a special name in case you ever want to install a different wiki.
- Database table prefix - fw_. Not really necessary but probably a good idea.
- Wiki name - FIXwiki
- Administrator account - "WikiSysop". Be sure to note down the admin password you choose.
- Contact email - [email protected]
- MySQL database - Note also that you need to have a MySQL user name set up (using root is fine if you know it).
Once MediaWiki is installed, modify the standard MediaWiki LocalSettings.php so that it includes another file LocalSettings.inc. This file is where you will put the FIXwiki specific configuration settings. It can start off blank. You should add the following as the last line in LocalSettings.php:
include('LocalSettings.inc');
Lastly you should make the images subdirectory of the MediaWiki installation writable. This is where uploaded files will be written.
FIXwiki uses the following standard extensions:
These need to be downloaded and unpacked into the wiki's "extensions" subdirectory.
Once the extensions are present in the "extensions" subdirectory, you can copy the LocalSettings.inc across. See the following section.
Note that the ConfirmAccount extension requires that some SQL is run on the underlying database as well. See the doc for that extension.
If the computer that the wiki is running on does not have its own email support (eg through Postfix or Sendmail on a Linux box), then you have to install special PHP email support and you also need to specify email configuration details in LocalSettings.inc.
The relevant configuration lines will already be in the standard LocalSettings.inc. Comment them out if they are not needed. They are:
$wgSMTP = array(
'host' => "mail.angel.net", //could also be an IP address
'IDHost' => "fixwiki.fixprotocol.org",
'port' => 25,
'auth' => false
);
See http://www.mediawiki.org/wiki/Manual:$wgSMTP for detailed doc on the $wgSMTP config.
The PHP email support comes from the PEAR project which supplies all sorts of PHP libraries in the form of "packages". Specifically you need the Mail package plus those packages that it depends on.
This file contains special configuration settings for FIXwiki, including the lines needed to complete the configuration of the standard wiki extensions mentioned above.
It assumes that you are using Apache as your web server.
The latest version of the file can be found in FPL's Subversion repository. A sample copy of the file appears in the Appendix below.
This file should be in place and working as expected for the wiki before running the FIXwiki build script (described below). In particular, the url rewriting rules should be working correctly. Links in imported wiki pages will contain urls according to the configured rules. These cannot be changed later.
Getting the url rewriting working can be troublesome because it depends on the Apache server being correctly configured. See the comments in LocalSettings.inc for guidelines and links to external documentation.
This file must be copied into the wiki's "maintenance" subdirectory. It is a special PHP utility written for FIXwiki which speeds up the build process.
It is based on the standard importTextFile.php utility which can be
found in the "maintenance" subdirectory. The original FIXwiki was built
using importTextFile.php. However that utility only imports one text file
at a time and the FIXwiki build process was becoming rather long.
Using bulkImportTextFiles.php makes the build process 3 to 4 times faster.
There is a page in FIXwiki for every FIX message, component, field, value or type. The FIX repository contains the official definitions of all FIX messages, components, fields, values and types. So generating FIXwiki involves creating a wiki page for each of these FIX repository definitions.
bulkImportTextFiles.php creates wiki pages from the contents
of text files. So the software which generates FIXwiki copies each
FIX repository definition into a text file and creates a PHP script which
imports each text file.
Running the resulting PHP script creates FIXwiki.
This software is written in Java.
To summarize, once the wiki is correctly configured, generating the FIXwiki content (ie all the FIXwiki pages) is a two stage process:
- Read the FIX repository and generate a number of text files and a PHP script which creates a wiki page from each file
- Run the generated PHP script
These are generated automatically from the FIX repository and other sources by the FIXwiki generation code (described below). There are around 40,000 data files totalling about 180Mb.
These files will be imported into the wiki by running the build script which is generated along with the text files.
buildFIXwiki.sh is located in the root directory of the text files.
- It needs to be run from the wiki's "maintenance" subdirectory.
- It needs to be run by a user with full update access to the MySQL wiki database.
- The wiki needs to be fully configured using LocalSettings.inc as described above before running the script. In particular the url rewriting rules need to be functioning correctly.
- It can take 3 or 4 hours to complete.
- After the script has completed it can still take a while for subpages to display correctly. Until that is complete, you may see messages like "Page has no subpages to list".
Running the build script will replace any existing specification wiki pages. However, it will not replace any user content pages.
The command line to run the script (from the wiki "maintenance" directory) is simply:
sh <path to data files root directory>/buildFIXwiki.sh
Sometimes you may see some pages - typically Message and Component pages - with missing lists of sub pages. If so, you may need to force the pages to rebuild by running (in the maintenance subdirectory):
php rebuildall.php
Then you might need to wait a few minutes as the processing completes.
The rest of this document will describe the Java software that actually generates the text files and build script.
The software is kept in FPL's Subversion repository.
It is broken up into the following modules.
The FIX repository is in the form of a number of interrelated XML files. The files are arranged in a directory hierarchy. RepoInfo takes the root directory of a FIX repository and processes all the files in the repository into an internal Java object based representation.
In fact, RepoInfo also process some extra files - defined as resources in the Java code - ie you will find them checked in with the Java source code. These extra files add more extensive definitions for messages, fields etc than what is currently held in the FIX repository. (These definitions come directly from the specification, however, so they are "official").
The intention is that the FIX repository will eventually hold these "elaborated" definitions, but it does not currently. Part of the value of FIXwiki is that it contains these more complete definitions, so for now they need to be supplied by these extra files (ComponentDesc.xml, EnumDesc.xml, Glossary.txt, MessageDesc.xml).
Many pages contain references to other pages. For example, the description about a given FIX message will typically mention some of the fields that make up the message. Each of these fields will have its own FIXwiki page and it is highly desirable for someone reading the message description to be able to quickly go to the page of a mentioned field by simply clicking on the field's name.
LinkDetector processes the text files searching for such links and converting the text accordingly.
MediaWiki uses some very simple rules for formatting text. For example, you indicate a new paragraph by simply leaving a blank line. (Anyone who has contributed to Wikipedia will be familiar with these rules).
This code scans the text files and attempts to convert them to use wiki formatting so that the text displays nicely in FIXwiki.
This is the main module of the FIXwiki generating code. It uses the other modules as required.
Apart from calling the other modules as needed, the main function of this code is to actually generate the text files which end up being FIXwiki pages, as well as the PHP script which imports those text files into the wiki.
The generated FIXwiki pages use templates to standardize the layout of pages. For example, pages that display information about FIX messages all have a common format - only the displayed data changes. FIXwiki imposes that common format by using a template called "Message info" to display the data on each message page. A template is like a Java method. It receives its data passed in as parameters and then does something with that data - eg displays the data on the page.
For example, look at the source for the FIXwiki TestRequest page. It will look like:
{{FPL:TestRequest}}
=Notes=
{{Invitation to post}}
In fact, most FIXwiki pages will look very similar to this. Two templates are called here: "FPL:TestRequest" and "Invitation to post". The "Invitation to post" displays the standard text inviting people to contribute. The "FPL:TestRequest" displays the contents of the TestRequest page in the protected FPL namespace. See the FIXwiki doc - Structure of FIXwiki - for a discussion of the use of namespaces. Basically we are making sure that users can't modify the part of FIXwiki that comes from the specification.
If you look at the source of the page FPL:TestRequest you will see something like:
{{Message info
| Description=The test request message forces a heartbeat...
| AbbrName=TestRequest
| MsgType=1
| Name=TestRequest
| ComponentID=2
| SectionID=Session
| CategoryID=Session
| NotReqXML=1
| added=FIX.2.7
| SegmentType=Message
}}
As you can see, this calls the standard "Message info" template passing in all the data specific to the TestRequest message. The data names are those used in the repository files. One exception is "SegmentType" which is a data name created by FIXwiki. A "segment" is the generic term FIXwiki uses for any collection of fields. This can be a FIX message or a FIX component (also known as a group). So SegmentType can only have the values Message or Component.
This approach means that the appearance of all FIXwiki pages could be changed by simply modifying these formatting templates.
You can look at all the templates, including the "Message info" template, by going to the special Uncategorized templates page.
Currently, FIXwiki is hosted by Bluehost. FPL has an account with Bluehost. Contact the FPL Operations Director, Courtney Doyle McGuinn or John Cameron if you need access to this account, or the FIXwiki server.
Whenever a new Extension Pack is released, the FIX repository will be updated and a new version of FIXwiki needs to be generated, following the process described above.
To generate a new FIXwiki...
- Make the production FIXwiki "read only" by uncommenting the $wgReadOnly variable assignment at the top of the LocalSettings.inc file
- Copy the production FIXwiki down to a copy on a test server.
- Generate the new updated FIXwiki on the test server and test it there.
- Backup the test FIXwiki's MySQL database and restore it to the production FIXwiki's database.
-
Allow changes to the production FIXwiki again by commenting out the
$wgReadOnly line in LocalSettings.inc
</li>
The most common way to back up a MySQL data base is as follows:
mysqldump -uroot -ppassword -r backupfile.sql fixwikio_db
Upload that file to the Bluehost server and then restore it to the production data base. To do this, log on to the Bluehost server, then execute the following from the Bluehost command line:
mysql -ufixwikio_user -ppassword fixwikio_db < backupfile.sql
Note that the FIXwiki database name, user name and password can be found in the FIXwiki configuration file LocalSettings.php
Here is a sample LocalSettings.inc. It may not be completely up to date. It is included here just to give you an idea of what is in it. Get the current version from the FPL Subversion repository.
<?php#$wgReadOnly = "Backing up FIXwiki";
$wgEnableUploads = true;
#Enable short urls.
#The following only works if...
See, for example, http://www.ducea.com/2006/05/30/managing-apache2-modules-the-debian-way/
#RewriteEngine On #RewriteRule ^/?fixwiki(/.)?$ %{DOCUMENT_ROOT}/w/index.php [L] #RewriteRule ^/$ %{DOCUMENT_ROOT}/w/index.php [L]
$wgArticlePath = "/fixwiki/$1"; $wgUsePathInfo = true;
$wgUseAjax = true; $wgEnableMWSuggest = true; $wgLogo = "$wgScriptPath/images/FIX_logo.146px.png" ; $wgFavicon = "$wgScriptPath/images/favicon.ico"; $wgRightsIcon = "$wgScriptPath/images/cameronedge_logo_blueSmall.png"; $wgRightsUrl = "http://cameronedge.com/"; $wgRightsText = "FIXwiki copyrights"; $wgRightsPage = "FIXwiki:Copyrights"; $wgEmailConfirmToEdit = true;
$wgFileExtensions[] = 'txt'; //Used for FIXnames.txt file
$wgNamespacesWithSubpages[NS_MAIN] = true; $wgNamespacesWithSubpages[NS_TALK] = true; $wgNamespacesWithSubpages[NS_CATEGORY] = true; $wgNamespacesWithSubpages[NS_CATEGORY_TALK] = true;
#Wiki requires login to read or edit $wgGroupPermissions['']['edit'] = false; $wgGroupPermissions['']['read'] = true; $wgWhitelistRead = array( 'FIXwiki', 'FIXwiki:About', 'FIXwiki:Copyrights', 'FIXwiki:Structure', 'FIXwiki:Use', 'Special:RecentChanges', 'MediaWiki:Common.css', 'MediaWiki:Monobook.css', 'MediaWiki:Modern.css', '-', 'Special:RequestAccount', );
#Add FPL namespace define("NS_FPL", 100); define("NS_FPL_TALK", 101); $wgExtraNamespaces = array( NS_FPL => 'FPL', NS_FPL_TALK => 'FPL_talk' );
$wgNamespacesWithSubpages[NS_FPL] = true; $wgNamespacesWithSubpages[NS_FPL_TALK] = true;
#Namespace protection $wgNamespaceProtection[NS_PROJECT] = array('editinterface'); $wgNamespaceProtection[NS_FPL] = array('editinterface'); $wgNamespaceProtection[NS_TEMPLATE] = array('editinterface'); $wgNamespaceProtection[NS_HELP] = array('editinterface'); $wgNamespaceProtection[NS_CATEGORY] = array('editinterface');
$wgExternalLinkTarget = '_blank';
require_once( "$IP/extensions/ParserFunctions/ParserFunctions.php" ); require_once("$IP/extensions/SubPageList/SubPageList.php"); // for SubPageList extension
##Confirm account edit stuff require_once("$IP/extensions/ConfirmAccount/ConfirmAccount.php"); $wgAccountRequestToS = true; $wgUseRealNamesOnly = true; $wgRequestExtraInfo = false; $wgAccountRequestMinWords = 10; $wgAllowAccountRequestFiles = false; $wgConfirmAccountContact = "[email protected]" ;
$wgSMTP = array( 'host' => "mail.fixwiki.org", 'IDHost' => "fixwiki.org", 'port' => 26, 'auth' => true, 'username' => "[email protected]", 'password' => "qmc]Kps43)TQ" );
$wgPasswordSender = "[email protected]"; $wgPasswordSenderName = "FIXwiki administrator";
Added by [email protected] on 23 Feb 2011
function lessSpecialPages (&$list) { global $wgUser; if ( ! $wgUser->isAllowed('editinterface') ) { unset( $list['Listusers'] ); unset( $list['Listgrouprights'] ); unset( $list['Activeusers'] ); } return true; } $wgHooks['SpecialPage_initList'][] = 'lessSpecialPages';
Added by [email protected] on 23 Feb 2011
$wgConfirmAccountRejectAge = 5 * 24 * 3600; // 5 days
Added by [email protected] on 24 Feb 2011
$wgRightsPage = 'FIXwiki:Terms_of_Use';
$wgDefaultSkin = 'monobook';