Installation

Here are some instructions you can use to obtain the Remote Data Toolbox and try it out.

You can:

• Get the Remote Data Toolbox and run the included demos.
• Get isetbio and the Unit Test Toolbox and run full validations with data access via Remote Data Toolbox (no Dropbox).

It's also possible to publish data from the Unit Test Toolbox, via the Remote Data Toolbox. We can add those instructions later, once we iron out any wrinkles encountered so far.

Get Remote Data Toolbox

Clone (or pull) the latest code for the Remote Data Toolbox with Git:

git clone https://github.com/isetbio/RemoteDataToolbox.git

Then add the RemoteDataToolbox folder and sub-folders to your Matlab path. In fall/winter 2015/2016 we made a lot of changes to the RemoteDataToolbox, so you should be sure to update your path even if you already have RemoteDataToolbox.

Run Remote Data Toolbox Included Demos

There are 4 example scripts you can read and run. These live in the RemoteDataToolbox/examples folder.

brainard-archiva/rdtExamplePublishData.m shows how you can create some data locally, upload it to our demo repository, and view the data in your web browser. This example uses image data, so it's easy to view.

brainard-archiva/rdtExampleReadData.m follows the previous. It shows how you can download data from our demo repository -- in this case the image you just created.

brainard-archiva/rdtExampleQueryRepository.m shows how you can use the Remote Data Toolbox to list and query artifacts that live in our demo repository.

maven-central/rdtExampleMavenCentral.m shows that Remote Data Toolbox can access other data repositories, not just our own server. In this case it accesses a repository called "Maven Central" which stores lots of open-source Java stuff.

Get isetbio and Unit Test Toolbox

We recently modified isetbio and the Unit Tests Toolbox to work with the Remote Data Toolbox. You should clone (or Pull) these repositories to get the lastest code.

For isetbio:

git clone https://github.com/isetbio/isetbio.git

For Unit Test Toolbox:

git clone https://github.com/isetbio/UnitTestToolbox.git

Then add the isetbio and UnitTestToolbox folders and sub-folders to your Matlab path if you haven't already.

Run isetbio Full Validation via Remote Data Toolbox

We now have isetbio "full" and "fast" validation data hosted in our own isetbio repository.

Configuration Overview

So now we can configure the Unit Test Toolbox to fetch data from the repository on demand, instead looking in a local folder managed by Dropbox. The necessary configuration appears in the ieSetUnitTestPreferencesTemplate script.

Here are configuration highlights:

'alternateFastDataDir',  '',  ...
'alternateFullDataDir',  '', ...
'useRemoteDataToolbox',  true, ...
'remoteDataToolboxConfig', 'isetbio', ...

useRemoteDataToolbox tells Unit Test Toolbox to fetch data from our remote repository instead of looking in a local folder. remoteDataToolboxConfig tells the Remote Data Toolbox the name of a JSON file with details about our repository.

When using the Remote Data Toolbox, we can omit alternateFastDataDir and alternateFullDataDir because we are not using a local data folder.

Run Your Configuration!

You should set your local configuration as described above. You might just execute ieSetUnitTestPreferencesTemplate. Or, you might modify your own configuration script like above, then execute that. Either way, make sure to execute your configuration script. At least two of us have found it easy to forget this step.

Validation

Finally, try an isetbio full validation run:

ieValidateFullAll

This may take a couple minutes and should produce lots of output. For various reasons, not every test may pass. But you should see some tests passing with FAST and FULL validation, indicating that the toolboxes successfully fetched data from our remote repository. For example:

[ 20] 'v_wvfSpatialSampling': ... Internal validation: PASSED ; Runtime status: NO EXCEPTION RAISED ; Fast validation: PASSED ; Full validation: PASSED ; 

You should also see various urls in the test output which point at our remote repository. For example:

Ground truth url : http:/52.32.77.154/repository/isetbio/validation/fast/wavefront/wvfSpatialSampling
-----------------------------------------------------------------------------------------------------
Ground truth url : http:/52.32.77.154/repository/isetbio/validation/full/wavefront/wvfSpatialSampling

You can visit these urls in your browser to see where the data live:

• http:/52.32.77.154/repository/isetbio/validation/fast/wavefront/wvfSpatialSampling
• http:/52.32.77.154/repository/isetbio/validation/full/wavefront/wvfSpatialSampling