Sharing libraries between blocks

[t-saideep]

Blocks may require shared libraries. There should be a place to these. Currently the only shared code that all blocks see are in datablox_framework.

[jfischer]

That's what Engage is for. You just need to have an engage driver for the library and make that a dependency of both blocks. If you need to do that for something, let me know.

[t-saideep]

Would it work if the library is not publicly available? I'm looking at a scenario where I write a custom library to extract relevant links from an HTML page for the bookmarks example. This library is only going to be used by the web-crawler and bookmark-query blocks and probably no one else. So this is probably not going be uploaded online anywhere but will instead be present with Datablox sources.

Right now this is pretty small and I don't need a separate library just for this issue, but I was thinking of the general case.

[jfischer]

Good point. I think the best way to handle this scenario is as follows:

1. Create a python package (e.g. with a setup.py file) for the library.
2. Put the source tree for this package somewhere under the datablox source tree (maybe datablox/libraries).
3. Create the Engage metadata for this package. No driver is needed, as it can just use the pip driver for installation.
4. The Makefile should tar up the package and copy it to datablox/engage/engage-dist/sw_packages.
5. The blocks that require this library would have it listed as a dependency in their resource definitions.
6. Engage will install the library on any nodes running the associated blocks.

[t-saideep]

Thanks! I think that is the best way to go.