Directory reorganisation and scoped packages (OriginProtocol#1738)

* Directory changes

* Remove stray yarn file

* Update docker compose

* Update lerna

* Update dockerfiles

* Scope packages

* Fix two packages

* Update imports

* More import fixing

* Fix paths

* Update Dockerfile

* Update eslint

* Update eslint react

* More eslint

* More eslint

* More eslint react

* More fixes

* Update docker-compose for tests

* Move new files

* Update docs

* Update readme

* Move new files

* Update admin dockerfile

* Update chart URLs

* Update TravisCI

* Cleanup dir

* Update package versions

* Fix travis file

* Package updates

* More package syncing

* Add missing CNAME step to docs build

* Move cloudbuild file

* Update ganache

* Package lock refresh

* Reorder TravisCI file

* 32 CPU machine for builds

* Fix some paths

* Fix import

* Revert "Update ganache"

This reverts commit e972d43.

* Package lock refresh

* Revert dockerfile names

* Sync ganache versions

* Use ganache 2.4.0 in contracts

Author: Tom Linton

.travis.yml

@@ -3,15 +3,6 @@ git:
   depth: 6
 matrix:
   include:
-    - name: "origin-graphql unit tests"
-      language: node_js
-      node_js: 10
-      install:
-        - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-contracts --scope origin-eventsource --scope origin-ipfs --scope origin-messaging-client --scope origin-services --scope origin-validator --scope origin-linker-client --scope origin-graphql
-      script:
-        - npm run test --prefix origin-graphql
-
     # Run integration tests using docker compose
     # - name: "integration tests"
     #   language: minimal
@@ -37,7 +28,16 @@ matrix:
       script:
         - npm run lint
 
-    - name: "origin-bridge unit tests"
+    - name: "@origin/graphql unit tests"
+      language: node_js
+      node_js: 10
+      install:
+        - npm install --ignore-scripts
+        - npm run bootstrap -- --scope @origin/contracts --scope @origin/eventsource --scope @origin/ipfs --scope @origin/messaging-client --scope @origin/services --scope @origin/validator --scope @origin/linker-client --scope @origin/graphql
+      script:
+        - npm run test --prefix packages/graphql
+
+    - name: "@origin/bridge unit tests"
       language: node_js
       node_js: 10
       addons:
@@ -48,23 +48,23 @@ matrix:
         - DATABASE_URL=postgres://postgres@localhost/travis_ci_test
       before_script:
         - psql -c 'create database travis_ci_test;' -U postgres
-        - lerna run migrate --scope origin-bridge
+        - lerna run migrate --scope @origin/bridge
       install:
         - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-bridge
+        - npm run bootstrap -- --scope @origin/bridge
       script:
-        - npm run test --prefix origin-bridge
+        - npm run test --prefix infra/bridge
 
-    - name: "origin-contracts unit tests"
+    - name: "@origin/contracts unit tests"
       language: node_js
       node_js: 10
       install:
         - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-contracts
+        - npm run bootstrap -- --scope @origin/contracts
       script:
-        - npm run test --prefix origin-contracts
+        - npm run test --prefix packages/contracts
 
-    - name: "origin-discovery unit tests"
+    - name: "@origin/discovery unit tests"
       language: node_js
       node_js: 10
       addons:
@@ -78,16 +78,16 @@ matrix:
         - ATTESTATION_ACCOUNT=0x99C03fBb0C995ff1160133A8bd210D0E77bCD101
       before_script:
         - psql -c 'create database travis_ci_test;' -U postgres
-        - lerna run migrate --scope origin-discovery
-        - lerna run migrate --scope origin-growth
-        - lerna run migrate --scope origin-identity
+        - lerna run migrate --scope @origin/discovery
+        - lerna run migrate --scope @origin/growth
+        - lerna run migrate --scope @origin/identity
       install:
         - npm install
-        - lerna run build --scope origin
+        - lerna run build --scope @origin/js
       script:
-        - npm run test --prefix origin-discovery
+        - npm run test --prefix infra/discovery
 
-    - name: "origin-growth unit tests"
+    - name: "@origin/growth unit tests"
       language: node_js
       node_js: 10
       addons:
@@ -98,38 +98,38 @@ matrix:
         - DATABASE_URL=postgres://postgres@localhost/travis_ci_test
       before_script:
         - psql -c 'create database travis_ci_test;' -U postgres
-        - lerna run migrate --scope origin-growth
-        - lerna run migrate --scope origin-identity
+        - lerna run migrate --scope @origin/growth
+        - lerna run migrate --scope @origin/identity
       install:
         - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-contracts --scope origin-eventsource --scope origin-ipfs --scope origin-messaging-client --scope origin-services --scope origin-validator --scope origin-linker-client --scope origin-graphql --scope origin-identity --scope origin-token --scope origin-growth
+        - npm run bootstrap -- --scope @origin/contracts --scope @origin/eventsource --scope @origin/ipfs --scope @origin/messaging-client --scope @origin/services --scope @origin/validator --scope @origin/linker-client --scope @origin/graphql --scope @origin/identity --scope @origin/token --scope @origin/growth
       script:
-        - npm run test --prefix origin-growth
+        - npm run test --prefix infra/growth
 
-    - name: "origin-js unit tests"
+    - name: "@origin/js unit tests"
       language: node_js
       node_js: 10
       install:
         - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-contracts --scope origin
+        - npm run bootstrap -- --scope @origin/contracts --scope @origin/js
       script:
-        - npm run test --prefix origin-js
+        - npm run test --prefix packages/origin-js
 
-    - name: "origin-notifications unit tests"
+    - name: "@origin/notifications unit tests"
       language: node_js
       node_js: 10
       install:
         - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-notifications
+        - npm run bootstrap -- --scope @origin/notifications
       script:
-        - npm run test --prefix origin-notifications
+        - npm run test --prefix infra/notifications
 
-    - name: "origin-token-transfer unit tests"
-      language: node_js
-      node_js: 10
-      install:
-        - npm install --ignore-scripts
-        - npm run bootstrap -- --scope origin-contracts --scope origin-token --scope origin-token-transfer-client --scope origin-token-transfer-server
-      script:
-        - npm run test --prefix origin-token-transfer-server
-        - npm run test --prefix origin-token-transfer-client
+    # - name: "@origin/token-transfer-* unit tests"
+    #  language: node_js
+    #  node_js: 10
+    #  install:
+    #    - npm install --ignore-scripts
+    #    - npm run bootstrap -- --scope @origin/contracts --scope @origin/token --scope @origin/token-transfer-client --scope @origin/token-transfer-server
+    #  script:
+    #    - npm run test --prefix infra/token-transfer-server
+    #    - npm run test --prefix infra/token-transfer-client

DEVELOPMENT.md

@@ -17,31 +17,31 @@ git clone https://github.com/OriginProtocol/origin
 cd origin && npm install
 ```
 
-2. You can then start the DApp using:
+2. You can then start the marketplace DApp using:
 
 ```
-cd origin-dapp && npm start
+npm start
 ```
 
 This will start a `webpack-dev-server` with hot reloading on `http://localhost:3000.`. When you open it you should see the message `No marketplace contract?`.
 
 3. Deploy contracts (optional)
 
-By default the DApp will start its own Ethereum blockchain using Ganache. Because it is a fresh network you'll need to deploy some contracts and create some sample listings using the `origin-admin` tool. This can be done by running:
+By default the DApp will start its own Ethereum blockchain using Ganache. Because it is a fresh network you'll need to deploy some contracts and create some sample listings using the `admin` tool. This can be done by running:
 
 ```
-cd origin-admin && npm start
+cd admin && npm start
 ```
 
 Then open your browser to `http://localhost:3001` and:
 
 - Select the Settings page (last icon on the right)
 - Click the green `Populate` button.
-- Copy and pasting the commands at the bottom of the page into the console for `origin-dapp`.
+- Copy and pasting the commands at the bottom of the page into the console for `dapp`.
 
 ### Network selection
 
-You can also change the Ethereum network being used by `origin-dapp` by appending a network name to the URL.
+You can also change the Ethereum network being used by the marketplace DApp by appending a network name to the URL.
 
 - http://localhost:3000/docker - Local Ganache and services run by Docker Compose (see below for further instructions)
 
@@ -61,7 +61,7 @@ You can view the state of the network at https://testnet.originprotocol.com/.
 
 ### Other settings
 
-The DApp includes a settings page at `http://localhost:3000/settings` which is useful if you want to switch individual services, e.g. use a different Web3 provider or atteestation server.
+The marketplace DApp includes a settings page at `http://localhost:3000/settings` which is useful if you want to switch individual services, e.g. use a different Web3 provider or atteestation server.
 
 ### About the Origin repository
 
@@ -74,19 +74,19 @@ There is a Docker Compose configuration available for running a variety of backe
 ```
 - elasticsearch on http://localhost:9200
 - postgresql
-- origin-services (ipfs server and Ethereum blockchain using ganache on http://localhost:8545)
-- origin-bridge on http://localhost:5000
-- origin-discovery
-- origin-dapp on http://localhost:3000
-- origin-event-listener
-- origin-discovery (apollo server on http://localhost:4000)
-- origin-growth (apollo server on http://localhost:4001)
-- origin-ipfs-proxy on http://localhost:9999
-- origin-messaging on http://localhost:9012
-- origin-notifications on http://localhost:3456)
+- @origin/services (ipfs server and Ethereum blockchain using ganache on http://localhost:8545)
+- @origin/bridge on http://localhost:5000
+- @origin/discovery
+- @origin/marketplace on http://localhost:3000
+- @origin/discovery (event-listener)
+- @origin/discovery (apollo server on http://localhost:4000)
+- @origin/growth (apollo server on http://localhost:4001)
+- @origin/ipfs-proxy on http://localhost:9999
+- @origin/messaging on http://localhost:9012
+- @origin/notifications on http://localhost:3456)
 ```
 
-⚠️  If you want to run the Docker Compose setup ensure that both `origin-dapp` and `origin-admin` are not running before you start the services. The required ports will not be available if either of those two are started before running `docker-compose up`.
+⚠️  If you want to run the Docker Compose setup ensure that both `@origin/marketplace` and `@origin/admin` are not running before you start the services. The required ports will not be available if either of those two are started before running `docker-compose up`.
 
 ### System Requirements
 
@@ -136,7 +136,7 @@ Start and stop the environment:
 Spawn a shell (command line) in a container:
 
 	docker exec -ti <container_name> /bin/bash
-	docker exec -ti origin-dapp /bin/bash
+	docker exec -ti dapp /bin/bash
 
 Follow log output for all containers:
 
@@ -189,7 +189,7 @@ If a container is failing with code 137 it could be that it has encountered Out
 
 #### Port errors
 
-The environment requires a number of ports to be free on your machine (3000, 5000, 5002, 8080, 8081 and 8545). If one of these ports isn't available spinning up the development environment may fail. This includes `origin-dapp` and `origin-admin`. Ensure you start those after you run `docker-compose up`.
+The environment requires a number of ports to be free on your machine (3000, 5000, 5002, 8080, 8081 and 8545). If one of these ports isn't available spinning up the development environment may fail. This includes `@origin/dapp` and `@origin/admin`. Ensure you start those after you run `docker-compose up`.
 
 #### Metamask errors
 

Dockerfile

@@ -9,33 +9,32 @@ COPY ./scripts/wait-for.sh /usr/local/bin/
 # Docker to cache the npm install steps if none of the dependencies have changed
 COPY ./lerna.json ./
 COPY ./package.json ./
-COPY ./origin-discovery/package.json ./origin-discovery/
-COPY ./origin-ipfs-proxy/package.json ./origin-ipfs-proxy/
-COPY ./origin-js/package.json ./origin-js/
-COPY ./origin-messaging/package.json ./origin-messaging/
-COPY ./origin-notifications/package.json ./origin-notifications/
-COPY ./origin-tests/package.json ./origin-tests/
-COPY ./origin-growth/package.json ./origin-growth/
-COPY ./origin-identity/package.json ./origin-identity/
-COPY ./origin-token/package.json ./origin-token/
-COPY ./origin-bridge/package.json ./origin-bridge/
-COPY ./origin-dapp/package.json ./origin-dapp/
-COPY ./origin-graphql/package.json ./origin-graphql/
-COPY ./origin-ipfs/package.json ./origin-ipfs/
-COPY ./origin-validator/package.json ./origin-validator/
-COPY ./origin-messaging-client/package.json ./origin-messaging-client/
-COPY ./origin-linker-client/package.json ./origin-linker-client/
-COPY ./origin-eventsource/package.json ./origin-eventsource/
-COPY ./origin-services/package.json ./origin-services/
+COPY ./packages/origin-js/package.json ./packages/origin-js/
+COPY ./packages/graphql/package.json ./packages/graphql/
+COPY ./packages/ipfs/package.json ./packages/ipfs/
+COPY ./packages/validator/package.json ./packages/validator/
+COPY ./packages/messaging-client/package.json ./packages/messaging-client/
+COPY ./packages/linker-client/package.json ./packages/linker-client/
+COPY ./packages/eventsource/package.json ./packages/eventsource/
+COPY ./packages/services/package.json ./packages/services/
+COPY ./infra/discovery/package.json ./infra/discovery/
+COPY ./infra/messaging/package.json ./infra/messaging/
+COPY ./infra/ipfs-proxy/package.json ./infra/ipfs-proxy/
+COPY ./infra/notifications/package.json ./infra/notifications/
+COPY ./infra/tests/package.json ./infra/tests/
+COPY ./infra/growth/package.json ./infra/growth/
+COPY ./infra/identity/package.json ./infra/identity/
+COPY ./infra/token/package.json ./infra/token/
+COPY ./infra/bridge/package.json ./infra/bridge/
 COPY ./scripts/ ./scripts/
 
 # Complete contracts source needs to be available so that `truffle compile contracts`
 # which is calleed by the prepare script can succeed
-COPY ./origin-contracts ./origin-contracts
-COPY ./origin-js ./origin-js
+COPY ./packages/contracts ./packages/contracts
+COPY ./packages/origin-js ./packages/origin-js
 
 # Running of postinstall script requires --unsafe-perm
 RUN npm install --unsafe-perm
 
-# Build origin-js for event-listener
-RUN npm run build --prefix origin-js
+# Build js for event-listener
+RUN npm run build --prefix packages/origin-js

README.md

@@ -10,19 +10,71 @@ Visit our [Developer's page](https://www.originprotocol.com/developers) to learn
 
 You can see the Origin ecosystem in action at https://dapp.originprotocol.com.
 
-## Our Projects:
-
-* Our [DApp](https://github.com/OriginProtocol/origin/tree/master/origin-dapp) is the first marketplace powered by the Origin Protocol. Make the code your own!
-* [origin-graphql](https://github.com/OriginProtocol/origin/tree/master/origin-graphql) lets you quickly build   DApps and server applications that interact with the protocol.
-* The [discovery server](https://github.com/OriginProtocol/origin/tree/master/origin-discovery) provides a GraphQL service for quickly searching Origin blockchain data.
-* Our [bridge server](https://github.com/OriginProtocol/origin/tree/master/origin-bridge) provides identity attestations.
-* Run every part of our infrastructure locally for development and testing with [Docker Compose](https://github.com/OriginProtocol/origin/blob/master/DEVELOPMENT.md#running-docker-compose). 
-* Finally, we have a [testnet faucet](https://github.com/OriginProtocol/origin/tree/master/origin-faucet) and a distributed [messaging hub](https://github.com/OriginProtocol/origin/tree/master/origin-messaging).
-
 ## Development
 
 Ready to get started? Have a look at our [developer quickstart](DEVELOPMENT.md) and our [contributing guidelines](CONTRIBUTING.md).
 
+## This repository
+
+This repository is a monorepo containing many [npm packages](https://www.npmjs.com/). It is managed using [Lerna.](https://github.com/lerna/lerna)
+
+### Core packages
+
+These packages are used to build DApps on Origin.
+
+| Package | Description |
+|---------|-------------|
+| [`@origin/contracts`](/packages/contracts) | Smart contracts|
+| [`@origin/eventsource`](/packages/eventsource) | Derives current state of listings and offers from contract events |
+| [`@origin/graphql`](/packages/graphql) | GraphQL interface to Origin Protocol |
+| [`@origin/ipfs`](/packages/ipfs) | Convenience methods for getting and setting data in IPFS |
+| [`@origin/linker-client`](/packages/linker-client) | Client for mobile linking |
+| [`@origin/messaging-client`](/packages/messaging-client) | Client for Origin messaging|
+| [`@origin/origin-js`](/packages/origin-js) | (DEPRECATED) Library for interacting with smart contracts |
+| [`@origin/services`](/packages/services) | Utility package for running Ganache and IPFS |
+| [`@origin/token`](/packages/token) | Package for manipulating Origin Tokens (OGN)|
+| [`@origin/validator`](/packages/validator) | JSON Schema validation |
+
+### DApp packages
+
+Example DApps that we have built.
+
+| Package | Description |
+|---------|-------------|
+| [`@origin/admin`](/dapps/contracts) | DApp similar to `@origin/marketplace` but exposes more functionality |
+| [`@origin/graphql-simple-demo`](/dapps/eventsource) | Example of building a DApp with `@origin/graphql` |
+| [`@origin/marketplace`](/dapps/graphql) | Our marketplace DApp |
+
+### Infrastructure packages
+
+Servers and packages that provide extra functionality to DApps (e.g. search or attestations).
+
+| Package | Description |
+|---------|-------------|
+| [`@origin/bridge`](/infra/bridge) | Server providing attestation services |
+| [`@origin/cron`](/infra/cron) | Runs background tasks |
+| [`@origin/dapp-creator-client`](/infra/dapp-creator-client) | Client that generates configs for `@origin/marketplace` |
+| [`@origin/dapp-creator-server`](/infra/dapp-creator-server) | Server that generates configs for `@origin/marketplace` |
+| [`@origin/discovery`](/infra/discovery) | Provides search features to `@origin/marketplace` |
+| [`@origin/faucet`](/infra/faucet) | Token faucet |
+| [`@origin/growth`](/infra/growth) | Growth engine |
+| [`@origin/identity`](/infra/identity) | Database models for storing identity |
+| [`@origin/ipfs-proxy`](/infra/ipfs-proxy) | Layer between IPFS and clients to prevent malicious use |
+| [`@origin/linking`](/infra/linking) | Linking server for mobile integration|
+| [`@origin/messaging`](/infra/messaging) | Messaging server |
+| [`@origin/notifications`](/infra/notifications) | Delivers in browser notifications |
+| [`@origin/tests`](/infra/tests) | Runs integration tests in Docker Compose |
+| [`@origin/token-transfer-client`](/infra/token-transfer-client) | Client for delivering tokens |
+| [`@origin/token-transfer-server`](/infra/token-transfer-server) | Server for delivering tokens |
+
+
+### Mobile
+
+| Package | Description |
+|---------|-------------|
+| [`@origin/mobile`](/mobile) | Mobile application |
+
+
 ## Contributing
 
 Origin is an 100% open-source and community-driven project and we welcome contributions of all sorts. There are many ways to help, from reporting issues, contributing code, and helping us improve our community.