Skip to content

Commit

Permalink
docs: Add introduction page (#431)
Browse files Browse the repository at this point in the history 
New documentation providing an overview of the repository, main
practices, and links to key sections of the documentation.

---------

Co-authored-by: Pierre Slamich <[email protected]>
Co-authored-by: Github copilot
  • Loading branch information
@alexgarel @teolemon
alexgarel and teolemon authored Oct 17, 2024
1 parent c087cee commit 7b586a7
Show file tree
Hide file tree
Showing 8 changed files with 200 additions and 101 deletions.
22 changes: 11 additions & 11 deletions .github/labeler.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ GitHub Actions:
- changed-files:
- any-glob-to-any-file: '.github/**/*'

🪦 postmortems:
"🪦 postmortems":
- changed-files:
- any-glob-to-any-file: 'docs/reports/**/*'

Expand Down Expand Up @@ -49,20 +49,20 @@ moji:
- changed-files:
- any-glob-to-any-file: '**/*moji*'

🔒 SSL:
"🔒 SSL":
- changed-files:
- any-glob-to-any-file: '**/*ssl*'

🐳 docker:
"🐳 docker":
- changed-files:
- any-glob-to-any-file: '**/*docker*'

📨 email:
"📨 email":
- changed-files:
- any-glob-to-any-file: 'docs/mail.md'
- any-glob-to-any-file: 'confs/common/systemd/system/[email protected]'

📈 Matomo:
"📈 Matomo":
- changed-files:
- any-glob-to-any-file: 'docs/matomo.md'

Expand All @@ -82,26 +82,26 @@ proxmox:
- changed-files:
- any-glob-to-any-file: '**/*proxmox*'

🧴 Open Beauty Facts:
"🧴 Open Beauty Facts":
- changed-files:
- any-glob-to-any-file: '**/*obf*'

🪶 Apache:
"🪶 Apache":
- changed-files:
- any-glob-to-any-file: '**/*apache*'

Odoo:
- changed-files:
- any-glob-to-any-file: '**/*odoo*'

🏷️ Folksonomy Project:
"🏷️ Folksonomy Project":
- changed-files:
- any-glob-to-any-file: '**/*folksonomy*'

📸 Open Products Facts:
"📸 Open Products Facts":
- changed-files:
- any-glob-to-any-file: '**/*openproductsfacts*'
observability:

"🔭 observability":
- changed-files:
- any-glob-to-any-file: 'docs/observability.md'
8 changes: 4 additions & 4 deletions .github/workflows/readme-writer.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ jobs:
- run: |
pip install -r scripts/readme-writer/requirements.txt
python scripts/readme-writer/readme_writer.py
cat README.md
- name: Commit README.md
cat docs/virtual-machines.md
- name: Commit docs/virtual-machines.md.md
uses: stefanzweifel/git-auto-commit-action@v5
with:
branch: develop
repository: .
file_pattern: README.md
commit_message: Update README.md
file_pattern: docs/virtual-machines.md
commit_message: Update docs/virtual-machines.md
commit_options: '--no-verify --signoff'
84 changes: 4 additions & 80 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,6 @@ Sysadmin repository for the various parts of the Open Food Facts infrastructure.
* We have a status page at https://status.openfoodfacts.org/, driven by https://github.com/openfoodfacts/openfoodfacts-upptime
* We also have a [specific repository regarding monitoring](https://github.com/openfoodfacts/openfoodfacts-monitoring)

## Current priorities

As of 2023 our current priorities are:

* server migration with hardware upgrade and a clean containerized install - and zfs syncs
* better encrypted two-way communication between data centers (stunnel + https)
* backups checks
* testing backups through staging
* with automated deployment of new clones
* monitoring backups
* better monitoring
* more/better dashboards
* more active checks (monitoring - alerts)
* less false positives in alerts
* GPU server for inference and possibly one for training (not hosted) - may

## Incident logs

We started logging incidents by server:
Expand All @@ -34,38 +18,8 @@ We started logging incidents by server:

## Documentation

Link to [Github Page](https://openfoodfacts.github.io/openfoodfacts-infrastructure/)

The infrastructure documentation is as follows:

- [Overview](./docs/overview.md)

- [Mail](./docs/mail.md) - servers mail setup
- [Free Datacenter](./docs/free-datacenter.md) - Data center with main production servers
- [Linux Server](./docs/linux-server.md) - servers general setup
- [Mail](./docs/mail.md) - servers mail setup
- [An introduction to ZFS](./docs/zfs-overview.md) - ZFS is much used in our infrastructure
- [Proxmox](./docs/proxmox.md) - about proxmox management
- [CICD](./docs/cicd.md) - continuous integration and deployment
- [Observability](./docs/observability.md) - doc on monitoring / logs / etc.
- [Docker Onboarding](./docs/docker_onboarding.md)
- [Docker Infrastructure](./docs/docker_architecture.md)
- [Virtual Machines](#virtual-machines)

The main services:
- [MongoDB](./docs/mongodb.md) the MongoDB database
- [Redis](./docs/redis.md) we also use Redis
- [Open Food Facts Query](./docs/openfoodfacts-query.md) service computing aggregations

Some services:

- [Discourse](./docs/discourse.md) for forum
- [NGINX reverse proxy](./docs/nginx-reverse-proxy.md) the reverse proxy for OVH services
- [Folksonomy](./docs/folksonomy.md) user editable labels and values
- [Matomo](./docs/matomo.md) for web analytics
- [Producers sftp](./docs/producers_sftp.md) to push product updates on producer platform
- [Zammad](./docs/zammad.md) for support
- [Odoo](./docs/odoo.md) the CRM
See our [Introduction](./docs/index.md),
also on [Github Page](https://openfoodfacts.github.io/openfoodfacts-infrastructure/)

Also look at all install and post-mortem reports in [docs/reports](./docs/reports/)

Expand All @@ -80,36 +34,6 @@ Also look at all install and post-mortem reports in [docs/reports](./docs/report
* The meeting will handle Agenda items first, and if time permits, collaborative bug triage.
</details>

## Requests

### Virtual Machines

<!-- This table is auto-generated by the readme-writer action -->
<!-- VM table -->
| Title |State | OS | CPU # | RAM | SSD (Local) | HDD (Remote) | Services |
|-------------------------------------------------------------------------------------------------------------------------------------------------|------|------------------------------|-----------------|---------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|---------------------|---------------------------------------------------------------------------------------|
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/200>Wordpress test CT [#200]</a> |open |Debian last Stable. | 3|[Explain if > 4 Gb.] |6 GB. | 0|Apache, PHP, Wordpress. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/159>Monitoring - VM (QEMU host for docker) [#159]</a> |open |Debian |* 4 CPUs |* 12G for we have influxdb and elastic-search that needs memory|* 30 Go disk (it is currently around 14G, but this will grow because we want to harvest more logs and more metrics)|* 50Go for ES backups|Docker, docker-compose |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/80>CT for new blog engine [#80]</a> |open |Debian stable. |3 CPU. |2 GB. |10 GB |-- |LAMP + wordpress. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/76>CT for Folksonomy Engine API dev [#76]</a> |open |Default to Debian last Stable.|2 |1 GB |12 GB. |- |PostgreSQL, Python3. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/37> Wild School Eco-Score project [#37]</a> |open |Debian 10 |4 |16 Gb |30 Gb |0 |MongoDB |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/36> slack-org [#36]</a> |open |Debian 10 |1 |1 Gb |10 Gb |None |Node.js |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/29> adminer-org [#29]</a> |open |Debian 10 |2 |512 Mb. |4 Gb or even less. |0 |Nginx, PHP, Adminer. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/28>Containers (x2) to build a replica set for OFF database [#28]</a>|open |Debian 10 |4 |32 GB |50 GB (DB = 20 GB). |0 |Mongodb. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/27> feedme-org [#27]</a> |open |Debian 10 |3 |3 Gb. |15 Gb. |0 |PostgreSQL, Node.js, Nginx. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/21> off-wiki-org [#21]</a> |open |Debian 10 |2 |3 Gb |14 Gb. |14 Gb |Apache, PHP, MySQL, Mediawiki. |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/124>VM for the Community Portal [#124]</a> |closed|Debian last Stable. |[Explain if > 4.]|[Explain if > 4 Gb.] |[Explain if > 32 Gb.] |[Explain if > 1 Tb.] |Python/Django, probably PostgreSQL, probably Apache and all Dockerized |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/123>VM for the Taxonomy Editor [#123]</a> |closed|Debian last Stable. |[Explain if > 4.]|[Explain if > 4 Gb.] |[Explain if > 32 Gb.] |[Explain if > 1 Tb.] |Python, probably PostgreSQL, probably Apache for lightweight API serving from Docker |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/71>New VM QEMU for prod docker containers [#71]</a> |closed|Debian 11 (stable) |8 |24 GB |256 GB. |- |Services deployed in production: |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/59> monitoring [#59]</a> |closed|Debian 11 |4 |32GB |64GB |500GB (ovh3 mount) |Docker: ElasticSearch (Kibana?, Logstash?), Grafana, InfluxDB, Prometheus, Alertmanager|
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/55> impactestimator-net [#55]</a> |closed|Debian 11 |1 |1GB |1Gb |0 |https://github.com/openfoodfacts/impactestimator |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/53> robotoff-ml [#53]</a> |closed|Debian 11 |8 |96GB (Tensorflow, ANN) |192GB [ML models] |100GB |Tensorflow + ElasticSearch |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/51> robotoff-net [#51]</a> |closed|Debian 11 |4 |16GB (DB 4GB, Services 8GB) |92GB |0GB |Robotoff API + Schedulers + Workers, PostgreSQL DB |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/45> mongo-dev [#45]</a> |closed|Debian 10 |2 |16GB |40GB | |MongoDB running in Docker |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/41> off-net [#41]</a> |closed|Debian 10 |4 |16GB (PO needs > 6GB) |192GB |0GB |ProductOpener frontend + backend, MongoDB, PostgreSQL, Memcached |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/40> robotoff-dev [#40]</a> |closed|Debian 10 |4 |8 Gb |32 Gb |100 Gb |robotoff, elastic search, tensorflow, postgresql |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/24> Matomo [#24]</a> |closed|Debian 10 |No idea. |No idea. |No idea. |No idea. |LAMP |
|<a href=https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/20> robotoff-org [#20]</a> |closed|Debian 10 |4 |8 Gb |32 Gb |100 Gb |robotoff, elastic search, tensorflow, postgresql |
<!-- VM table -->
## Virtual machines

<a href="https://github.com/openfoodfacts/openfoodfacts-infrastructure/issues/new?assignees=cquest&labels=container&template=vm-template.md&title="><img src="./scripts/add.png" style="background: transparent; vertical-align: middle" width="30"/>&nbsp;&nbsp;Request a VM</img></a>
See [docs/virtual-machines.md](./docs/virtual-machines.md)
Loading

0 comments on commit 7b586a7

Please sign in to comment.