From 4d437535a58f61381a49f96a659e8a06e2d7b5d5 Mon Sep 17 00:00:00 2001 From: Robert Lin Date: Wed, 10 Jan 2024 13:14:46 -0800 Subject: [PATCH] engineering/managed-services: add generated docs --- content/departments/engineering/index.md | 4 + .../engineering/managed-services/cloud-ops.md | 55 +++++++++++++ .../managed-services/cody-analytics.md | 49 +++++++++++ .../managed-services/gatekeeper.md | 47 +++++++++++ .../engineering/managed-services/index.md | 81 +++++++++++++++++++ .../managed-services/msp-testbed.md | 79 ++++++++++++++++++ .../engineering/managed-services/pings.md | 48 +++++++++++ .../engineering/managed-services/sams.md | 80 ++++++++++++++++++ .../managed-services/support-integration.md | 49 +++++++++++ .../managed-services/telemetry-gateway.md | 73 +++++++++++++++++ .../managed-services/incidents.md | 4 +- .../core-services/managed-services/pings.md | 2 + .../managed-services/platform.md | 37 +-------- .../managed-services/telemetry-gateway.md | 2 + 14 files changed, 576 insertions(+), 34 deletions(-) create mode 100644 content/departments/engineering/managed-services/cloud-ops.md create mode 100644 content/departments/engineering/managed-services/cody-analytics.md create mode 100644 content/departments/engineering/managed-services/gatekeeper.md create mode 100644 content/departments/engineering/managed-services/index.md create mode 100644 content/departments/engineering/managed-services/msp-testbed.md create mode 100644 content/departments/engineering/managed-services/pings.md create mode 100644 content/departments/engineering/managed-services/sams.md create mode 100644 content/departments/engineering/managed-services/support-integration.md create mode 100644 content/departments/engineering/managed-services/telemetry-gateway.md diff --git a/content/departments/engineering/index.md b/content/departments/engineering/index.md index 1c00adbeb864..c0345a68b155 100644 --- a/content/departments/engineering/index.md +++ b/content/departments/engineering/index.md @@ -38,6 +38,10 @@ The Engineering department at Sourcegraph consists of: - [Our software development lifecycle (SDLC)](sdlc.md) - [Bi-Weekly Status Updates](bi-weekly-updates.md) +## Resources + +- [Managed Services infrastructure](./managed-services/index.md) + ## What's in a feature? For every feature we ship, consider: diff --git a/content/departments/engineering/managed-services/cloud-ops.md b/content/departments/engineering/managed-services/cloud-ops.md new file mode 100644 index 000000000000..5773867d528a --- /dev/null +++ b/content/departments/engineering/managed-services/cloud-ops.md @@ -0,0 +1,55 @@ +# Cloud Ops Dashboard infrastructure operations + + + +This document describes operational guidance for Cloud Ops Dashboard infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|----------------------------------------------------------------------------------------------------------------------------| +| Service ID | [`cloud-ops`](https://github.com/sourcegraph/managed-services/blob/main/services/cloud-ops/service.yaml) | +| Owners | **cloud** | +| Service kind | Cloud Run service | +| Environments | [prod environment](#prod-environment) | +| Docker image | `us-central1-docker.pkg.dev/control-plane-5e9ee072/docker/apiserver` | +| Source code | [`github.com/sourcegraph/controller` - `cmd/apiserver`](https://github.com/sourcegraph/controller/tree/HEAD/cmd/apiserver) | + +## Environments + +### prod environment + +| PROPERTY | DETAILS | +|----------------|----------------------------------------------------------------------------------------------------| +| Project ID | [`cloud-ops-prod-dd32`](https://console.cloud.google.com/run?project=cloud-ops-prod-dd32) | +| Category | **internal** | +| Resources | [prod Redis](#prod-redis) | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=cloud-ops-prod-dd32) | +| Domain | [cloud-ops.sgdev.org](https://cloud-ops.sgdev.org) | +| Cloudflare WAF | ✅ | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| GCP project read access | [Entitle request for the 'Internal Services' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiNzg0M2MxYWYtYzU2MS00ZDMyLWE3ZTAtYjZkNjY0NDM4MzAzIiwidGhyb3VnaCI6Ijc4NDNjMWFmLWM1NjEtNGQzMi1hN2UwLWI2ZDY2NDQzODMwMyIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | +| GCP project write access | [Entitle request for the 'Internal Services' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiZTEyYTJkZDktYzY1ZC00YzM0LTlmNDgtMzYzNTNkZmY0MDkyIiwidGhyb3VnaCI6ImUxMmEyZGQ5LWM2NWQtNGMzNC05ZjQ4LTM2MzUzZGZmNDA5MiIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | + +#### prod Cloud Run + +| PROPERTY | DETAILS | +|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=cloud-ops-prod-dd32) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=cloud-ops-prod-dd32) | + +#### prod Redis + +| PROPERTY | DETAILS | +|----------|-------------------------------------------------------------------------------------------------------------------------| +| Console | [Memorystore Redis instances](https://console.cloud.google.com/memorystore/redis/instances?project=cloud-ops-prod-dd32) | diff --git a/content/departments/engineering/managed-services/cody-analytics.md b/content/departments/engineering/managed-services/cody-analytics.md new file mode 100644 index 000000000000..553ea87c6c0a --- /dev/null +++ b/content/departments/engineering/managed-services/cody-analytics.md @@ -0,0 +1,49 @@ +# Cody Analytics infrastructure operations + + + +This document describes operational guidance for Cody Analytics infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|--------------------------------------------------------------------------------------------------------------------| +| Service ID | [`cody-analytics`](https://github.com/sourcegraph/managed-services/blob/main/services/cody-analytics/service.yaml) | +| Owners | **cody-strat** | +| Service kind | Cloud Run service | +| Environments | [dev environment](#dev-environment) | +| Docker image | `us-central1-docker.pkg.dev/sourcegraph-dev/cody-analytics/service` | +| Source code | [`github.com/sourcegraph/cody-analytics` - `.`](https://github.com/sourcegraph/cody-analytics/tree/HEAD/.) | + +## Environments + +### dev environment + +| PROPERTY | DETAILS | +|----------------|--------------------------------------------------------------------------------------------------------| +| Project ID | [`cody-analytics-dev-bd34`](https://console.cloud.google.com/run?project=cody-analytics-dev-bd34) | +| Category | **test** | +| Resources | | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=cody-analytics-dev-bd34) | +| Domain | [cody-analytics.sgdev.org](https://cody-analytics.sgdev.org) | +| Cloudflare WAF | ✅ | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------| +| GCP project read access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | +| GCP project write access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | + +#### dev Cloud Run + +| PROPERTY | DETAILS | +|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=cody-analytics-dev-bd34) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=cody-analytics-dev-bd34) | diff --git a/content/departments/engineering/managed-services/gatekeeper.md b/content/departments/engineering/managed-services/gatekeeper.md new file mode 100644 index 000000000000..f58c3498bc70 --- /dev/null +++ b/content/departments/engineering/managed-services/gatekeeper.md @@ -0,0 +1,47 @@ +# Cody Gatekeeper infrastructure operations + + + +This document describes operational guidance for Cody Gatekeeper infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|------------------------------------------------------------------------------------------------------------| +| Service ID | [`gatekeeper`](https://github.com/sourcegraph/managed-services/blob/main/services/gatekeeper/service.yaml) | +| Owners | **cody-services** | +| Service kind | Cloud Run job | +| Environments | [prod environment](#prod-environment) | +| Docker image | `us.gcr.io/sourcegraph-dev/abuse-ban-bot` | +| Source code | [`github.com/sourcegraph/abuse-ban-bot` - `.`](https://github.com/sourcegraph/abuse-ban-bot/tree/HEAD/.) | + +## Environments + +### prod environment + +| PROPERTY | DETAILS | +|------------|-----------------------------------------------------------------------------------------------------| +| Project ID | [`gatekeeper-prod-1c93`](https://console.cloud.google.com/run/jobs?project=gatekeeper-prod-1c93) | +| Category | **test** | +| Resources | | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=gatekeeper-prod-1c93) | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------| +| GCP project read access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | +| GCP project write access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | + +#### prod Cloud Run + +| PROPERTY | DETAILS | +|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run job](https://console.cloud.google.com/run/jobs?project=gatekeeper-prod-1c93) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_job%22;summaryFields=labels%252F%2522run.googleapis.com%252Fexecution_name%2522,jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=gatekeeper-prod-1c93) | diff --git a/content/departments/engineering/managed-services/index.md b/content/departments/engineering/managed-services/index.md new file mode 100644 index 000000000000..fea9a4d23aee --- /dev/null +++ b/content/departments/engineering/managed-services/index.md @@ -0,0 +1,81 @@ +# Managed Services infrastructure + + + +These pages contain generated operational guidance for the infrastructure of [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md) services. +This includes information about each service, configured environments, Entitle requests, common tasks, monitoring, etc. +In addition to service-specific guidance, [General guidance](#general-guidance) is also available. + +MSP is owned by [Core Services](../teams/core-services/index.md), but individual teams are responsible for the services they operate on the platform. + +Services are defined in [`sourcegraph/managed-services`](https://github.com/sourcegraph/managed-services), though service source code may live elsewhere. + +> [!NOTE] +> This page may be out of date if a service or environment was recently added or updated - reach out to #discuss-core-services for help updating these pages. + +## Customer Support + +Managed Services Platform services owned by `Customer Support`: + +- [Support Integration](./support-integration.md) + +## cloud + +Managed Services Platform services owned by `cloud`: + +- [Cloud Ops Dashboard](./cloud-ops.md) + +## cody-services + +Managed Services Platform services owned by `cody-services`: + +- [Cody Gatekeeper](./gatekeeper.md) + +## cody-strat + +Managed Services Platform services owned by `cody-strat`: + +- [Cody Analytics](./cody-analytics.md) + +## core-services + +Managed Services Platform services owned by `core-services`: + +- [MSP Testbed](./msp-testbed.md) +- [Pings Service](./pings.md) +- [Sourcegraph Accounts](./sams.md) +- [Telemetry Gateway](./telemetry-gateway.md) + +## General guidance + +### Infrastructure access + +For MSP service environments other than `category: test`, access needs to be requested through Entitle. +Test environments are placed in the "Engineering Projects" GCP folder, which should have access granted to engineers by default. + +Entitle access to a production MSP project is most easily provisioned through the `mspServiceReader` and custom roles, which provide read-only and editing access respectively. +You can request access to a project in Entitle by following these steps: + +- Go to [app.entitle.io/request](https://app.entitle.io/request) and select **Specific Permission** +- Fill out the following: + - Integration: **GCP Production Projects** + - Resource types: **Project** + - Resource: name of MSP project you are interested in + - Role: `mspServiceReader` (or `mspServiceEditor` if you need additional privileges - use with care!) + - Duration: choose your own adventure! + +These custom roles are configured [in in the infrastructure repo](https://github.com/sourcegraph/infrastructure/blob/main/gcp/custom-roles/msp.tf). + +### Terraform Cloud access + +Terraform Cloud (TFC) workspaces for MSP [can be found using the `msp` workspace tag](https://app.terraform.io/app/sourcegraph/workspaces?tag=msp). + +To gain access to MSP project TFC workspaces, [log in to Terraform Cloud](https://app.terraform.io/app/sourcegraph) and _then_ [request membership to the `Managed Services Platform Operators` TFC team via Entitle](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjM2MDAiLCJqdXN0aWZpY2F0aW9uIjoiRU5URVIgSlVTVElGSUNBVElPTiBIRVJFIiwicm9sZUlkcyI6W3siaWQiOiJiMzg3MzJjYy04OTUyLTQ2Y2QtYmIxZS1lZjI2ODUwNzIyNmIiLCJ0aHJvdWdoIjoiYjM4NzMyY2MtODk1Mi00NmNkLWJiMWUtZWYyNjg1MDcyMjZiIiwidHlwZSI6InJvbGUifV19). +This TFC team has access to all MSP workspaces, and is [configured here](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/infrastructure/-/blob/terraform-cloud/terraform.tfvars?L44:1-48:4). + +Note that you **must [log in to Terraform Cloud](https://app.terraform.io/app/sourcegraph) before making your Entitle request**. +If you make your Entitle request, then log in, you will be removed from any team memberships granted through Entitle by Terraform Cloud's SSO implementation. + +For more details, also see [creating and configuring services](https://github.com/sourcegraph/managed-services#operations). diff --git a/content/departments/engineering/managed-services/msp-testbed.md b/content/departments/engineering/managed-services/msp-testbed.md new file mode 100644 index 000000000000..cdef83850ff8 --- /dev/null +++ b/content/departments/engineering/managed-services/msp-testbed.md @@ -0,0 +1,79 @@ +# MSP Testbed infrastructure operations + + + +This document describes operational guidance for MSP Testbed infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|----------------------------------------------------------------------------------------------------------------------------------| +| Service ID | [`msp-testbed`](https://github.com/sourcegraph/managed-services/blob/main/services/msp-testbed/service.yaml) | +| Owners | **core-services** | +| Service kind | Cloud Run service | +| Environments | [test environment](#test-environment) | +| Docker image | `us.gcr.io/sourcegraph-dev/msp-example` | +| Source code | [`github.com/sourcegraph/sourcegraph` - `cmd/msp-example`](https://github.com/sourcegraph/sourcegraph/tree/HEAD/cmd/msp-example) | + +## Environments + +### test environment + +| PROPERTY | DETAILS | +|------------|-----------------------------------------------------------------------------------------------------------------------------------| +| Project ID | [`msp-testbed-test-77589aae45d0`](https://console.cloud.google.com/run?project=msp-testbed-test-77589aae45d0) | +| Category | **test** | +| Resources | [test Redis](#test-redis), [test PostgreSQL instance](#test-postgresql-instance), [test BigQuery dataset](#test-bigquery-dataset) | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=msp-testbed-test-77589aae45d0) | +| Domain | [msp-testbed.sgdev.org](https://msp-testbed.sgdev.org) | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------| +| GCP project read access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | +| GCP project write access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | + +#### test Cloud Run + +| PROPERTY | DETAILS | +|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=msp-testbed-test-77589aae45d0) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=msp-testbed-test-77589aae45d0) | + +#### test Redis + +| PROPERTY | DETAILS | +|----------|-----------------------------------------------------------------------------------------------------------------------------------| +| Console | [Memorystore Redis instances](https://console.cloud.google.com/memorystore/redis/instances?project=msp-testbed-test-77589aae45d0) | + +#### test PostgreSQL instance + +| PROPERTY | DETAILS | +|-----------|-------------------------------------------------------------------------------------------------------------| +| Console | [Cloud SQL instances](https://console.cloud.google.com/sql/instances?project=msp-testbed-test-77589aae45d0) | +| Databases | `primary` | + +To connect to the PostgreSQL instance in this environment, use `sg msp` in the [`sourcegraph/managed-services`](https://github.com/sourcegraph/managed-services) repository: + +```bash +# For read-only access +sg msp pg connect msp-testbed test + +# For write access - use with caution! +sg msp pg connect -write-access msp-testbed test +``` + +#### test BigQuery dataset + +| PROPERTY | DETAILS | +|-----------------|------------------------------------------------------------------------------------------------------------------------| +| Dataset Project | `msp-testbed-test-77589aae45d0` | +| Dataset ID | `msp_testbed` | +| Tables | [`example`](https://github.com/sourcegraph/managed-services/blob/main/services/msp-testbed/example.bigquerytable.json) | diff --git a/content/departments/engineering/managed-services/pings.md b/content/departments/engineering/managed-services/pings.md new file mode 100644 index 000000000000..785cf7c7b42f --- /dev/null +++ b/content/departments/engineering/managed-services/pings.md @@ -0,0 +1,48 @@ +# Pings Service infrastructure operations + + + +This document describes operational guidance for Pings Service infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|----------------------------------------------------------------------------------------------------------------------| +| Service ID | [`pings`](https://github.com/sourcegraph/managed-services/blob/main/services/pings/service.yaml) | +| Owners | **core-services** | +| Service kind | Cloud Run service | +| Environments | [prod environment](#prod-environment) | +| Docker image | `index.docker.io/sourcegraph/pings` | +| Source code | [`github.com/sourcegraph/sourcegraph` - `cmd/pings`](https://github.com/sourcegraph/sourcegraph/tree/HEAD/cmd/pings) | + +## Environments + +### prod environment + +| PROPERTY | DETAILS | +|------------|--------------------------------------------------------------------------------------------------------| +| Project ID | [`pings-prod-2f4f73edf1db`](https://console.cloud.google.com/run?project=pings-prod-2f4f73edf1db) | +| Category | **external** | +| Resources | | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=pings-prod-2f4f73edf1db) | +| Domain | [pings.sourcegraph.com](https://pings.sourcegraph.com) | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| GCP project read access | [Entitle request for the 'Managed Services ' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiYTQ4OWM2MDktNTBlYy00ODAzLWIzZjItMzYzZGJhMTgwMWJhIiwidGhyb3VnaCI6ImE0ODljNjA5LTUwZWMtNDgwMy1iM2YyLTM2M2RiYTE4MDFiYSIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | +| GCP project write access | [Entitle request for the 'Managed Services' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiODQzNTYxNzktZjkwMi00MDVlLTlhMTQtNTY3YTY1NmM5MzdmIiwidGhyb3VnaCI6Ijg0MzU2MTc5LWY5MDItNDA1ZS05YTE0LTU2N2E2NTZjOTM3ZiIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | + +#### prod Cloud Run + +| PROPERTY | DETAILS | +|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=pings-prod-2f4f73edf1db) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=pings-prod-2f4f73edf1db) | diff --git a/content/departments/engineering/managed-services/sams.md b/content/departments/engineering/managed-services/sams.md new file mode 100644 index 000000000000..cfaf7d90a1e0 --- /dev/null +++ b/content/departments/engineering/managed-services/sams.md @@ -0,0 +1,80 @@ +# Sourcegraph Accounts infrastructure operations + + + +This document describes operational guidance for Sourcegraph Accounts infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Service ID | [`sams`](https://github.com/sourcegraph/managed-services/blob/main/services/sams/service.yaml) | +| Owners | **core-services** | +| Service kind | Cloud Run service | +| Environments | [dev environment](#dev-environment) | +| Docker image | `us-central1-docker.pkg.dev/sourcegraph-dev/sams/accounts-server` | +| Source code | [`github.com/sourcegraph/accounts.sourcegraph.com` - `cmd/accounts-server`](https://github.com/sourcegraph/accounts.sourcegraph.com/tree/HEAD/cmd/accounts-server) | + +## Environments + +### dev environment + +| PROPERTY | DETAILS | +|----------------|-----------------------------------------------------------------------------------------------------------------------------| +| Project ID | [`sams-dev-bfec`](https://console.cloud.google.com/run?project=sams-dev-bfec) | +| Category | **test** | +| Resources | [dev Redis](#dev-redis), [dev PostgreSQL instance](#dev-postgresql-instance), [dev BigQuery dataset](#dev-bigquery-dataset) | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=sams-dev-bfec) | +| Domain | [accounts.sgdev.org](https://accounts.sgdev.org) | +| Cloudflare WAF | ✅ | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------| +| GCP project read access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | +| GCP project write access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | + +#### dev Cloud Run + +| PROPERTY | DETAILS | +|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=sams-dev-bfec) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=sams-dev-bfec) | + +#### dev Redis + +| PROPERTY | DETAILS | +|----------|-------------------------------------------------------------------------------------------------------------------| +| Console | [Memorystore Redis instances](https://console.cloud.google.com/memorystore/redis/instances?project=sams-dev-bfec) | + +#### dev PostgreSQL instance + +| PROPERTY | DETAILS | +|-----------|---------------------------------------------------------------------------------------------| +| Console | [Cloud SQL instances](https://console.cloud.google.com/sql/instances?project=sams-dev-bfec) | +| Databases | `accounts`, `cody_management` | + +To connect to the PostgreSQL instance in this environment, use `sg msp` in the [`sourcegraph/managed-services`](https://github.com/sourcegraph/managed-services) repository: + +```bash +# For read-only access +sg msp pg connect sams dev + +# For write access - use with caution! +sg msp pg connect -write-access sams dev +``` + +#### dev BigQuery dataset + +| PROPERTY | DETAILS | +|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Dataset Project | `sams-dev-bfec` | +| Dataset ID | `sams` | +| Tables | [`user_emails`](https://github.com/sourcegraph/managed-services/blob/main/services/sams/user_emails.bigquerytable.json), [`events`](https://github.com/sourcegraph/managed-services/blob/main/services/sams/events.bigquerytable.json) | diff --git a/content/departments/engineering/managed-services/support-integration.md b/content/departments/engineering/managed-services/support-integration.md new file mode 100644 index 000000000000..6e8ddbbd9ded --- /dev/null +++ b/content/departments/engineering/managed-services/support-integration.md @@ -0,0 +1,49 @@ +# Support Integration infrastructure operations + + + +This document describes operational guidance for Support Integration infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|------------------------------------------------------------------------------------------------------------------------------| +| Service ID | [`support-integration`](https://github.com/sourcegraph/managed-services/blob/main/services/support-integration/service.yaml) | +| Owners | **Customer Support** | +| Service kind | Cloud Run service | +| Environments | [prod environment](#prod-environment) | +| Docker image | `index.docker.io/sourcegraph/support-integration` | +| Source code | [`github.com/sourcegraph/support-integration` - `.`](https://github.com/sourcegraph/support-integration/tree/HEAD/.) | + +## Environments + +### prod environment + +| PROPERTY | DETAILS | +|----------------|---------------------------------------------------------------------------------------------------------------| +| Project ID | [`support-integration-prod-549b`](https://console.cloud.google.com/run?project=support-integration-prod-549b) | +| Category | **internal** | +| Resources | | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=support-integration-prod-549b) | +| Domain | [support-integration.sgdev.org](https://support-integration.sgdev.org) | +| Cloudflare WAF | ✅ | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| GCP project read access | [Entitle request for the 'Internal Services' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiNzg0M2MxYWYtYzU2MS00ZDMyLWE3ZTAtYjZkNjY0NDM4MzAzIiwidGhyb3VnaCI6Ijc4NDNjMWFmLWM1NjEtNGQzMi1hN2UwLWI2ZDY2NDQzODMwMyIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | +| GCP project write access | [Entitle request for the 'Internal Services' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiZTEyYTJkZDktYzY1ZC00YzM0LTlmNDgtMzYzNTNkZmY0MDkyIiwidGhyb3VnaCI6ImUxMmEyZGQ5LWM2NWQtNGMzNC05ZjQ4LTM2MzUzZGZmNDA5MiIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | + +#### prod Cloud Run + +| PROPERTY | DETAILS | +|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=support-integration-prod-549b) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=support-integration-prod-549b) | diff --git a/content/departments/engineering/managed-services/telemetry-gateway.md b/content/departments/engineering/managed-services/telemetry-gateway.md new file mode 100644 index 000000000000..a70dff94f40d --- /dev/null +++ b/content/departments/engineering/managed-services/telemetry-gateway.md @@ -0,0 +1,73 @@ +# Telemetry Gateway infrastructure operations + + + +This document describes operational guidance for Telemetry Gateway infrastructure. +This service is operated on the [Managed Services Platform (MSP)](../teams/core-services/managed-services/platform.md). + +If you need assistance with MSP infrastructure, reach out to the [Core Services](../teams/core-services/index.md) team in #discuss-core-services. + +## Service overview + +| PROPERTY | DETAILS | +|--------------|----------------------------------------------------------------------------------------------------------------------------------------------| +| Service ID | [`telemetry-gateway`](https://github.com/sourcegraph/managed-services/blob/main/services/telemetry-gateway/service.yaml) | +| Owners | **core-services** | +| Service kind | Cloud Run service | +| Environments | [dev environment](#dev-environment), [prod environment](#prod-environment) | +| Docker image | `index.docker.io/sourcegraph/telemetry-gateway` | +| Source code | [`github.com/sourcegraph/sourcegraph` - `cmd/telemetry-gateway`](https://github.com/sourcegraph/sourcegraph/tree/HEAD/cmd/telemetry-gateway) | + +## Environments + +### dev environment + +| PROPERTY | DETAILS | +|------------|-----------------------------------------------------------------------------------------------------------| +| Project ID | [`telemetry-gateway-dev-0050`](https://console.cloud.google.com/run?project=telemetry-gateway-dev-0050) | +| Category | **test** | +| Resources | | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=telemetry-gateway-dev-0050) | +| Domain | [telemetry-gateway.sgdev.org](https://telemetry-gateway.sgdev.org) | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------| +| GCP project read access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | +| GCP project write access | no Entitle request needed; all engineers have access to this environment through the 'Engineering Projects' folder | + +#### dev Cloud Run + +| PROPERTY | DETAILS | +|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=telemetry-gateway-dev-0050) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=telemetry-gateway-dev-0050) | + +### prod environment + +| PROPERTY | DETAILS | +|------------|------------------------------------------------------------------------------------------------------------| +| Project ID | [`telemetry-gateway-prod-acae`](https://console.cloud.google.com/run?project=telemetry-gateway-prod-acae) | +| Category | **external** | +| Resources | | +| Alerts | [GCP monitoring](https://console.cloud.google.com/monitoring/alerting?project=telemetry-gateway-prod-acae) | +| Domain | [telemetry-gateway.sourcegraph.com](https://telemetry-gateway.sourcegraph.com) | + +MSP infrastructure access needs to be requested using Entitle for time-bound privileges. +Test environments have less stringent requirements. + +| ACCESS | ENTITLE REQUEST TEMPLATE | +|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| GCP project read access | [Entitle request for the 'Managed Services ' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiYTQ4OWM2MDktNTBlYy00ODAzLWIzZjItMzYzZGJhMTgwMWJhIiwidGhyb3VnaCI6ImE0ODljNjA5LTUwZWMtNDgwMy1iM2YyLTM2M2RiYTE4MDFiYSIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | +| GCP project write access | [Entitle request for the 'Managed Services' folder](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjEwODAwIiwianVzdGlmaWNhdGlvbiI6IkVOVEVSIEpVU1RJRklDQVRJT04gSEVSRSIsInJvbGVJZHMiOlt7ImlkIjoiODQzNTYxNzktZjkwMi00MDVlLTlhMTQtNTY3YTY1NmM5MzdmIiwidGhyb3VnaCI6Ijg0MzU2MTc5LWY5MDItNDA1ZS05YTE0LTU2N2E2NTZjOTM3ZiIsInR5cGUiOiJyb2xlIn1dfQ%3D%3D) | + +#### prod Cloud Run + +| PROPERTY | DETAILS | +|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Console | [Cloud Run service](https://console.cloud.google.com/run?project=telemetry-gateway-prod-acae) | +| Logs | [GCP logging](https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%20-logName%3D~%22logs%2Frun.googleapis.com%252Frequests%22;summaryFields=jsonPayload%252FInstrumentationScope,jsonPayload%252FBody,jsonPayload%252FAttributes%252Ferror:false:32:end?project=telemetry-gateway-prod-acae) | diff --git a/content/departments/engineering/teams/core-services/managed-services/incidents.md b/content/departments/engineering/teams/core-services/managed-services/incidents.md index ccbbaaea7dbe..60f9c5241ec4 100644 --- a/content/departments/engineering/teams/core-services/managed-services/incidents.md +++ b/content/departments/engineering/teams/core-services/managed-services/incidents.md @@ -2,7 +2,7 @@ This page includes incident response playbooks the [Core Services team](../index.md) can use when operating the [Managed Services Platform](./platform.md) fleet. -**For more MSP user/operator-oriented guidance, refer to the [operations guide](./platform.md#operating-services) instead**. +**For more MSP user/operator-oriented guidance, refer to the [Managed Services infrastructure](../../../managed-services/index.md) pages instead**. ## Basics @@ -28,6 +28,8 @@ Quick links and brief summary below - for more details refer to [the more genera - `catogory: test` services: All engineers should have access by default (test services are placed in the `Engineering Projects` folder) - `mspServiceEditor` and `mspServiceReader` are available for convenience, and are configured [in `gcp/org/customer-roles/msp.tf` in the infrastructure repo](https://github.com/sourcegraph/infrastructure/blob/main/gcp/custom-roles/msp.tf). Additional roles can be requested directly via Entitle. +Service-specific guidance is generated in [Managed Services infrastructure](../../../managed-services/index.md) pages. + ### Changing infrastructure #### CLI-apply mode diff --git a/content/departments/engineering/teams/core-services/managed-services/pings.md b/content/departments/engineering/teams/core-services/managed-services/pings.md index 88e8a677f4b6..a8d4e2b4ff83 100644 --- a/content/departments/engineering/teams/core-services/managed-services/pings.md +++ b/content/departments/engineering/teams/core-services/managed-services/pings.md @@ -5,6 +5,8 @@ The Pings service is the service that collects [ping requests](https://docs.sour - As of 5.2.0, ping requests are sent to `https://pings.sourcegraph.com/updates` directly. - For Sourcegraph instances that prior to 5.2.0, all ping requests sent to `https://sourcegraph.com/.api/updates` are being transparently forwarded to `https://pings.sourcegraph.com/updates`. +> NOTE: There is now a generated operations reference for the MSP-based Pings service in [Pings infrastructure](../../../managed-services/pings.md), generated by `sg msp operations` - as the generated docs improve, some content from this page will be migrated away. + ## Service images Source code for Pings service is in [sourcegraph/sourcegraph/cmd/pings](https://github.com/sourcegraph/sourcegraph/tree/main/cmd/pings). The image gets built the same way as any other Sourcegraph service, i.e. with `insiders`, the standard `main`-branch and `main-dry-run` tags. diff --git a/content/departments/engineering/teams/core-services/managed-services/platform.md b/content/departments/engineering/teams/core-services/managed-services/platform.md index 4d493839e7c0..44a930304639 100644 --- a/content/departments/engineering/teams/core-services/managed-services/platform.md +++ b/content/departments/engineering/teams/core-services/managed-services/platform.md @@ -18,7 +18,7 @@ MSP supports single-container: From a simple service configuration YAML ([examples](https://github.com/sourcegraph/managed-services/tree/main/services)) and the `sg msp` toolchain for managing configuration, we currently support: -- Generating infrastructure-as-code, deployed via [Terraform Cloud](#terraform-cloud) +- Generating infrastructure-as-code, deployed via Terraform Cloud - [Service initialization and runtime boilerplate](#building-a-new-service) via [sourcegraph/lib/managedservicesplatform](https://github.com/sourcegraph/sourcegraph/tree/main/lib/managedservicesplatform), which includes: - initialization of OpenTelemetry tracing and metrics, logging, and error reporting - integration guidance for provisioned data backends like Redis and PostgreSQL @@ -35,6 +35,7 @@ From a simple service configuration YAML ([examples](https://github.com/sourcegr - Commands for easy access to infrastructure - Shortcuts to relevant UIs in `sg msp tfc view`, `sg msp logs`, etc. - Securely connect to your PostgreSQL instance using `sg msp pg connect` +- Generated infrastructure guidance, rendered in the [Managed Services infrastructure](../../../managed-services/index.md) pages. See [our GitHub roadmap](https://github.com/orgs/sourcegraph/projects/375/views/1) and [2023 Q3 Managed Services Platform (MSP) proof-of-concept update](https://docs.google.com/document/d/1DSqKqCgXW2m0TCVBmDSasY2Hxb9cp9Uv_NgF4MEfAto/edit) for more details on things we will be adding to MSP. @@ -68,35 +69,5 @@ Refer to the [sourcegraph/managed-services README](https://github.com/sourcegrap ## Operating services -This is a user/operator-oriented guide. -Guidance for MSP incidents is available in [Managed Services incident response](./incidents.md). - -### Infrastructure access - -For MSP service environments other than `category: test`, access needs to be requested through Entitle. -The test environment ("Engineering Projects" GCP folder) should have access granted to engineers by default. - -Entitle access to a production MSP project is most easily provisioned through the `mspServiceReader` and `mspServiceEditor` custom roles, which provide read-only and editing access respectively. -You can request access to a project in Entitle by following these steps: - -- Go to [app.entitle.io/request](https://app.entitle.io/request) and select **Specific Permission** -- Fill out the following: - - Integration: `GCP Production Projects` - - Resource types: `Project` - - Resource: name of MSP project you are interested in - - Role: `mspServiceReader` (or `mspServiceEditor` if you need additional privileges - use with care!) - - Duration: choose your own adventure! - -These custom roles are configured [in `gcp/org/customer-roles/msp.tf` in the infrastructure repo](https://github.com/sourcegraph/infrastructure/blob/main/gcp/custom-roles/msp.tf). - -### Terraform Cloud - -Terraform Cloud (TFC) workspaces for MSP [can be found using the `msp` workspace tag](https://app.terraform.io/app/sourcegraph/workspaces?tag=msp). - -To gain access to MSP project TFC workspaces, [log in to Terraform Cloud](https://app.terraform.io/app/sourcegraph) and _then_ [request membership to the `Managed Services Platform Operators` TFC team via Entitle](https://app.entitle.io/request?data=eyJkdXJhdGlvbiI6IjM2MDAiLCJqdXN0aWZpY2F0aW9uIjoiRU5URVIgSlVTVElGSUNBVElPTiBIRVJFIiwicm9sZUlkcyI6W3siaWQiOiJiMzg3MzJjYy04OTUyLTQ2Y2QtYmIxZS1lZjI2ODUwNzIyNmIiLCJ0aHJvdWdoIjoiYjM4NzMyY2MtODk1Mi00NmNkLWJiMWUtZWYyNjg1MDcyMjZiIiwidHlwZSI6InJvbGUifV19). -This TFC team has access to all MSP workspaces, and is [configured here](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/infrastructure/-/blob/terraform-cloud/terraform.tfvars?L44:1-48:4). - -Note that you **must [log in to Terraform Cloud](https://app.terraform.io/app/sourcegraph) before making your Entitle request**. -If you make your Entitle request, then log in, you will be removed from any team memberships granted through Entitle by Terraform Cloud's SSO implementation. - -For more details, also see [creating and configuring services](https://github.com/sourcegraph/managed-services#operations). +- **Guidance for service operators** is available in the [Managed Services infrastructure](../../../managed-services/index.md) pages. +- **Guidance for broad MSP incidents** is available in [Managed Services incident response](./incidents.md) - this is generally intended for Core Services. diff --git a/content/departments/engineering/teams/core-services/managed-services/telemetry-gateway.md b/content/departments/engineering/teams/core-services/managed-services/telemetry-gateway.md index dcdcb402805c..0ab014c19732 100644 --- a/content/departments/engineering/teams/core-services/managed-services/telemetry-gateway.md +++ b/content/departments/engineering/teams/core-services/managed-services/telemetry-gateway.md @@ -6,6 +6,8 @@ The Telemetry Gateway service is the service that ingests [telemetry v2 events]( - For Sourcegraph instances that prior to 5.2.0, no events are exported. A [custom mechanism did exist for exporting events specifically from Cloud instances](https://docs.sourcegraph.com/dev/background-information/data-usage-pipeline) based on individual service agreements with customers - the new telemetry events will supersede this mechanism. +> NOTE: There is now a generated operations reference for the MSP-based Telemetry Gateway service in [Telemetry Gateway infrastructure](../../../managed-services/telemetry-gateway.md), generated by `sg msp operations` - as the generated docs improve, some content from this page will be migrated away. + ## Service images Source code for Telemetry Gateway service is in [sourcegraph/sourcegraph/cmd/telemetry-gateway](https://github.com/sourcegraph/sourcegraph/tree/main/cmd/telemetry-gateway).