Merge branch 'master' into update-example

content/community/events.md

@@ -9,16 +9,15 @@ g3Teaser:
   detail: About every other month, there is a virtual Gen3 Forum for the community of Gen3 developers, operators, sponsors and users of Gen3 data platforms. These events aim to share information about how to set up new Gen3 instances, build a community that can help each other, and create clear paths for support from the Gen3 core development team.
 
 g3Upcoming:
-- Title: Testing code with the new Gen3 testing framework
-  Date: November 6, 3:30-5:00 pm CDT; November 7, 2024, 8:30-10:00 am AEDT
-  Description: Gen3 values code contributions from our open-source community. However, the continuous integration testing available within our GitHub repositories has generally not been available to contributors external to CTDS.  This has made it difficult for contributors to assess their code changes and get them merged into the Gen3 codebase.  We will discuss in the forum the new Gen3 testing framework, which makes these tests available to anyone.  We will also discuss how contributors can create and contribute their own tests as part of their pull requests.  Speakers will include Peter Vassilatos, Director of Engineering, and Hara Juvvala, Principal Software Engineer in Test, both from the Center for Translational Data Science at the University of Chicago.
-  Image: /figs/gen3_new_logo.png
-  Register: https://uchicago.zoom.us/meeting/register/tJ0ofuyhrTgrE9CPuv6-oBNcLYp-JLDp67-D
-
 
 
 
 g3past:
+- Title: Testing code with the new Gen3 testing framework
+  Date: November 6, 3:30-5:00 pm CDT; November 7, 2024, 8:30-10:00 am AEDT
+  Description: Gen3 values code contributions from our open-source community. However, the continuous integration testing available within our GitHub repositories has generally not been available to contributors external to CTDS.  This has made it difficult for contributors to assess their code changes and get them merged into the Gen3 codebase.  We will discuss in the forum the new Gen3 testing framework, which makes these tests available to anyone.  We will also discuss how contributors can create and contribute their own tests as part of their pull requests.  Speakers will include Peter Vassilatos, Director of Engineering, and Hara Juvvala, Principal Software Engineer in Test, both from the Center for Translational Data Science at the University of Chicago.
+  Youtube: Ea3tkR_D8VM
+  Slides: Gen3 Forum November 2024 - Testing framework.pdf
 - Title: GA4GH standards in Gen3
   Date: September 18, 4:30-5:30 pm CDT; September 19, 2024, 7:30-8:30am AEST
   Description: GA4GH is an international standards setting body for Genomics and Health. Gen3 aims to follow GA4GH standards whenever possible in order to improve our interoperability with other systems and to simplify the use of a Gen3 Data Commons. Join us at our next community forum where we will provide an overview of our GA4GH compliant services and share plans for the future. Speakers will include Robert Grossman and Michael Lukowski from the University of Chicago, Center for Translational Data Science and Kyle Ellrott from the Oregon Health and Science University.
@@ -84,11 +83,13 @@ g3past:
   </div>
 </section>
 
+<!--
 <section class="g3-space__padding-sm-top g3-space__padding-sm-bottom">
     <div class="g3-inner-wrapper">
         <h2>Upcoming Events</h2>
     </div>
 </section>
+-->
 
 {{< events "g3Upcoming" >}}
 

gen3/docs/blog/authors.yml

@@ -1,17 +1,9 @@
 authors:
   qureshi:
     name: Jawad Qureshi
-    description: Lead Platform Engineer
+    description: Lead Platform Engineer, CTDS
     avatar: https://github.com/jawadqur.png
-
-# yangshun:
-#   name: Yangshun Tay
-#   title: Front End Engineer @ Facebook
-#   url: https://github.com/yangshun
-#   image_url: https://github.com/yangshun.png
-
-# slorber:
-#   name: Sébastien Lorber
-#   title: Docusaurus maintainer
-#   url: https://sebastienlorber.com
-#   image_url: https://github.com/slorber.png
+  elise:
+    name: Elise Castle
+    description: Platform Engineer, CTDS
+    avatar: https://github.com/EliseCastle23.png

gen3/docs/blog/posts/gen3-laptop.md

@@ -6,6 +6,7 @@ date: 2023-09-13
 slug: Gen3 on laptop
 categories:
   - Operator
+  - CTDS
 tags:
   - Kubernetes
   - Minikube

gen3/docs/blog/posts/gen3-observability.md

@@ -0,0 +1,83 @@
+---
+title: "Observability"
+slug: observability
+authors:
+ - elise
+date: 2024-10-15
+categories:
+  - Operator
+  - CTDS
+tags:
+  - Observability
+  - Helm Chart
+  - Grafana
+  - Loki
+  - Mimir
+  - Alloy
+  - Faro Collector
+  - Real User Monitoring (RUM)
+  - Metrics
+  - Log Aggregation
+  - Kubernetes
+  - Dashboards
+  - Kubernetes Monitoring
+  - Time-Series Database
+  - Alerting
+  - Frontend Monitoring
+  - Grafana Dashboards
+  - Open Source Monitoring
+---
+
+# Deploying a Comprehensive Observability Stack with Helm
+Monitoring and observability are essential for maintaining modern infrastructure and applications. With the new Observability Helm Chart, setting up a robust monitoring system is easier than ever. This chart provides an integrated stack featuring Grafana for visualizations, Loki for log aggregation, and Mimir for metrics storage and querying. Alloy can then be deployed in any cluster to collect logs and metrics to foward to Loki and Mimir. Additionally, you can optionally deploy the Faro Collector Helm Chart to further enhance observability by supporting Real User Monitoring (RUM) via the Fence Service.
+
+## Overview of the Observability Helm Chart
+The Observability Helm Chart deploys a complete observability solution to your Kubernetes cluster. It bundles three core components:
+
+### Grafana:
+An industry-leading visualization platform that allows users to create dashboards, track metrics, and set alerts.
+### Mimir:
+A scalable time-series database optimized for efficiently storing and querying metrics across applications and infrastructure.
+### Loki:
+A log aggregation system designed to index and query logs with minimal resource usage, seamlessly integrating with Grafana.
+
+## General Architecture
+
+In this setup, Loki and Mimir are configured with internal ingress resources, enabling Alloy to send metrics and logs securely via VPC peering connections. Both Loki and Mimir write the ingested data to Amazon S3 for scalable and durable storage. This data can be queried and visualized through Grafana, which is hosted behind an internet-facing ingress. Access to Grafana can be restricted using CIDR ranges defined through the ALB ingress annotation: alb.ingress.kubernetes.io/inbound-cidrs: "cidrs". Additionally, the chart supports SAML authentication for Grafana, configured through the grafana.ini field, ensuring secure user access.
+
+![Grafana architecutre](Grafana.png)
+
+
+### Fips compliant images
+
+Gen3 provides FIPS-compliant images, which are set as the default in the values file for Grafana, Mimir, and Loki. These images are self-hosted and maintained by the Gen3 Platform Team, ensuring secure and compliant operations. The Platform Team is responsible for managing image upgrades, and service versions will be updated as deemed necessary by the team.
+
+### Built-in Gen3 Alerts
+
+This Helm chart comes equipped with built-in Gen3 alerts, defined in the 'alerting' section of the values.yaml. These alerts enable you to immediately leverage your logs and metrics as soon as Grafana is up and running.
+
+### Built-in Gen3 Dashboards
+
+We'll soon be releasing Gen3 dashboards, providing users with Gen3-specific visualizations. Please check back here to see if they have been released.
+
+## Alloy and Faro: Enhancing Observability
+
+### Alloy:
+Collects logs and metrics from your services and sends them to Loki and Mimir for storage and analysis. Alloy acts as a bridge between your services and the observability stack, ensuring data flows smoothly to the right destinations.
+### Faro Collector:
+A specialized configuration of Alloy designed to collect Real User Monitoring (RUM) data from Grafana Faro. This setup captures frontend metrics.
+
+## Helm Charts Overview
+Observability Helm Chart: Deploys Grafana, Loki, and Mimir as the foundation of your observability platform.
+
+Alloy Helm Chart: Configures Alloy to collect logs and metrics and forward them to Loki and Mimir. Alloy can be deployed in a separate cluster or VPC or it can be deployed in multiple clusters/vpcs.
+
+Faro Collector Helm Chart: Adds RUM data collection to the stack by configuring Alloy to receive frontend metrics from Grafana Faro.
+
+## Conclusion
+This new suite of Helm charts provides everything you need to monitor your Gen3 instance.
+
+To see detailed instructions on how to set up these charts, please refer to the following links:
+- [observability.md](https://github.com/uc-cdis/gen3-docs/blob/main/docs/tutorials/observability.md)
+- [alloy.md](https://github.com/uc-cdis/gen3-docs/blob/main/docs/tutorials/alloy.md)
+- [faro.md](https://github.com/uc-cdis/gen3-docs/blob/main/docs/tutorials/faro.md)

gen3/docs/blog/posts/k8s-for-dev.md

@@ -5,6 +5,7 @@ authors:
 date: 2023-08-13
 categories:
   - Operator
+  - CTDS
 slug: k8s for development
 tags:
   - Kubernetes

gen3/docs/blog/posts/k8s-productivity-tools.md

@@ -6,6 +6,7 @@ date: 2023-10-13
 slug: k8s tools
 categories:
   - Operator
+  - CTDS
 tags:
   - Kubernetes
   - Minikube

gen3/docs/gen3-resources/developer-guide/architecture-new.md

@@ -86,8 +86,10 @@ Analysis tools are developed using React, Gen3 core/frontend packages, and custo
 
 ## Architectural Diagrams
 
-### Gen3 Graph Data Flow
+### Architecture Overview
+![Gen3 Architecture Overview](/gen3-resources/developer-guide/img/Gen3_architecture_overview.png)
 
+### Gen3 Graph Data Flow
 ![Gen3 Graph Data Flow](/gen3-resources/developer-guide/img/Gen3 graph data flow.png)
 
 

gen3/docs/gen3-resources/developer-guide/microservices.md

@@ -1,55 +1,56 @@
+# Gen3 Microservices
 
 Gen3 features and functionality are enabled by independent and modular microservices.  These services can be developed, set up, and scaled independently. Microservices provide important advantages for large-scale systems that require scalability and must continue to evolve even as their code base grows very large, but increases the complexity of operating small-scale systems.
 
 While the average user does not need to know the details and names of each microservice, if you are interested in adding new features or modifying your Gen3 system in some way it may be helpful to have a deeper understanding of a specific microservice.  We have included brief descriptions below along with a link to their documentation in GitHub.
 
-### [Aggregated Metadata Service (AggMDS)][aggmds github]
+## [Aggregated Metadata Service (AggMDS)][aggmds github]
 The aggregated MDS is a service which caches metadata from commons metadata services and becomes a centralize API for browsing Metadata with clients such as the Ecosystem browser. The AggMDS holds the content viewable in a Data Portal Discovery page for a Data Mesh.
 
-### [Arborist][arborist github]
+## [Arborist][arborist github]
 Arborist is an attribute-based access control (ABAC) policy engine, designed for use with the Gen3 stack. Arborist tracks resources requiring access control, along with actions which users may perform to operate on these resources, and roles, which aggregate permissions to perform one or more actions
 
-### [Data Portal][data portal github]
+## [Data Portal][data portal github]
 The data portal service is an interactive website that allows users to explore, submit, and download data. The Windmill service utilizes the APIs offered by the data commons just as any other externally built app could.
 
-### [Fence][fence github]
+## [Fence][fence github]
 The Fence service controls access to the metadata, submission, indexing, and data itself. Fence is an authentication (AuthN) and authorization (AuthZ) service which utilizes OpenID Connect flow (an extension of OAuth2) to generate tokens for clients. It can also provide tokens directly to a user. Clients and users may then use those tokens (JWT) with other Gen3 Data Commons services to access protected endpoints that require specific permissions. Fence can be configured to support different Identity Providers (IDPs) for AuthN. At the moment, supported IDPs include Google, and Shibboleth supporting providers such as NIH iTrust.
 
-### [Guppy][guppy github]
+## [Guppy][guppy github]
 Server that support GraphQL queries on data from elasticsearch.
 
-### [Hatchery][hatchery github]
+## [Hatchery][hatchery github]
 Hatchery creates Kubernetes Pods for workspace services. Workspace services must expose HTTP servers. Ambassador is used to proxy user traffic through to their container workspace once it is launched by Hatchery.
 
-### [Helm][helm github]
+## [Helm][helm github]
 Gen3 relies upon Helm to manage installation and management of Kubernetes applications. Helm is used to build ”charts”, which are packages of Kubernetes resources that are used to deploy apps to a cluster. Helm is the recommended way to deploy Gen3.
 
-### [Manifest Service][manifest service github]
+## [Indexd][indexd github]
+The Indexd service provides permanent digital IDs for data objects. These IDs can be used to retrieve the data, or query the metadata associated with the object. The Indexd service tracks the locations and hash of every asset (file) in the data commons object store. It exports RESTful APIs for registering a new asset, and retrieving data for an existing asset.
+
+## [Manifest Service][manifest service github]
 This service handles reading from and writing to a user's S3 folder containing their manifests. A manifest is a JSON file that lists records a researcher may be interested in analyzing. This service stores a manifest to a user folder in an s3 bucket and delivers it for later use, such as when the researcher wants to mount the manifest in their workspace. If the "prefix" config variable is set, user folders will be stored in a directory of that name within the s3 bucket.
 
-### [Sheepdog][sheepdog github]
-The Sheepdog service is responsible for herding user submissions of metadata into the graph database. The submissions are quality controlled against the data dictionary to ensure all required fields are present and have appropriate data values. The Sheepdog service is also responsible for supporting bulk export of the metadata into TSV or JSON formats.
+## [Metadata Service (MDS)][MDS github]
+The Metadata Service provides an API for retrieving JSON metadata of GUIDs. It is a flexible option for "semi-structured" data (key:value mappings).  The content of the MDS powers the Data Portal Discovery Page for a Data Commons. The Gen3 SDK can be used to upload and edit the metadata.
 
-### [Peregrine][peregrine github]
+## [Peregrine][peregrine github]
 Peregrine is the high speed metadata seeking service which responds to GraphQL search queries. The GraphQL service allows Commons operators and users to precisely query only the information they are most interested in from the metadata collections. The service translates the GraphQL search into the appropriate statements which are run against the PostgreSQL backend before being returned as friendly JSON.
 
+## [Sheepdog][sheepdog github]
+The Sheepdog service is responsible for herding user submissions of metadata into the graph database. The submissions are quality controlled against the data dictionary to ensure all required fields are present and have appropriate data values. The Sheepdog service is also responsible for supporting bulk export of the metadata into TSV or JSON formats.
 
-### [Indexd][indexd github]
-The Indexd service provides permanent digital IDs for data objects. These IDs can be used to retrieve the data, or query the metadata associated with the object. The Indexd service tracks the locations and hash of every asset (file) in the data commons object store. It exports RESTful APIs for registering a new asset, and retrieving data for an existing asset.
-
-### [Metadata Service (MDS)][MDS github]
-The Metadata Service provides an API for retrieving JSON metadata of GUIDs. It is a flexible option for "semi-structured" data (key:value mappings).  The content of the MDS powers the Data Portal Discovery Page for a Data Commons. The Gen3 SDK can be used to upload and edit the metadata.
-
-### [Sower][sower github]
+## [Sower][sower github]
 Sower dispatches Kubernetes jobs.
 
-### [Tube][tube github]
+## [Tube][tube github]
 Microservice that controls the ETL process of structured data.
 
-### [Workspace Token Service][wts github]
+## [Workspace Token Service][wts github]
 The Gen3 workspace token service acts as an OIDC client which acts on behalf of users to request refresh tokens from Fence. This happens when a user logs into a workspace from the browser. WTS then stores the refresh token for that user, and manages access tokens and refresh tokens for workers that belong to specific users in the workspace.
 
-### Microservice NGINX Route Table
+
+# Microservice NGINX Route Table
 
 This table is helpful for debugging errors in front-end applications like [Windmill: data portal](https://github.com/uc-cdis/data-portal) or other Gen3 clients. You can easily identify the running service that is returning an error, based on its absolute HTTP request path. [Source](https://github.com/uc-cdis/cloud-automation/tree/master/kube/services/revproxy/gen3.nginx.conf).
 
@@ -72,6 +73,7 @@ This table is helpful for debugging errors in front-end applications like [Windm
 | jupyterhub-service           | /lw-workspace/                   | https://github.com/uc-cdis/cloud-automation/tree/master/kube/services/jupyterhub           |
 | jupyterhub-service           | /lw-workspace/hub/logout         | https://github.com/uc-cdis/cloud-automation/tree/master/kube/services/jupyterhub           |
 | manifestservice-service      | /manifests/                      | https://github.com/uc-cdis/manifestservice                                                 |
+| metadata-service     | /mds/                      | https://github.com/uc-cdis/metadata-service
 | peregrine-service            | /peregrine/_status               | https://github.com/uc-cdis/peregrine                                                       |
 | peregrine-service            | /peregrine/_version              | https://github.com/uc-cdis/peregrine                                                       |
 | peregrine-service            | /api/search                      | https://github.com/uc-cdis/peregrine                                                       |