Using Dynatrace to monitor Java Enterprise Edition, Quarkus and Node.js apps, Kafka, Istio Service Mesh etc
The workshop will cover the following topics:
-
Facilitate end-to-end tracing of transactions using Dynatrace OneAgent SDK.
The Dynatrace OneAgent SDK explicitly model remote calls, database requests, web requests, message passing, in-process context passing and more.
In the labs, we use the OneAgent SDK for Java. Deploying agents for Node.js, C/C++, Python and .NET is very similar.
-
Automate manual steps to speed up your Jenkins and Tekton CI/CD pipeline and identify quality issues earlier in the software lifecycle.
Dynatrace automates manual quality validation processe using quality gates. Events, tags and APIs enable seamless integration with CI/CD workflows delivering automated deployments for faster time-to-value.
-
Dynatrace and OpenShift Service Mesh
OpenShift 4.x (we included automation to deploy OpenShift 4 on baremetal at Install but all labs should work fine regardless of how and where you deploy OpenShift).
Also tested on RHPDS (RedHat Product and Demo System) « OpenShift 4.4 Workshop ».
Each module should be able to be completed in between 30-45 minutes plus 15 mins of slides introducing the content.
Youtube playlist (step-by-step for each lab): https://www.youtube.com/playlist?list=PLg1pvyPzFye38HA6eB3bzftH3t-YBZVOI
This workshop contains reusable components/automation to:
1. Deploy OpenShift 4.4+ on baremetal
2. Connect your OpenShift clusters to Dynatrace to take advantage of the dedicated OpenShift overview page
(using ActiveGate)
3. Deploying JBoss EAP applications with the Dynatrace OneAgent SDK to
facilitate end-to-end tracing of transactions.
4. OpenShift CI/CD with Jenkins and Sonarqube
5. Publishing artifacts to Sonatype Nexus using Jenkins Pipelines
6. End to end monitoring for proprietary Java frameworks
(using https://mvnrepository.com/artifact/com.dynatrace.oneagent.sdk.java/oneagent-sdk)
7. Automate Quality Gates - Jenkins / Dynatrace integration - Performance Signature
(collects performance values during a Jenkins build and evaluates and
compares the result with previous builds and non-functional requirements).
8. Automate Quality Gates - Tekton Pipelines with Dynatrace-CLI running in containers using podman
9. Deploy Ansible Tower on baremetal
See also lab 17: automate the onboarding of applications into Dynatrace using Ansible Tower
10. Optional: Deploy Ansible Tower 3.7.1 on OpenShift 4.4
11. Ansible/Z and mainframe monitoring (https://www.dynatrace.com/technologies/mainframe-monitoring/)
Automation to deploy ZD&T emulated mainframe, Ansible/Z and multiple OpenShift 4 releases at
https://github.com/marcredhat/z/blob/master/zdt.adoc
12. Dynatrace and OpenShift Service Mesh (https://github.com/marcredhat/dynatrace/blob/master/servicemesh.adoc)
14. Publish Spring Boot Actuator custom metrics on Dynatrace
15. Build and deploy Quarkus apps on OpenShift 4.4+
Package the Quarkus apps with Dynatrace OneAgent SDK; monitor Quarkus apps running in OpenShift
16. OpenShift 4.4+ / Dynatrace / NeoLoad 7.4
17. Automate the onboarding of applications into Dynatrace using Ansible Tower
18. Get deep insights into Kafka on OpenShift (Strimzi/AMQ Streams) using Dynatrace
19. Monitor SSL/TLS certificate expiry for external and internal services using Dynatrace
20. Scale your Dynatrace API operations; API mocking and testing with Microcks on OpenShift
21. Dynatrace real user monitoring on OpenShift 4.4+; Dynatrace Visually complete and Speed index
Drupal 9.0.3 with AMP module and PostgreSQL 12.3 backend on OpenShift 4.4
22. Keptn is an event-based control plane for continuous delivery and automated operations for cloud-native applications.
Use Keptn to deploy to OpenShift 4.5 with Service Mesh.
Monitor and automate operations using Dynatrace.
23. Deploy a 3-tier Node.js app on OpenShift 4.4+. Monitor with Dynatrace.
24. Deploying a React App with an Express Backend on OpenShift 4.4+. Monitor with Dynatrace.
For details on how to fully automate the installation of OpenShift 4.4 and 4.5 (for lab/test purposes) on a baremetal server with KVM/libvirt, please see https://github.com/marcredhat/z/blob/master/zdt.adoc, starting at the "Install OpenShift 4" paragraph
If you are using RHPDS (the RedHat Product and Demo System), please see https://www.youtube.com/watch?v=yC-OUR7Z9HA
Prerequisites 1) Dynatrace API token from Settings > Integration > Dynatrace API 2) Dynatrace PaaS token PaaS (used to download OneAgent and ActiveGate installers) from Settings > Integration > Platform as a Service 3) apiUrl - URL to the API of your Dynatrace environment. In Dynatrace SaaS it will look like https://<ENVIRONMENTID>.live.dynatrace.com/api Mine is https://eye15053.live.dynatrace.com/api
oc new-project dynatrace
oc -n dynatrace create secret generic oneagent \ --from-literal="apiToken=<API token>" --from-literal="paasToken=<PaaS token>" You may update this Secret at any time to rotate the tokens.
Check the latest release at https://github.com/Dynatrace/dynatrace-oneagent-operator/branches/active oc apply -f \ https://github.com/Dynatrace/dynatrace-oneagent-operator/releases/download/v0.8.0/openshift.yaml customresourcedefinition.apiextensions.k8s.io/oneagentapms.dynatrace.com configured customresourcedefinition.apiextensions.k8s.io/oneagents.dynatrace.com configured mutatingwebhookconfiguration.admissionregistration.k8s.io/dynatrace-oneagent-webhook created serviceaccount/dynatrace-oneagent created serviceaccount/dynatrace-oneagent-operator created serviceaccount/dynatrace-oneagent-webhook created role.rbac.authorization.k8s.io/dynatrace-oneagent-operator created role.rbac.authorization.k8s.io/dynatrace-oneagent-webhook created clusterrole.rbac.authorization.k8s.io/dynatrace-oneagent-operator created clusterrole.rbac.authorization.k8s.io/dynatrace-oneagent-webhook created rolebinding.rbac.authorization.k8s.io/dynatrace-oneagent-operator created rolebinding.rbac.authorization.k8s.io/dynatrace-oneagent-webhook created clusterrolebinding.rbac.authorization.k8s.io/dynatrace-oneagent-operator created clusterrolebinding.rbac.authorization.k8s.io/dynatrace-oneagent-webhook created service/dynatrace-oneagent-webhook created deployment.apps/dynatrace-oneagent-operator created deployment.apps/dynatrace-oneagent-webhook created securitycontextconstraints.security.openshift.io/dynatrace-oneagent-privileged created
curl -o cr.yaml https://raw.githubusercontent.com/Dynatrace/dynatrace-oneagent-operator/master/deploy/cr.yaml
Update cr.yaml with apiUrl and the name of secret we create above ("oneagent").
In my case,
apiUrl: https://eye15053.live.dynatrace.com/api
tokens: "oneagent"
as shown below.
apiVersion: dynatrace.com/v1alpha1
kind: OneAgent
metadata:
# a descriptive name for this object.
# all created child objects will be based on it.
name: oneagent
namespace: dynatrace
spec:
# dynatrace api url including `/api` path at the end
# either set ENVIRONMENTID to the proper tenant id or change the apiUrl as a whole, e.q. for Managed
apiUrl: https://eye15053.live.dynatrace.com/api
# disable certificate validation checks for installer download and API communication
skipCertCheck: false
# name of secret holding `apiToken` and `paasToken`
# if unset, name of custom resource is used
tokens: "oneagent"
.....
If you want Dynatrace to monitor OpenShift Service Mesh deployments, set enableIstio: true
oc apply -f cr.yaml oneagent.dynatrace.com/oneagent configured
oc get pods NAME READY STATUS RESTARTS AGE dynatrace-oneagent-operator-788fd7f5b4-6lt67 1/1 Running 0 4m21s dynatrace-oneagent-webhook-84747567df-lmltw 2/2 Running 0 4m21s oneagent-4j9xf 0/1 Running 0 102s oneagent-55p2k 0/1 Running 0 106s oneagent-b7qlb 0/1 Running 0 108s oneagent-jhk2f 0/1 Running 0 107s
oc logs oneagent-jhk2f 23:19:49 Started agent deployment as a container, PID 1352627. 23:19:49 Downloading agent to /tmp/Dynatrace-OneAgent-Linux.sh via https://eye15053.live.dynatrace.com/api/v1/deployment/installer/agent/unix/default/latest?Api-Token=***&arch=x86&flavor=default 23:20:18 Download complete 23:20:18 Downloaded version: 1.195.161.20200720-160625 23:20:18 Verifying agent installer signature 23:20:21 Verification successful 23:20:21 Deploying to: /mnt/host_root 23:20:21 Starting installer... 23:20:22 Warning: Parameter APP_LOG_CONTENT_ACCESS is deprecated and will be removed in future release. Please use --set-app-log-content-access instead. For details, see https://www.dynatrace.com/support/help/shortlink/oneagentctl 23:20:23 Checking root privileges... 23:20:23 OK 23:20:23 Installation started, version 1.195.161.20200720-160625, build date: 20.07.2020, PID 1352627. 23:20:25 Detected platform: LINUX arch: X86 23:20:25 Detected bitness: 64 23:20:25 Checking free space in /opt/dynatrace/oneagent 23:20:27 Extracting... 23:20:28 Unpacking. This may take a few minutes... 23:20:52 Unpacking complete. 23:20:52 Moving new binaries into lib folders... 23:20:54 User 'dtuser' added successfully. 23:20:57 Non-privileged mode is enabled. 23:20:57 Applying agent configuration 23:20:58 Storing SELinux policy sources in /opt/dynatrace/oneagent/agent. 23:20:58 Installing SELinux Dynatrace module. This may take a while... 23:21:35 dynatrace_oneagent module was successfully installed
If you are using NFS, please see https://github.com/marcredhat/upi/blob/master/nfs/nfs.adoc
2. Connect your OpenShift clusters to Dynatrace to take advantage of the dedicated OpenShift overview page
Connecting your OpenShift clusters to Dynatrace to take advantage of the dedicated OpenShift overview page requires that you run an ActiveGate in your environment (version 1.163+). See https://www.dynatrace.com/support/help/technology-support/cloud-platforms/openshift/monitoring/monitor-openshift-clusters-with-dynatrace/
oc project dynatrace oc apply -f https://www.dynatrace.com/support/help/codefiles/kubernetes/kubernetes-monitoring-service-account.yaml serviceaccount/dynatrace-monitoring created clusterrole.rbac.authorization.k8s.io/dynatrace-monitoring-cluster created clusterrolebinding.rbac.authorization.k8s.io/dynatrace-monitoring-cluster created
oc config view --minify -o jsonpath='{.clusters[0].cluster.server}'
https://api.ocp4.local:6443
Get the Bearer token for the OpenShift cluster using the following command:
Copy the secret returned by the following command:
oc get secret $(oc get sa dynatrace-monitoring -o jsonpath='{.secrets[1].name}' -n dynatrace) \
-o yaml | grep token
echo "<token>" | base64 --decode
The result of the command above is the Bearer token that you'll use to connect your OpenShift cluster to Dynatrace.
#oc get secret $(oc get sa dynatrace-monitoring -o jsonpath='{.secrets[1].name}' -n dynatrace) -o jsonpath='{.data.token}' -n dynatrace | base64 --decode
#oc get secret $(oc get sa dynatrace-monitoring -o jsonpath='{.secrets[1].name}' -n dynatrace) -o yaml | grep token
Connect your OpenShift cluster to Dynatrace You'll need the Bearer token and the Kubernetes API URL mentioned above to set up the connection to the Kubernetes API. Go to Settings > Cloud and virtualization > Kubernetes. Click Connect new cluster. Provide a Name, Kubernetes API URL, and the Bearer token for the OpenShift cluster.
If your OpenShift cluster does not already have a Dynatrace ActiveGate, you'll be required to install one. Click on "Install a new Environment ActiveGate"
In this lab, you will deploy Jenkins and Sonarqube.
These will be used in the following labs to deploy CI/CD pipelines with Dynatrace.
git clone https://github.com/marcredhat/devsecops-workshop cd devsecops-workshop/
./scripts/provision.sh deploy --ephemeral --user <user> #./scripts/provision.sh deploy --ephemeral --user marc
oc delete job cicd-demo-installer oc delete pods -l job-name=cicd-demo-installer
oc get pods NAME READY STATUS RESTARTS AGE jenkins-2-4xs59 1/1 Running 0 27m jenkins-2-deploy 0/1 Completed 0 27m sonarqube-1-74tdp 1/1 Running 0 27m sonarqube-1-deploy 0/1 Completed 0 27m
|
Note
|
(ignore if you are not using RHPDS) If you see OOMKilled pods, it is probably necessary to edit the default RHPDS limit range (this will be in effect for new projects, not existing projects): oc edit template project-request -n openshift-config To redeploy in a new project, just set another "user" value and re-run ./scripts/provision.sh deploy --ephemeral --user <newuservalue> Here is the config I used successfully: https://github.com/marcredhat/dynatrace/blob/master/rhpds_project_request_template.yaml |
oc expose svc/sonarqube route.route.openshift.io/sonarqube exposed
oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD jenkins jenkins-cicd-marc.apps.cluster-dynatrac-03ed.dynatrac-03ed.example.opentlc.com jenkins <all> edge/Redirect None sonarqube sonarqube-cicd-marc.apps.cluster-dynatrac-03ed.dynatrac-03ed.example.opentlc.com sonarqube 9000-tcp None
Connect to Sonarqube as admin/admin
In this lab, you will deploy Sonatype Nexus and run a Jenkins Pipeline to build a Java .war file that includes the OneAgent SDK and save this .war file to Sonatype Nexus
See https://github.com/OpenShiftDemos/nexus contains OpenShift templates and scripts for deploying Sonatype Nexus 2 an 3 and pre-configuring Red Hat and JBoss maven repositories on Nexus via post deploy hooks. You can modify the post hook in the templates and add other Nexus repositories by using helper functions.
oc project cicd-<user> In my case, oc project cicd-marc wget https://raw.githubusercontent.com/OpenShiftDemos/nexus/master/nexus3-template.yaml oc new-app -f nexus3-template.yaml --param=NEXUS_VERSION=3.13.0 --param=MAX_MEMORY=2Gi
oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD jenkins jenkins-cicd-marc.apps.ocp4.local jenkins <all> edge/Redirect None nexus nexus-cicd-marc.apps.ocp4.local nexus 8081-tcp None
https://jenkins-cicd-marc.apps.ocp4.local/ http://nexus-cicd-marc.apps.ocp4.local/ Jenkins: use your OpenShift credentials Nexus: admin/admin123
We can now configure Jenkins and Nexus.
Screenshots are provided below; a video showing step-by-step Jenkins and Nexus configuration can be found at https://www.youtube.com/watch?v=R9VTzEQYt7U
My Jenkins Pipeline is at https://github.com/marcredhat/dynatrace/blob/master/jenkins_pipeline_nexus
Here are the artifacts uploaded to Nexus by our Jenkins Pipeline:
Video showing the pipeline building the .war file with Dynatrace OneAgent SDK and uploading it to Nexus at https://www.youtube.com/watch?v=muoz7uoj0S8
Features of OneAgent SDK for Java (https://github.com/Dynatrace/OneAgent-SDK-for-Java) Trace database requests Trace messaging Outgoing web requests Incoming web requests Custom request attributes In process linking Trace incoming and outgoing remote calls
Add the following to https://github.com/marcredhat/cargotracker/blob/master/pom.xml
and
Restart the Jenkins Nexus pipeline above
<!-- Marc added https://mvnrepository.com/artifact/com.dynatrace.oneagent.sdk.java/oneagent-sdk -->
<dependency>
<groupId>com.dynatrace.oneagent.sdk.java</groupId>
<artifactId>oneagent-sdk</artifactId>
<version>1.7.0</version>
<scope>compile</scope>
</dependency>
The Performance Signature plugin collects performance values during a Jenkins build and evaluates and compares the result with previous builds and non-functional requirements. Several software tests can be run and evaluated automatically, so that the most important key performance indicators (KPI) can be summarized and be available for all project participants very quickly.
Install Jenkins Dynatrace plugins
Navigate to Manage Jenkins -> Configure System -> Performance Signature: Dynatrace SaaS/Managed Enter name of the Server (ex. Dynatrace Server) Your Tenant URL (https://xxxxxxxx.live.dynatrace.com) – SaaS, (https://asdfa.Dynatrace-managed.com/e/asdfafa) - Managed API Token Note: You many need to Save and exit this screen and come back for the “Add” dropdown to works, thanks to a bug in Jenkins
Video showing Dynatrace-CLI with podman: https://youtu.be/_fnv78Q9sDU
Tekton tasks are generic, take input and output resources, execute steps as container images and can be reused for several pipelines. As we can run the Dynatrace CLI tasks in containers (using podman, see https://github.com/marcredhat/dynatrace/blob/master/build.sh and https://github.com/marcredhat/dynatrace/blob/master/testdynatraceclicontainer.sh) *** we can easily integrate Dynatrace into Tekton Pipelines to get *** *** performance metrics (Dynatrace "Performance Signature"), automate quality gates etc *** Parallel tasks can be run in separate containers, which is very efficient.
|
Note
|
See OpenShift Pipelines examples at https://sysdig.com/blog/securing-tekton-pipelines-openshift/ and a short video at https://bit.ly/marcredhatsysdigcicd |
https://raw.githubusercontent.com/marcredhat/tower/master/towerbaremetal.sh
wget https://raw.githubusercontent.com/marcredhat/tower/master/tower.sh chmod +x ./tower.sh ./tower.sh
Automation to deploy ZD&T emulated mainframe, Ansible/Z and multiple OpenShift 4 releases at https://github.com/marcredhat/z/blob/master/zdt.adoc
Drupal 9.0.3 with AMP module and PostgreSQL 12.3 backend on OpenShift 4.4
22. Keptn is an event-based control plane for continuous delivery and automated operations for cloud-native applications.
Use Keptn to deploy to OpenShift 4.5 with Service Mesh. Monitor and automate operations using Dynatrace.
All credit for the React app goes to Jan Kleinert.
Links: Sysdig OPA Image Scanner on OpenShift 4.5.2 - short video at https://bit.ly/marcredhatopa Step-by-step instructions at https://github.com/marcredhat/sysdig/blob/master/ImageScanningAdmissionController-OpenPolicyAgent.adoc http://redhatgov.io/workshops/secure_software_factory/ https://github.com/RedHatGov/devsecops-workshop https://eye15053.live.dynatrace.com/ https://github.com/redhat-cop/agnosticd/blob/development/training/01_Introduction/05_AgnosticD_in_Prod.adoc https://github.com/redhat-cop/agnosticd/tree/development/training https://github.com/dynatrace-acm/dtacmworkshop https://github.com/redhat-cop/agnosticd/blob/development/ansible/software_playbooks/tower.yml https://github.com/Dynatrace/dynatrace-oneagent-operator https://www.dynatrace.com/support/help/reference/dynatrace-concepts/access-tokens/