Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

jlester-msft · 2024-02-27T23:37:04Z

No description provided.

…tstrapper

…pload the log to the storage account task dir

… exit code capture

… confusing bash. remove extra spaces to avoid $'\r': command not found errors

…art task

Upload debug information using single azcopy on error only add notes for making setup of vm_monitor work in the start task

MattMcL4475 · 2024-03-25T20:06:28Z

src/vm_monitor/vm_monitor_scripts/collect_azure_vm_perf.sh

@@ -0,0 +1,13 @@
+#!/bin/bash


JL to delete

MattMcL4475 · 2024-03-25T20:16:09Z

TODO:

1. Add configuration setting; something like AdvancedVmPerformanceMonitoringEnabled; default false to the "BatchNodes" area
2. Move the download to the start task
3. Merge origin/main into this branch

…a4gh-tes into jlester/vm_node_perf

…release.

Co-authored-by: Blair L Murri <[email protected]>

…active batch pool

Checking in azure_env compatible docker_deploy.sh changes

…ver input.filecount for watching filesystem changes in task directory

… exclude matches (!mtimeFilter)

BMurri · 2024-04-15T19:12:56Z

I'd prefer if we could build the binary download packages in the build rather than check those binaries into git

Update dockerfile caching Update telegraf configs with better timestamp behavior and collection cleanup

jlester-msft · 2024-04-15T21:56:23Z

I'd prefer if we could build the binary download packages in the build rather than check those binaries into git

Agreed this would need some integration into the build pipeline to generate the binary package.

Right now src/vm_monitor/build_tes_deployment_archive.sh contains an example script that will do the building of the telegraf binary + build a .tar.gz of everything. The basic steps are:

Build a custom stripped down version of telegraf
1. The local build process should probably be skipped in favor of the docker style build in src/vm_monitor/build_telegraf_remote.sh. This is setup to build on a remote VM but doesn't need to. It is basically dockerfile build + run the docker to copy out the telegraf binary
Make a flattened .tar.gz file with the binary + the scripts needed
Put that binary somewhere consumable

jlester-msft and others added 28 commits February 27, 2024 15:30

Update of start-up script to download tes_vm_perf.tgz and execute boo…

a2e5c16

…tstrapper

Side-step build tests

1a3da64

Add dummy function for the correct call

c663328

add catch for GetInternalTesBlobUrlAsync

1712d1c

chmod typo fixed

3618aea

Add script to be uploaded every time

70eff6a

remove try/catch

2d20d3d

extract explictly to AZ_BATCH_TASK_DIR

bd0c9eb

disable extraction and running integration test

fa98c05

Add absolute paths for tar

ffdae36

Wrap batch_script execution in a function, call it with screen, and u…

baaf634

…pload the log to the storage account task dir

Updating bash script

3964c62

Update scripts and use python to parse nodetaskurl

901e207

Update python script, msi auth for azcopy, add task complete, and add…

d1a294c

… exit code capture

missing semicolon

e5c9cb2

remove windows line endings

bc0e7db

batchNodeScript is ending up with Windows Style line endings which is…

c4733d3

… confusing bash. remove extra spaces to avoid $'\r': command not found errors

export env and load it inside screen. Also run outside screen once

432c8d4

fix missing quote

5316a4b

convert script to @Strings

6750b7d

Add shebang disable direct script call

e3e0689

Cleanup of bash_script construction, build pipeline test

459b90f

remove unused functions

0331bdd

Add additional debugging capture of host system on failure, change st…

9049421

…art task

Add debugging tools

4c224a7

Always collect debug info

510268c

Update naming of perf archive to tes_vm_monitor.tar.gz

72a81fd

Upload debug information using single azcopy on error only add notes for making setup of vm_monitor work in the start task

Merge branch 'main' into jlester/vm_node_perf

38a986f

MattMcL4475 reviewed Mar 25, 2024

View reviewed changes

src/vm_monitor/vm_monitor_scripts/collect_azure_vm_perf.sh

@@ -0,0 +1,13 @@

#!/bin/bash

Copy link

Collaborator

MattMcL4475 Mar 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JL to delete

remove old tes_vm_perf.tar.gz

0dec8bc

jlester-msft and others added 6 commits March 25, 2024 13:41

update path in csproj

c68386f

Merge branch 'jlester/vm_node_perf' of https://github.com/microsoft/g…

270dd63

…a4gh-tes into jlester/vm_node_perf

Update telegraf dockerfile to pull latest golang and latest telegraf …

a5f840d

…release.

Update bash scripting, remove jq requirement, add nvme mounting (#648)

656162d

Co-authored-by: Blair L Murri <[email protected]>

Merge branch 'main' into jlester/vm_node_perf

c6c2394

formatting

1e33b7c

BMurri changed the title ~~Update of start-up script to download tes_vm_perf.tgz and execute boo…~~ Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper Mar 25, 2024

BMurri added 5 commits March 25, 2024 17:13

remove unused binary

28e20cc

surface vm_monitor in VisualStudio IDE

80b6ca9

Add configuration (disabled by default) to gate this feature

a0acec0

move monitor download to start task

d73742e

make download conditional

b4afa2b

BMurri added the Needs Issue PR needs at least one associated issue label Mar 26, 2024

jlester-msft added 5 commits March 27, 2024 13:57

cleanup python code in pool creation + add pulling of config from an …

b43abc6

…active batch pool

Updating help comment

01d5373

Correct tags in vm_monitor collection

eed6263

Checking in azure_env compatible docker_deploy.sh changes

Add mtime filtering to filestat input plugin. Prefer input.filestat o…

54a1110

…ver input.filecount for watching filesystem changes in task directory

Update dockerfile to patch in builder. Also update filtering logic to…

3a56cf9

… exclude matches (!mtimeFilter)

Add url validation to azure_append_blob

59f3a31

Update dockerfile caching Update telegraf configs with better timestamp behavior and collection cleanup

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

jlester-msft commented Feb 27, 2024 •

edited by BMurri

Loading

MattMcL4475 Mar 25, 2024

MattMcL4475 commented Mar 25, 2024 •

edited by BMurri

Loading

BMurri commented Apr 15, 2024

jlester-msft commented Apr 15, 2024

Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

Are you sure you want to change the base?

Update of start-up script to download tes_vm_perf.tgz and execute bootstrapper #622

Conversation

jlester-msft commented Feb 27, 2024 • edited by BMurri Loading

MattMcL4475 Mar 25, 2024

Choose a reason for hiding this comment

MattMcL4475 commented Mar 25, 2024 • edited by BMurri Loading

BMurri commented Apr 15, 2024

jlester-msft commented Apr 15, 2024

jlester-msft commented Feb 27, 2024 •

edited by BMurri

Loading

MattMcL4475 commented Mar 25, 2024 •

edited by BMurri

Loading