From 259e56f192f9d38cd7af45e3923ba5bdae574d38 Mon Sep 17 00:00:00 2001 From: Peter Hessey Date: Thu, 9 Mar 2023 13:21:07 +0000 Subject: [PATCH 1/6] =?UTF-8?q?=F0=9F=93=9D=20Add=20azureml=20doc=20templa?= =?UTF-8?q?te?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Peter Hessey --- auto3dseg/docs/running_in_azureml.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 auto3dseg/docs/running_in_azureml.md diff --git a/auto3dseg/docs/running_in_azureml.md b/auto3dseg/docs/running_in_azureml.md new file mode 100644 index 0000000000..e6f515fd5f --- /dev/null +++ b/auto3dseg/docs/running_in_azureml.md @@ -0,0 +1,22 @@ +# Training Auto3dSeg in AzureML + +Introductory para: + +- explain feature +- advantages of training in AzureML + +## AzureML Pre-requisites + +- workspace + config.json +- datasets set up in blob storage +- datastores set up in AzureML workspace + +## Running + +- task.yaml needed inputs +- same as running normally but with --azureml flag +- explain outputs + +### additional options + +all other configuration options + some examples? From 35e35b3344d69c94dfa2c69c70a6491da77d233c Mon Sep 17 00:00:00 2001 From: Peter Hessey Date: Mon, 13 Mar 2023 11:22:08 +0000 Subject: [PATCH 2/6] Add running in azureml doc content Signed-off-by: Peter Hessey --- auto3dseg/docs/running_in_azureml.md | 105 ++++++++++++++++++++++--- auto3dseg/figures/azureml_job_tabs.png | Bin 0 -> 24928 bytes 2 files changed, 94 insertions(+), 11 deletions(-) create mode 100755 auto3dseg/figures/azureml_job_tabs.png diff --git a/auto3dseg/docs/running_in_azureml.md b/auto3dseg/docs/running_in_azureml.md index e6f515fd5f..87c9c818ed 100644 --- a/auto3dseg/docs/running_in_azureml.md +++ b/auto3dseg/docs/running_in_azureml.md @@ -1,22 +1,105 @@ # Training Auto3dSeg in AzureML -Introductory para: +It is possible to train Auto3dSeg using your AzureML resources. This page documents the minimal set up required to utilise AzureML hardware to run your Auto3dSeg training tasks. -- explain feature -- advantages of training in AzureML +## AzureML Setup -## AzureML Pre-requisites +Before being able to train in the cloud you will need to have your AzureML resources set up properly. If you have not already completed this, please follow [this tutorial](https://hi-ml.readthedocs.io/en/latest/azure_setup.html) from Microsoft's Medical Imaging research team. -- workspace + config.json -- datasets set up in blob storage -- datastores set up in AzureML workspace +### GPU requirements + +The GPU requirements needed to successfully run Auto3dSeg depend on the size of your data. For running the data used by the [Hello World Example](https://github.com/Project-MONAI/tutorials/blob/main/auto3dseg/notebooks/auto3dseg_hello_world.ipynb) then a single GPU with 8GB RAM is sufficient. The GPU nodes you have available in AzureML will depend upon your region and Azure subscription. For larger datasets it is highly recommended to use nodes with multiple GPUs and more RAM (>= 16GB). + +## Configuration + +### Pre-requisites + +As well as the requirements outlined in the [tutorials README](https://github.com/Project-MONAI/tutorials), you will need to install [hi-ml-azure](https://pypi.org/project/hi-ml-azure/) in your environment by running the following command: + +- `pip install "hi-ml-azure>=0.2.19"` + +### Configure environment - `environment(-azure).yaml` + +- To be discussed with Dong et. al. + +### Configure AzureML connection - `config.json` + +To connect Auto3dSeg to your AzureML workspace you will need to download the configuration file associated with that workspace by following the instructions [here](https://hi-ml.readthedocs.io/en/latest/azure_setup.html#accessing-the-workspace). Place this file in your new directory: + +```shell +. +└── your_new_dir/ + └── config.json +``` + +### Configure data - `datalist.json` + +The same as when running Auto3dSeg locally, you will need to define a `datalist.json` file for your data and place it in your directory. Check out the [tasks folder](https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg/tasks) for example datalists. Make sure that all paths are relative to the head of your Azure storage blob that contains your dataset. + +Your directory should now look like so: + +```shell +. +└── your_new_dir/ + ├── config.json + └── datalist.json +``` + +### Configure Auto3dSeg - `task.yaml` + +All arguments passed to the Auto3dSeg autorunner and the AzureML submission will need to be defined in a configuration file. Create a new `task.yaml` file from this template: + +```yaml +name: +task: segmentation + +modality: +datalist: .json +dataroot: /path/to/local/data # Only necessary if you also want to run locally + +multigpu: # set to true if you want to use multiple GPUs for training + +azureml_config: + compute_cluster_name: "" + input_datasets: + - "" # currently only 1 input dataset is supported + default_datastore: "" +``` + +Your directory should now look like this: + +```dir +. +└── your_new_dir/ + ├── config.json + ├── datalist.json + └── task.yaml +``` ## Running -- task.yaml needed inputs -- same as running normally but with --azureml flag -- explain outputs +Once the above configuration is completed, to run the Auto3dSeg AutoRunner in AzureML simply use the following command (in `your_new_dir/`): + +```python +python -m monai.apps.auto3dseg AutoRunner run --input='./task.yaml` --azureml +``` + +Make sure to follow any instructions in the terminal regarding authenticating on AzureML. After the job is successfully uploaded, you should see the following output: + +```shell +URL to job: https://ml.azure.com/runs/?wsid=/subscriptions//resourcegroups//workspaces/&tid= +``` + +Following this link will take you to your AzureML job where the AutoRunner is running. Below the job name you will see the following tabs, the most important of which are explained below. + +![azureml_job_tabs](../figures/azureml_job_tabs.png) + +### Outputs + +### Metrics + +### Hardware Monitoring -### additional options +## Additional Configuration Options all other configuration options + some examples? diff --git a/auto3dseg/figures/azureml_job_tabs.png b/auto3dseg/figures/azureml_job_tabs.png new file mode 100755 index 0000000000000000000000000000000000000000..3cc2c0a73341372df3c2c9de098fdf91034f3c71 GIT binary patch literal 24928 zcmdqIXH-*L_daSlf(nQV(u*EcI*9Zd0hQj7-Xv7%(g`FAB0W*MRFx)3F@W?UNC-W2 z2mvC5PC_S?knqnr@9(|$ez_m+r#r@PjFh$a&fcrcxt{f`XU@bK>T57u=eU01!UYB` z&8Nl}F3>1Y%T-rtsGm=ZXtSwb7yXSj)GkyGajjDi{&H2-Q@wBj4x>AJd4+m>%~#XX z|H1{92mib;Y8l_(xp2YZQ0u9xX^0(Rjv>d!Ibv-WqS;BK-mLk`v7q{)_T9Rte{PG- zR$mt_HG+SA8lRGun1XW!5MtjKh$jNYY^_>UG^&GLHFnR{ zZwOO@pX87uMH8T62xYX}3ox<96D=4dPg* z51^;5$>=6mU>q?f*1)kT@dS4C@4iou0p92HlbtFvEY{oB9R9^lFdYuCb|jY~c$fta z(o@(pTuHG09SwIK%Q|%ze>G; z3-^S@vw-+dWdy7pepX{%Jk!oEcjD_73%@=0EWBT<9|Q?_UjPk%uEU7o?qz{4Ca0 z*5Ro840Nqm_q4I`d} zR2nu=N7_$TE=PD@&Wz)0sq3yWR!`(*V$pFed z_mFQ7dyE8dobzpB#qBPRT#r*#p+>imQ1&~Lr%l`n3U())w@}J8{F8faP-XbC`7InH zFSn1t(dj5=t|X?IcQpH>Gv;&6Ej ztyxqzw`1`~<*@6CEK3axdCG0_=`2?7yCQVc4PW)}aQ)>ao5RYlpLdh#p>Av`Z=EgX=FsTM zo8A*){TrDVvh=Bw+^vp_ShmY=Cll8)t&a1)$Z0GtLw3I7?apm{M_nD48v(GXwnc*) zjV6Y*nv#LIXZ1n)Pp63iO@AU5>byKWn?TP702P9uFaH{Nl(IbfBDs*l>8%PKU<@We z_pO5wD@2i4I4IZ?{K`ChQ=_Qf9yv4H%jAK9*se%7JCY&A&V|G-$@4~G7D`BPx!&$a zRVt2vp6HkjhNIEDr$E-OV|7?;f&tmM2OvAY@>%Am#g{D7{Fd=*{^ z7u|u7ku^39oS$-v&a2m`&R`|XADxKCBtpmdlLVEH!q0p?G;CUc24{~zCk~(yKeD}5 z{O4|xKG~RP{uG0P+yr1w3h-6lKf%ayPMI#z3@Hs0bx`emM1;t3)HR8dZ@qK%TCQE4 z*0Rqxp|)tn@lwHJhCTMr|AG3Jgd=W7A*=@@vELmLB}Dbe1AETc?Z|eNf-^)&I0TKx^!Q2hd|n zGPt%f>BOX(zj^d(eZ-g=I>JJeQ>U%%UY)lNjMdw^zKbkNOczraY;dzI7e%)hmSU>1 zn9e$#8JATg6(2@j&0h#S@qnCxe69wY*swrJrIf&ahS65DhUx_;$M65YP7GBKZ|an3t!3iJ50a zE$nr%O_MUE(_+`*DA>D^&^a_^_TBRn5^j5om6+24sC^Imz=v*@H6xpA*uW1Pr_qmO zT)36R($BlUZB$lByfU@f!{8RxHIkCx?O|eOfRS5tkj7xm^zYgnB?sce9qe4&B!ImT z1!k`@#Uo1@%kLht2bi2GRuylDAPKwZVBLiGZm@J=RICZAUBN5@0IKX?u?StqXOg$l z(AfU~7HMkdYQVu*A|p0Wv`B*Qq0CCLF4R5t{k5^LCuc?Qtr1K!ZGQ?*JmIJH7R;Gf zN!Uf#8y_nAVG)$=xH9kwefc-4TzCpWw$J3Pfo?Stl2Tl|ay+IG?>SwZVazu5LT$~; z&Gy4`74k#|`+!$q$bme{`8x9`uSjPo zzQ66qJ(&n{waVpdq|8dES3l zN9}!QQJ!N*rBl26!>EvfvpB+`7LI+sy@wd|C*&t%N9Inh7CP- z@e+}5moH8e6z1R$PM19LX$7R4F+Jgs1WZJ+exn<;6LVxsT}qrrta4(?R6ZFppN1+nO~ej zBczelj05p;(IBtys|d7gs^US0f`Z^K~jtUL*uqPVU9<|;JlPrXjqTj)_&amM5nII<^>mw3`uw3(1 zFzJxs4XZnt@tUKnl-13tORx#ouD@xo}=*WWA7-G^I@O2@VcZO!B)1n7nb<{U!tV8%g-qpn$B zR0=j}*?w?~m58pcijWagQVD*pqdIkpZ#*QNP>Lm45i^Rb=710>XK*z>bJed3$9)fZm{72dtmW8+Z6;?DS@HYYOM(paW>e2ET zam^nInfV$rW|Vsr)1@PpShV^$s%f9i2WwQl8(n}6U$1`b(Y6iBQK&1D%0gKuA!?op z(de8OQGL5nh9wt04#Wl)ee4vi+C8Sj)hO>dS6D4(! zo+Na{?VL-=b5m*BIUh=bZ!8|>!d_GeinWBAm=A|c5z9rwzJBmQx?Wt35lXul9ad{ z*Dxz#-8f}Fei&l=!abAmGXO)DC((R1X?{x(>#1k8@;@osK=lvUv2eEilv?2eNN@!E zY=@JX0HUc3(OB?4373g?{ueX36W?om#x>C`j*sb!&YE?v%x2vsSev=P{Ij{~%v=q0 zEt!Q?MwLnJxZY;R1wj*B*TD=0_V-J>4`X$@({7y4o@}Gi zQ=BDI&wv^p<)J>S5_u|KuXRIK%iIQ$*E#5r4NLfUma^y0TXH{<-JC-Sz0obhpPm~e zmR2<@ST#ikCXIx5Ap3Hzi~#h^Bjj0@GlDR1iS!z>U=p)8e_L><=fD-Z-i+l)y$P%R zKe%Z7Pwf7>IpLu}fH5VImEXg#rAxQ<#N}o)(V%0yuU?~%x7uH zeJq|1NET+|C`MpZnhG;6j&)~BwElR;9eoB`M*A(kyC(6)H~VyD8NJ_{eg28@@M>}` zps8&2=k4jdm^#)v;8iA9Zd&;!{2Zt3Ib(?nK8Y95%-{F^+tF5>1}4r^iTC^rKP>j* zMTJp;f&t*Adl~b=1=o-q$Y@EzQAeP>TsF|KCDcknIoQ)(aQxj*_a-Mxkud*zq95kz z4kyM@ME(%WGH^>sV|6+VTTm{sn=l^O{ws?$S4K<*8(^oa z{AFLV0Wb&!qee=W#)ZP6g;+b%X3|$l<o5ns6{_5Gb9vRXvog4K$_hGlkyyb080pKKq z4$f<=8Fuxu+^Vhd=D=Oa zXN4`_FvG|}^NHykv%g&$0p^w6eEFLO_ZUur+aFv{qEChL%RIj>FJ%ul8(5>LBK}yr zTgCEEbuCT}CP+87O!&@P?!xDZ>zQ>8n{)i#S7dAP)>{X!j9S*;%?hj#58=lLSCp&L zei2)CTj3jV`0Wmv+mT$K>RTKF7O-=>zis{ko_&kWSWd5QBB9mPl$`>&6vsEm(?Yyg z+a@0C82Ys)-M^xE^|kMNjm|*%^zv;*lME8{bym&tNRTpVZ1H?TIQ1H90L^t-td+6- z7ekg@^4sS4z{#$a#^_NGd#_CvWaq49gqLi%kk!?-nwF)4?~4Y2EMaV(4bOL(0G2Ji zv3~n~7t-Yf9Deilvb`#X!4b0%!yk5|}!lLd${J5wd-_qi6pg>5MwxPUX~O{$yyY>O*g?ECX6{dnp9 zBo#cn%#k_%HKb?C85J8LOWE*84D)t-S()k0uY5L@>4fZL3q0-$jXgSCint^cD#B3v zHRK~kpWC23KGes~4tK#?PbL%7JXObI-6C+$-4qWAZREA&RlQNx$}TQq8ljj6`-y&SiiC19$@1(+ zT2MwM{DRDcDaYtlx4ZJf%KdD`p4p_Xa6>@vG|T4QX)mHeMe_>K{W7u71oCs@1) z2!D0$OT%dsr=kA(ip9sLgzU*w6ls$Oij0xDB>pd^{iuUz_|hOOCDwcJV$f`C;6>CZ z#+g4*uNUgZHr2>NN}+R{xZF7M=6&2q_(*lbIdw<8+-B`KqWC_v2MDsBD&Nn`i?coP zphiqn|=;(>*$j1~R+7I3N zv-^v<9Kk)}6=$08Q{1{+vqZD?;wRhYYEY5(=!8kOFy6I1t|Z3yNh!HBra7Hq^X1`K zbb*a33@0VwGH0P;fN#K+_b@CV8@6N*v3RW!<#8)b4;ZnHd919W5xoGk;~ES5Ywnf* zVK=e{bme3-zy|!9%S1`BM41QTt5=TA8ubEl1-{ZI)hvGNMbFJ;7c9r)!Fujr{t)YW zP{EMh)V_t*I;8QVqUS%CexJLE-&$m;-x+Q=O^aa}-N{qilarda4S6jATm~I=Gt#kp<3APv zFKE@e5DcJ$TZDv7@1-d2(0Y~E?uZ0f#s)!gFC8!;FZw6;n)GDCs(u1%icF9SQzMtv z-{#O2@jd$kU8UUgGLZscu+O2Pe6+$sG2Xk>Zey^e_*$;~kM&tS=l;nD=+TIjhZ%r^ zLeq-;-~pdN3VDD?uGKC67e@k%Z(QcSTz{4$>A0{lna8TaJ9d)^1`rl&UZ6~${)un+ zvHH_)6PrR;;H(qW{(1e~QNSR08}2zli~;IJ-&M7bG9a-@ipk1ZJMPuaB9MGDTh?~F zx*sopX^jBb+JL*1{nyrF66Ipc`pX#U{kp*4ZCufO~4=9@k%=T!zehFeOn{O;J< zD2VSUKc|$?Ircx;UxaV}L}nHY#RIYZY7T+SR7;Sq6aYQ_&}Bw`Cpz7TX8XqvyX(#s zyuIh{(47;eb_doZ{6?ufUQZ)29&-BZOR`)*fH!;1m!ldD5Tn{n{M~OKk8&&OyXwKm zpYT;}hlkv@R{3HygB&H&M`1zpDjXoOTk^N1T%4bgOwi8|!gG5a6ua@$P^N_b*ubaF znAh2;2oD+0PU|t4f~_Z>e83*q<5*Ra~btZa|hK0Gl*^D;+h>Nm>RD@hHP(ni1+4u zJj#(qG(@=EW#q>IfsIOrD?wB5@uE>KHi`ZD^G46dGS6v{hGk{ZsHayJ&X%C z0|w$&+%03hHh&u_#W)7)994^{m<3>h!P%1OGsp2^P?5ySZ=v+Vv+$q0{VVyMi5GAx-$aj5We#^+uc%dQpw6k;f z6lDwFK{f*kd9l1(Gu*57Tk(U@wU5_}aErg08r+cgB*;5t^A~mXNpbn-*4tO;RT&~V z|19F_Cw+JQRI<0f^)a9WeIMp=`FjZF`}C#9azUPQnS_7NS&{pi>8IdmuEni%9M&s| zMjLZSJiTcZ9KdoHr4(QnbZcj?9-Uzp-2JQ~f67XUTy*at&x>_AE{TE7f4AVt@6EVr z6!M_9jq76!z!_W$D=Hy0Vos$x9J~CcVePoc6eNu|xb` zP~`FEyF`|_J!?Jh4+mr=k4~l(PvGk*c5zY`r(WY`4I>@l#OVr+@PR z?3K3iN#lh|N}2si*Kvl-)3a)6N{I*e*AluT^tRsasL95I9wXm1>I(P!`Za!_ z1+anMW#B~CMQQyv63`| zeqo!zSGrQuY|ES9l!3Dm$QLKS7RjAEal=k_xeN_+0{Dhn%G5rokd*ywb{A&=vCZ$# z=!?qRFd8NzRJyON2ZL0-j^*v=C5X6Y6{EykJpQ~Qw=`EfcV)SN%RT4D148`=dTfyz z|Lj={qXvK62#}IVx4z_}S6jRC{9egG;)55AZKjh!$VE94tPKq`J1Q&~>f1__XoK%( z@dhrR9`2)Sd=rRrK#naK{0&grAY?a2+wI$FJA!(Z>c486?b8s$Bi{ z9)E|v4$UF2r*CaulhRGJ%)s*W(=3EcdOLg0r_Ujb26w+)SIw}L&0~^`!MrgBUn_A- zOhH*m2}Jq52z7|4U3sW(;F)KC!Zu`gYn3Zs@%&z{pTva|=hDS*?h>mx?r0yMq`}6+ zZ%tLJ9n!DFHd`zn*VT8+g8s6$a*TlS9%N-g)QfbAQSowATEUFjel6rpo%7tF#7VmO46h1%fNHA)o=}O5gg{2a~2vZeg#V zeIcki2hPqmkh|qcsa29*&}6Kw9M? z=9OIkiWuv6rCLyHnPIO*ge1SUoozC%(vY(j%TYeDcUaS!EjAJ}L4=v<00Zxxg@Y6L zH@pX3en59&4=RulQkU`bBw^~Z0CyG}B@EvzDCtP~KI*d;n-nZM8+?+-&)S_8^ZISk zwL$nAXFcAZA%&fF?lvU5y&(j-a@>9Ag~);*4lS|$y<*NaNlG`NF<<#n|9esJo41Yc zBy}k=-s>ODvK)tAY&s$ufA2H^LLWX=Id$H|(CJ}lEzEzgofe`d}>{S*H!{Z1?m5Hjy z&q!9SN^RNaqS4!Seg?g}RB^vynKR*QsIS55T9b-Ki<1i%=#(1LN#Mja=GS!_@L1e| z*#_GwRsT?l zx6OY3sW~2)seoPaIS(mE8}~Rd=>1>3lTf;afoe>4C-1v<*Iu(hw3=Y?*B15GxI z)hty+IjG`Ix`BA?GFn*Jnukth?lvZvwJ*xwysNmvS^;9PTNBhpo*Ig)b+oI;52Q;_ z#Y2+J9sA~@@5;uwRpmxG7Ho_Bj36u|;w34e$`vyI7~paitmD@!{qN#`!|b=nu%CJD za)~G`pFKl4mByqlfN=nPh{E|Zu7n*VjX4heJVHXftbrynhCdcYD&T~55)l1ta3qyi zBvVCLY{0I0`3wH%Y+JpwjLjIAt?+FERe8XU`yWe+a*veWa{-5#Og%xRSD{cmyen-@ z55dDfVI_##b@UuD2A4hMe550C{-@-%j_w`sNotMbdW2#Vm{jLQ- z0Q#+?@7*bHUutPkfGAxn4)?M=Vc%3=?=$F@eLcnS(r~+6&1#*Jo9&1|NEW~gq^XPd~ljbXvH^lg!nSuETQFu%fM-$-dt{ zoC|31I_*5mL+-2-hu z-5a81=~|p7K$_a4(6GFvRfdVf2)~k^;!f@5kNqQ^Lre`M#rS@6XFKmG%?BU%a+a%2D#K zlnN)i_R=G01M7T5l=9}Vv^u)2-hSn-KOUBB)~`sDH^+9*&sZ#7gPEH!cp_K)qO1c zD||;$A+W2GXDWj2TW6c%a-_uxIZxfomQ=Y38lh^nU?rD)>=D<-D^Z95Z|bjqr2nE- z_f!#3Ol}2yw;K!g7 z@5=1rZr~x?w7}t6Uc(COo%FJe{`pqF-;yB5Z{m{n8;$nRcn$C4 z!LoFC*}J1X_VCGHwa@w)=oaxN#obH2OpcxJ@G|bNb@PH=7_;ve5 zAL(v@$>X@8Sh`9}6Kcl8H;>}ICX)gwr#H(!c~)nMCgQz3J;F=oc~ntWyU{m7gn}~x zz5*-i%ZsyMjUM)l^t-|bmP7GT2lCOFrP49nIKYnWds}|1fX^J2;JoXds}dbA3Z+U( zE7{i%JzheJr}Q(Ge(ubuk!O)@DgStcdm(3#j=4l;hKoqh5)GT;Og^*(^cVXia#2i}?xtIb3Zty5I^`e3Iq$Od>?fE&=i#G?CD{>5!vC z_@BTrs?^m-?=JBNvejv^Pfp(6bc2=DGi?1?G<DPrKM_!)6uUru_( zHrU)Vyv)~guf5#~JJ?nYi27pr;a+eer-3zdy?KjF=gkGS_Lp`xkfRkxj&)uAufPG(xfcNoo+r{`@)fyERO2>F#Xmtdr zmZhCEF1I@;+)w}d#xH5cqlR?dT7aGADw>=Vap~^}8$Ktp1}e6+%9`za_2UBC(XRA# zi$JP4zM;k1)hC2<;J6woUW0BQxPZ!lDOv%^N&D1@2n#bb@;OI(&gp#o%+S`KpDiVj z{ulo)0rw{|j5K&cofr~AG~+FpH}jgxNs&u1G1#>d#@;mc`=v46#dEt~ad`>df+)m7 z3bAW?*G5!%u`xHMtHyx$HdPu(%2!s9nZeMsExjOT^Lo^a#jSKtS}cFM@h_!Ga-&fw zw6blHsk57iSU4nXt{@JA&HNSuwMM~C|4AJzjEa5>RjrZQg1qwPN+G{U%C8Go(hpzl z!;4SESi%=SVt_-bc+@n{ruZkjmT$!9darl`sdhO~3kM8YRxlJh+xd#Nf*P_;Xw-0p zm;#Z(I2?Zt>k5xn+W+Jrt~*HJD@TmlIUH@8H2u=*`EFEY3up35rO(wo*lG_n8c~`b`*~(YB3x88G-|iq5=wx^QhIKQ??=BqrAAr8j!NMUw#52>!Hw$8?fZ#0N|WEe zcgs-{E!GobT5`j335(04?*2D3liQ(yQ4#+`xP~;U>Vq)|LS|}dJjEUJE&fnPc4dYO ze2`SCixA%;kB(*r55+i6P=is|{$ok4(YK4#RGa^)snokM;6J%#)HDC@*`EK)b1wgP zWRCv-+Y7dfQ2CY2hSfZfBZ){roS#_JW~AsV2=at3eGEBWF9D*fozVr!eT`>({W!q+ zJ^+_bSh>;KnFDcF2Sh@TG3{gw{$rc8tSqwqtiS!K zwY7B)XsZ!jkoMwqi?SHJPeL<}Y7Xaze>)w|-?JteR2*V_rKb-oX}ZBf9Sg$tC)#It zS3m-3yB%P+8*lCeE3>nNmYL~fh>51}V@dNz@r1a`?s8rUuVfuBbJG*YZieCE$oOli zHy+jJpDpq4=5B}cub}etTdOIprxR6BAWT{2IgH2lEWiCl_P7LJ(%DW7AW;&fBnc`G zX0;1>sdPAQa^nZR|Bc9`@EBfjNcb6!7j}MhPO*op=JuU=Ve0Pil7?ecOv95*ldz;# zY=SuCq<^=I$$**v)`xV;W;;bVFMO}}IJ^BMI|Rpx&1^MRXgH&6pM%RSYh3bMc74u? zKD$4wA}TV2YavB4`bC@sYsGb1@fnqI>)NEshQ)Zi(L^74dhoI9ngq-4S^1R)>z<_4p;_B@qVXebbAeN~h@%P8(<|5b9uLFaM!s6qd)p4=D#||8%VGw0C-&yW z$>D$eUJQmD1>?jii{gtIEG5YAy=E(^gAsfaa5@5WMCB>N12-oZu^a8zm>zcF(KR*g zVH(l$;Im_V)mzerc{@i%MaAwagtCg}n|PVd;5YdZX+w-jmG1d5rZXGqj?WDO~)cF|-5ywVr z1=+lp{es%;o4?f%FJ%>l3 zu0+|cob~3PX|=396BnA4IH#I1PdRMoOIs_xx_^7AG+Y~I~^G@q?(V*g%UzORjR zI@NABn*kxZLS)IE6;!; zlud=a5;lIv(gi&F9a$s2=DR>0a7g^8NL4zQ&wi(~`zV0@!14Jegk04?Io(XS;;W34 zmDowI^WCNme>53So@ctUvaBL45YA7t1VbJ;le=Agn?B+&&M;>3GQZ?!VELs6h=4UIO} zywHbsXN#=JZ0y+z{$t;e{A!M)HO#MKVa?%Nm2|^%Q}Oho`n< zZ&mHND_D|aOi$5{itBLcX3I4FliVK_Rg<{YfP*eNx%PEMEtf;>seMMAYZ$EC6Z}B7g`0v_STOSL`=0Xgw*o^?cAl;()54eM@sg`03_75y! zbOFBc(j=J*XFc385#zJ%$8pd7?W!D43{RL_`0;OEF-opIp9JLM!uP7SrUo-5WO7Zi zcE0#C4hNablJjFhI{;+j1C!fzX#v@l9>34&?Gnd}1V&Xz(?}IFCE>fvx4=gcM{`p@ zC*^eJio*K{|CmK3lyjLVng49QvX;?u`z#N%wQdx&6Bb#+m9M?&WR_ByJh~Tm^dqS# zv~m7XHuUp-XO{K6_nsBg_BmF$M22&JCGDwC2FsqJo9V^C{-0m=km<`z9wSr1{w#c# z_ts@UJlE<^B^CD0e}L+Y8MS%$aF(tMgkmuBEaKq{bP3LO_DpvzC{fkP%OJb+(!;)>=#ksPGXv4rY;k16P_WATDg67@T+wQ`}xR0{TBp1266*r%{ym?dSUs2OZ zTYbJy_*i=mQrr~ezC7bx+E%>vu$@QbGH2^cHMYYC5xEB8i%jT(+@2#&S#J@Zf`DSH zwwAB@B2#99=KL=@J=i@B0{=$F01>aB6e0P~=f;a_d-vQIinG0ic(#p~H`GTj5E|~M z$rs*fJF16}>Jw@XLL>U7A|{%b!E_PdwFy7Q=L?niz0RqNr-wPmGPtqj(eUHwsU1Z& zykp}^3xx}FrV>iRVC`)Bphv#__HLP26j-35WUu`rMVM?cbXDNUdMzf|?1cUy>)m!n zog$)^HlHU`dt7bGu)Q;&8t+s(UN2_OI6}MPpCMvte3mWMAMCfn7omDs*?yw3E(^Z9#$_R9j!YU@5d8DHGP%Z%HqQcF&}XZ8S;U0Czb9|!wt zgA(0O#x_|*hbO0nZ6#H+7@w7C-*gT_?YtS_Ps42Z{-E;Sh4psUD2(Z3Q(RZ(&Ipr$#6sX|r?LR?{|6!AOd*ga7lYtVP zt>Nw^{SF(ko{v9hq@#!qpZ`f2wvFRtzFMS8bKS!Q@#gURr`>Chnfq;hgESR#<2NIH zHy*meOq(3o8SYQoFG-@kX{Ow-8g@hygt$EmeyxIjaeRpw9D4bLGNiaI!g8Db&yMo! zvG>>J0OoHud!0->Ki~W$<7Kt0ay*A;OyrcU)VLBxC625avlr5?U9oKit$m4!kN$h} zv>nI2-NQqOqmfgUhTggILLRT-LiQbG)8}+L-wUIma~aH<0HURY=&wIl;QPz^s^gBV zs<^)50PsWVMLQRGNx)ckDvcZ_PU=_C4mzAa%2Fhhh9k#Q_|DZk*Q*wqOQrT08a~Db z3{dVFe5@B#hXN?a$4mSGc44#f*rV3%2n6mqqF3b};HGB8sv`DNgc|>(>WaUd%z9?B zip74iYV;@_n?IMki1DspnZP=qbmwvT>nTX-1KYQIF{*St1(%$&HL+4|8z&9bgq+BM z?XB-ERqqZUzZqniGd=?W)!V%$T#Qii6QTiSIysLD1J4H>Y%u&q(dQ({xer&w!3d(Z zdw1DNFI;6=g`q4wtS4In_*{Z}*Q$@B4yM`_KKCJUJRsGuo(F`7Bf@*$7)q;3ZrIzQ zNFfl&4@ND{)RR>5_(i<6w^F)>CCcwh%U4{dhjHk*wRM42#DqE*Xq0I^-fO{ckGm>Y zA@CKvlFi#qT_OSY`qS9F2^o2TvtRThrIe)Y*L~xnFHw~H;{lJt+49Vl!v>$SXRT_A zZ=vj7t5);@=ldLX+4K9fkR4jmc(c~281!O_fgDY0boe%iO1@RCT`KeDs5pus-X+Bl zHK}2Q4yRMmRE=8Dp50C)D>E%CWHlIE@_n$wxavTI&9ty>lP=;~+WLl| zzmerOPgrWL(L8lYN{FK|)&+d?s5JJ3D{_21rZn#>7tkwcXv3g`yDk5qBRPE>W#jR_ zojUm(>@{@`$j72K33tgXr{E=48Yy>B#_wOp8dO&6IH#JO9K=r+s3S%3o;eW9Is{GJ zFgxgRR-5MP914#$&7d4zGFY5Y9H6XQca(1JEl2?qg-P!z_xfW;EAA7*38gKR6_U14QS-ZAP>$s>m0PxCy7uJP8p$(#JX)SGb~VlB)$$NLm;92h zG~`+wCo&Oo_}$)i7Em~)?KNUGvwaPthkbuky}d`O%2MjW{`K2ZI@(KQis-786rn|N z$rdWD6VL80nT8albg`B2`Rplb?@KxJl9{>>joX|_=n{NQ_#4;z1z^6N{L$x))$u_W zNC|t1lIKEN&Wfa5oT`fc`a8cHqXtFbt0!pzT48?!KFG*(WK1Zl&9RB#+0IR+plm4u}}4M7_U%ONr)})(=lw_Ux;!ubh^5b4^*KE`%^9 zDf}RqLY*Q<91{3p3@Z-$~Nhsv8IdRE?WZr$p8(f%Hd?jBK(Fc^AE4WY;7ap(%6M)_om9L z`Fde|pPyY(KXtkg6PcwtcEAA^UazU|)^6q7PA>k3PO(B{Nzp~JJ?&h491{cgmzV>I z*P`uD(0hYK0!k4hk?C5V8OO6gQ{ zou;q1`iLg!j|`ujbjTRd+;n=tJt5rpfaos09PxlY$MD8uP&|34tLr=ia*H^RAGy4ag!RcL}tQUz#xcmLNG#yloi(<4wpEo{_Km|(*^f_Fq+7?bSq$H=Sp6Es7gXu^y9XDu{snD{;v|)Ry z_ZN$CtIk{>XbrTCiH$NU9@cj>gyC%056SX%XE%t#DY_wy9q}HOjm6VupOds5Qa29^ zu|052$x3+{F4s%m2f%B!zIdHo4k-KO@2zArC~E{Kly6yGVXM^J2_x4}o5~Q*g2^!O zXo+c)CcQmBQPhfXc=%wLRnsl_activ+Eu8k{iH2NdtK~ah2*DLut*iI!6snj8A65u z`J)zPrju*r1uoQeZr0=#B=3J3cdxDgu8PS5_pA(N*Jy;56_N8atekJAg8^h-C~ZWl z)!s1R>^Fd!)<3jd>WLP2C&h{Lcy@k#G(!*P_}9IC=UW)9MlIgh#~DWtOYy)T*$!Hr zD$ZD9{h%F;?YpCA0!~js4B-dVO^z@r1?VXw{r)C@uxBghCzk}MM$ z;kG{tcLnn$zC=rDw0*VvR+x2~6)=)dbTJn6IVes)%^?&RXgEMubv7|lm4zMBj+eOq zV%GG%dsFqx6su6%yH5{~hl1No2A(>Z-S$$$C84i=eo4*azi;)JKJJr%z^!en4sgd_ zUE}RV1CB-d?px19pYcP{7o84T4mYP*IZZ!z4$*3TGjwM*{*zJ3Z5MBH_)Xx~I*7hy zH@ng3<1IM9qZaVZ^|!II(Kq*uPMQBKP`x43=k8;+o;Fv=s&I0c#)UWp}{VX)f#^Wrh?s%NHpRxXbNY6+p%QmeM?Yxyi+QX+idDvo%%h4fxGtzM_jzZe+_S zi0vD2=Y7CH`JmEQ)Cqw%j%|8L>hv92j*I{Avw@YO*Lq17)SypteQCq~)enWw13O7_ zUKzDdnzY=mJ{o?avNXMf{jIx8RU25M+(UmB3v+;_4nh)Z0`>&Hzp{;VlKZ0iL|)HQ zv!Dqeu<1kHlJlHIngHF|OjefrJFE_ZJWs@EY_(xt71J`Kmq>3eu8)q@S-yJP6iuU? z`*zep(LJ`Cg}XJiPBrT;cTv+tK6&m3QhVr(!njvsOwJM{C&)sx0IOWxH6+whvrw92 zT`DK$x)=M5PGPB?e!l4aq4X` zk0W2dd0RWG`e5V5!?q8+w~2ht3e^5FxEJp_qzWE*xqn|dvMr@>S)WFX%ZtFXurf5E z>#U>t4i^-#b${sD?E1GIeQ67?(d28p#*Eb$x9ZLmC9W|bvvC~~sQRd5@1Z9JDytSB zhD>}TYx4|f5B#r{4L@A0m#|2G^h<;N1&7Gryb_u^){5t+QF`{y`jGty`Lt16o|DJz z<+sU#8=f1Fpw7k~|4eVE#CPFCxRV1lN*}GW(97&K>@e?YeY%MbNg0M!zmWV`=)BEp z?CMlB&ExoKiP~7 zQ{}na?l+)KicD=zG^(x&vwrZtDRHX$TuxCapH`E<@EMfJU6$8Zdo973W@(dsP&LAE*~ZNs|nZ-&ZSfk{xSkp^u6f!#k*j4Yg}yRUO- z4&tmzOyUKk;`T-ibxMTN6^rtbnrl*dnQ@l!ebqLXp!vPK@i+2dGz)B$BEC~8y_bk1vgC$LHA1v-n~w4@{}vMH6S7SjH4ijL({jC zV<7o&+eZS=c4U7--V66RDj0eOTfM4fu4!L?Bax{Q^DbNs14FWX7L3uqMks3u!^r;p ztn$)P|7UKy6J!fAqfm=ta}b=G{PKUwJM(`i|M=Y#4YEYYzGfRcBZM&7jh*Z=r7?tT zBl|j~vhQmQW(J9rv6eO2#y*6SrI;*{l6?$k`hLzY=ZEwAc|6X)a6ex6`~AGH*L7XD znPlvmBwvdD^i%#Sg3IQL7!$3sZN4f4JOsA`)KkYb@qf8WOcO!gnO36htr4h(0widH zqIA5G40=om8+e{fm--{@K#s=qc!f0JcAOT0n*IHHcHkpb8p6nr)j!UF+ruK!SG1sI zwXS?jdr#)vr}Jg$a*aA5yu^alx2c$o(;c1^ztM$D z$MK54oxj}F8W+B%uB~OSq50ww9g!va+ah4exP{hUT@c$NH&F!>s$3b1@qAw`!|*T; z0bVLP5Js-9=0^QaSsGiz)dm73G$#h&LISlla={Nq4~cfBp$y^APH!i{w$}NGB;TeP zo=-oF2(sa6)ZrJ(qm1OjZc=l`K(Hoi6m+FTrsU%?_run)F} zZLa>|p3fL-7NzRD4?U5H5+Zuyr`n%Bf0iScqZGbndmu+@o$;_8SAT_E8tWXoya%E} zN$eb`rWr-1MH|MVUAyucjbn=%3niS^0%c8+7VVNmT+j-zY27jsRNvBx5O11Z+X~%e z=uaH>6Cy3@4lMI!#g$`pBK%jG{-S4F)rH&-F905$fBTl%QpE#`giC0z zc)ttWydTl&3aHGqc;d#g)Dt>MWqeVs3W|*Ysz|}zx zqu-TX*-+6rTHxYkwLADE%52M8MP)Cpl2Sk({R2w`aEhiP5jC5}{o3Z>E-gX2_>2c{ zOO==nJozSn~aGqZ*h)ky*vcmd0=cMe;E2ZDX5MB z{h~`5jhUoGUYHWh4!dI_P#S~;BSg7BNAzN2p{tX8b}d|cl%-0(Y-jXQzGkFMI>WO@WOedX%G_xxL1>ws&U`D>)x_l<#j-hrE^!RQAlsNwLnMqox_Cz}Pq!y@ywT zx+ZtZqc9y4v-)4p_F$jWw@4fa;udUIfJnD;_f!=yXM5Zeh-L<7h^0s4eg2}N6`=to z(S0!{+Yb&n;p3AQD+QU7GT#a^{Ud$G_f;7hTlTcLPo@?}Wb)Nc2rk;PeXUgUMkh08 zAPNkMjpeiskKU0pFenBL&`Yg?)Gn$sdjPa-KB8`Ar}O|_h@$7Yo+=nVlmlZYl7*h| z#DYQgjogSZkSnDA@ul<~g%-E3N9`)r#WV7&E$rXmGm5#?oU@ysUL|*-$hxo(X5wRj zZ~M73nYHi{)pD>|Z^ zD8iFxy5hAuH|wcqY6M<|ciXi-X=;+^W3~Oe_3(F-G3Z|1fg9)QAnaTV*+B5+?V|1v zai(?0p^68i0XY{~K8?*;QfAt6$v}Ios+{1maq;Z3=X7~axbS%(pQ$S@p;-(jj(g8j zP{ssvJ8l%90e7Y0Y`NPjQpc!R_Bn8WU$z+A(io7sz6y_7@pu&~+s&K0V_cV>^js_T zc+=*325C8kUhiQ_GcOV-1(*RtBZ}{)Ahgzz>=>Ctx3AR@0u?*_CkR(aeWx?|>}LE` zL3j{7dp3b!j}h2BB7~g%^?JHZ32ylVKuZp$q2x)F2D<3fxT7y?wsv5Aj@~=DVV!U3 z$nbMqFi<;x-bxsio94DBdD*)2O4uNsl#_GG$$LV|dHZziORU21=L;#Y$@>t)@vsGf^{ z)df6?o5?pK}EQU>v%UuwRp+ zD+UJVb)l8!XMOGP1&f|TyG|;)e@}{E&t09F7E>QGX!V*4P8oK~cy&kXcQ;KRu}=zF zU~ORK-?~2jwvMwzK|=zeB0XwaK@B-JFElS2Lc2n*I^39M^c4bcMQI2=-;gR_>4ySR zVV)Nbb4$ZYXa7XtM<4JF@{`WKI|i8zmzU^fG!bm*uz)N;;j^9*GY!*PZ-``JT69~o zdtNWpsI&cEMqH!WQ?>IW-JeE)DX*4&`}0X9Nd>FdHfvRJT;XV&Gh&;mz-s)j;&0EU z$Uq%#C^VzWnG}3JB~Np#h)Oi|$J`8gw?}{^Z%6G4?9-DgOo@#9zBd{px#=snHvBxV z(cq)FN}}=xq}A^FmO0wzIaOgUN^Im5LzS(fR5x;hx>}r0w-h$r@J;c&#)Vm5M7>{D z2LU2{|JtCCh3|>G?(dZ7Utq-1Q(FP0Rjy@UH(uQ#9$KruO&w`fAeSQ0se`oeEa^4OET;VJ3Wtw zsDgFu!F44mcRKr`Mu_0bagcP=lFWvs;DwdDl+#wbydBEgQRGA-ZwwRG&@Y6?|L}tc z5tW=U#%(%Y=}2}rHBlHjyR%CTM#3Y6mQz_tRYtHf;BSeeRLN8tg| zH+dcJa|1G}bELF*be9nyANNFGa${9_iKTI5mS&Wu)(~c@ej}&yBD}qb$&}jCvl2Y^ z68(t+mvh+`3-=6}ISo4-gx`%VjXWw3G>!)^KRHa*ACdTe7aDqRDVgN;242k5!t`y*Nmv60$A|S64f#!TiqXimj2B*Y%XS9Mb6wI z`IP++E;O1i=W;5|2}NUVPqy}iypRr?l1z={(%B`gk4N;RI4be%@Iuy+!`mRX7M*A( z?c#`}Kk63*_WW$#Q#VErC1YgzeQj7V9Hbm_urT2LWtA9(>^AUCgZY%*4n%l;5TQVH z<$@27vSO|>*SvA})9c6DyP4@6LJ-2T zI#OCc`Y+tqD$F|E%49+vWAjplLeR_Hph)D3J7lGp+}ADIqv9FfezspfUb>-QBP{E^*bvC4AgQ0z7{+;r!LL_IQpRcgzFUp9KI;sWzB83(+I1lJlcpG&Bx^S zus`3wm)!&Is>M5L6t4E1cEZVK(whXH*?&q zZezE<6cfy0-+*C+hN(@&ceTvYLCEUM_MCy%Il0bLqC#j18w21mka7K}GUDX#TVx^d%V_Wc~aSHY$Ud1SZ-@P@VL-_Ct~(}gOTREM2-+9HAwoC zUX-D8i$;0yu>WdgP09ZIu8D7zdgeC`3mxCN7n%%;T_Hp~sG4NEFQM+ocVO$?S)rvc zt5n|-ZOeGT0AH9Wy-$_zgd%=n%;{)ZjPUt$LO+Khp&s$VkCZl^sm6Fa^}Q`;DXR;9 zgO0v+Ru$W;eK!Mn#twGgtPDcZj#LnRmri4_Y2SjYht)kh%&98UpjMg6;zG6JqgFHa z(#a3Nf@pr&vYgJJ6peh`DR35&IoA4Cp+?|7l(H*I(lzoXoA65d|JL-bzx__?!F2%I zOvKVEZ|dsH`(*>F=_lERUDLs{ZlMd>FxrsYotK(AaOvTx_BtQY3>lrMcGwV@J(!_AzbE)2D7r-aS}qt2ar=btGL z{$2}l+gQDZzl!JXuc8mzOKjIg6)%gy%^Ym}NFBW7A-z3t0msPBy^OqOnc*&M)O7J# zTINKjrzXRP-Nn<7g4)@Cr163YJh^$r9;gQY4Rmf~3`1qC}VVT!Do<*_x0_z+T?NzFA^3~9qO>5%m z4nRnHe)j{BRAqVRd<&-$G|uQ+5!fJ2iMc;Vn_`WzA8{dSG&Eskc!gqjC1Y*O=(JFG z1`PJSWhYo)S6XYlU~~1<*^QK1krD#pyaoeTWK{qI1w`wjr9{(MiXK}Xd*r%FH>O+W zYPr-4{GnoeT-=lanfz8UX^Kon;C~*SXF=qFGh7&EfLx#=2K{(XpDx+MXg+=ybqaC* z9((p9P{U;lMF8nkw+ubqa`!C3lKVPgB|l8}#nWwP2pd0wflmuxWDLYHrg~h6&>BxA zjcAFFf~ZQ2qAgFVrUq)a_XP3G0Jt=^$8?{SW(0LS0#Bmmh+$WYkwJPP6!FYZ0b`7b zhWi%Z0o%UgS(xIwq2^Oi_WG%~SYHIz4+g^mm-966}(< zOfGKA1AoVpp<(Hb&Y{k6nsT~EK}4JivkR%lqksV7VKCF9A4h$z;JM_JhnXgqKQaWU^xB z@KXfHo2X5vMrkkUl|WD#%Jm!@DLCi3M4$SO=A(J3l}J73im%vGOs83R>CQ?ZV7$=n z9mO!YB|rvo!%Q_8yIMdDbQI8Am$8W@z$?d~ZqeG=esC9VU+ETvQPJGMss^BIc)vxa zJiWI$zZpsRQ?=v%PKzrYRA9DX%qO7(HElBxFfM}X=ImCvk`ft&SGn-~c<%pDucqmr zp@i_uBm8F$i^(~P^mQ%CLx1mLn58>Kj+Z4OZcv1$SCc3$+Wu-OhtC#c6V;xgUIdg< z*0e<>lZfSR=&jFsF31Chot0qx-R~k@Pokp~zlY-Lr4Efr#~?yvwveUtr~=04LOwrx zUiGU-w;t6+ys9V7FMf5Oi*-N7J0y`~58P|~ZNrU=lbeeGvlOFY97u-V*UHM~w=$Ia z0)Qh!ENX@hmTDkf8#Jh>{|QL1+b^M%_dTR`>1Z~VJlh9;7PQnCOf~?BTT%0-=N?RD~FvaM+=-&_fFM^VGk6Rd`s``j9-n~mgXg~H28JvoyTPT zp7D!=^!$MlvLI&e!F$~)*mCtLsq_J_-=M*daW_(b2bJKQGXMqUJzM^1d?<*4r8Cyn z5SlS2!pt%Oy{q{=&-P_T%n{RmT2gk8zO=OTU1?`t+bvCAB$2fM+aRH^D|nzOaFak) zrIF8t(M)H5ELhX; zaD%eNYx>>-D(Lq0@q8Y5Rd7kK2s#l*EjUKC9741@m>R?e_maYN=7&e*EQTyu5)nCX z>RW%%gAR?=R7?wDdXxpat?Zu4GW~$>ys7sao16MBamtB3LIkR= z4U9sV}it8>9D@Adyc4596iH&P` z<`2xDI(aO-6qJwS#~yBGWGb}2^*?c6bB$YM}ljgKHp}R8MCz&bW z?+7zf{~UMkA!$IDH*S?KQVQiY1&S+95M|iEK;$I~Z5yRs8_4$1kHi%1jo#-4EKALD zhpqLK)aTXBE%@kT7~!@7DN}!{O27L9cx}&R(+{l{Cjk=XyYrM{)LZ1iTKi@lVmSW( z2?K5Qs|Cr9e`;`h)|f?Eoz)nWe_m@Bv@Mr#v9Ztb4CtpBa=R@edqr0ewglp-&(oy~ zQQhXp0|*MyTnAhm$z3M1)1l!MA@=h8$1@;V`+^WqJ@0bDYTyCGKq9y^O54U#uVja= zB2NST<MD8yZh>$jVO)YOk(8d8e~IN`5ZP$}Y_9k9Id%C^)%a+Z|9g+?(k&ooOW4 zruJUJgjb?~Z2rVz?L{FymRUnUtAGqfqF9)zbK+&6mY?XoN$9z7fm&AYB~7yUAonwP z00uF1(OXQ!(PAPW%;dRWa~9CdGLVKd`#Ves^ChKIL&se29;kx_=I`^G0iy$(Y1q*P zPf8Ux@^Hol+`UYXZl}W<97i8h2TSPKuc9r}g-G z+JK4^r}#pCo8mSLkAIDTH@fc{3gcu84f=yPvs7NZbp_x;8M3BcPj@SkN_f_>vwc|r zn(ZiiSLl$!YXwX^db>YpcguDRuR2)%?Xl0#00~aG{Y=DFtTc$yGfqb(U}1q<^-Z;r z?b1c2;BK6K6FTbwd{?%ly<=8N zw9K}@D{YB*9V8#UGJ;Q*e^=c~%e@gCE04QLprnW}G>?5$0|E^_cQA~qSBBs(dGmnx z7M4#V)-==u$aJ2UvLspo1n;U6`*sV%v%L@#JtU(OL4?xMm_rnGR4Y#Q!*znf|NMHt zGqxyV(#ypKUvoyfF6a9sSI2Uf-75)|mnHvm{#W{3F>Zn;@IhzwjjHU5r(aBw@B z0|Dd^tz_Gs>}hnRgWS0Cv4PjaD3$31YTb{eKjm5HmPN~5 zbCL=_ATVJm*LmBlikTabm$|xse0cBbmCACR>%!WIkme`C#Cu|%OA^;-*^V+P_*@{H z;p_3caDAJOGJ)fN;f*`HS@|3aTCdLnbsof4jK+Qq9N_eCJ|~TntG_LSP{(GoHB5WJ zb(e{O^C8O^n|9Buv-qITH9f?>!T%~O#7t}!c2oYpsCBQ0u-6|4KCf3!3o107NfM?EnA( literal 0 HcmV?d00001 From e2aaed3b86e1882b115630b8181b597fd3402c03 Mon Sep 17 00:00:00 2001 From: Peter Hessey Date: Mon, 13 Mar 2023 11:46:36 +0000 Subject: [PATCH 3/6] Add AzureML tab explanations Signed-off-by: Peter Hessey --- auto3dseg/docs/running_in_azureml.md | 33 ++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/auto3dseg/docs/running_in_azureml.md b/auto3dseg/docs/running_in_azureml.md index 87c9c818ed..a0659e72a0 100644 --- a/auto3dseg/docs/running_in_azureml.md +++ b/auto3dseg/docs/running_in_azureml.md @@ -94,11 +94,40 @@ Following this link will take you to your AzureML job where the AutoRunner is ru ![azureml_job_tabs](../figures/azureml_job_tabs.png) -### Outputs +### Outputs + logs + +Under the outputs and logs tabs you will be able to see all of the outputs produced by your Auto3dSeg run. After a successful run you should see the following folders: + +```shell +. +├── outputs/ +│ ├── ... +│ ├── datastats.yaml +│ └── input.yaml +├── system_logs/ +└── user_logs/ + ├── mpi_log.txt + └── std_log_process_0.txt +``` + +- `outputs/` contains all the files generated by Auto3dSeg for training the various architectures, as well as: + - A copy of your `task.yaml` (renamed to `input.yaml`) + - The results of the [Data Analyzer](./data_analyzer.md) in `datastats.yaml` +- `system_logs/` contains AzureML - related outputs such as Docker image building. It is unlikely you will ever need to access files in this folder. +- `user_logs/` contains all captured standard outputs (i.e. anything output by `print()` statements or loggers). All AutoRunner logs can be found in `std_log_process_0.txt`. ### Metrics -### Hardware Monitoring +Integration with the Metrics tab is currently not supported by MONAI but is a work in progress. + +### Monitoring + +Under the monitoring tab you can view metrics for all the hardware being used for your AutoRunner job, including, but not limited to: + +- CPU/GPU Utilization +- CPU/GPU Memory Usage +- GPU Energy Usage +- Network and Disk I/O ## Additional Configuration Options From 8074194d78b872dfd75cbf0ad1d41aba3b92a2f1 Mon Sep 17 00:00:00 2001 From: Peter Hessey Date: Mon, 13 Mar 2023 14:35:24 +0000 Subject: [PATCH 4/6] Finish tutorial draft Signed-off-by: Peter Hessey --- auto3dseg/docs/running_in_azureml.md | 40 +++++++++++++++++++++------- 1 file changed, 31 insertions(+), 9 deletions(-) diff --git a/auto3dseg/docs/running_in_azureml.md b/auto3dseg/docs/running_in_azureml.md index a0659e72a0..d94e1f8192 100644 --- a/auto3dseg/docs/running_in_azureml.md +++ b/auto3dseg/docs/running_in_azureml.md @@ -1,14 +1,14 @@ # Training Auto3dSeg in AzureML -It is possible to train Auto3dSeg using your AzureML resources. This page documents the minimal set up required to utilise AzureML hardware to run your Auto3dSeg training tasks. +It is possible to train Auto3dSeg using your the Azure Cloud. This page documents the minimal set up required to utilise Azure Machine Learning (AzureML) to run your Auto3dSeg training tasks. ## AzureML Setup Before being able to train in the cloud you will need to have your AzureML resources set up properly. If you have not already completed this, please follow [this tutorial](https://hi-ml.readthedocs.io/en/latest/azure_setup.html) from Microsoft's Medical Imaging research team. -### GPU requirements +### Graphical Processing Unit (GPU) requirements -The GPU requirements needed to successfully run Auto3dSeg depend on the size of your data. For running the data used by the [Hello World Example](https://github.com/Project-MONAI/tutorials/blob/main/auto3dseg/notebooks/auto3dseg_hello_world.ipynb) then a single GPU with 8GB RAM is sufficient. The GPU nodes you have available in AzureML will depend upon your region and Azure subscription. For larger datasets it is highly recommended to use nodes with multiple GPUs and more RAM (>= 16GB). +The GPU requirements needed to successfully run Auto3dSeg depend on the size of your data. For running the data used by the [Hello World Example](https://github.com/Project-MONAI/tutorials/blob/main/auto3dseg/notebooks/auto3dseg_hello_world.ipynb) then a single GPU with 8GB VRAM is sufficient. The GPU nodes you have available in AzureML will depend upon your region and Azure subscription. For larger datasets it is highly recommended to use nodes with multiple GPUs and more VRAM (4 GPUs with >= 16GB). ## Configuration @@ -22,13 +22,23 @@ As well as the requirements outlined in the [tutorials README](https://github.co - To be discussed with Dong et. al. +### Empty directory + +When hi-ml-azure starts your AutoRunner in AzureML, it copies *all* files in your current working directory and uploads them to Azure. To avoid uploading unnecessary files, create an empty directory and navigate into it by using the following command: + +```shell +mkdir azure_work_dir && cd azure_work_dir +``` + +**Note:** The directory can be named anything you like, but for this tutorial we will use the directory above, `azure_work_dir`. + ### Configure AzureML connection - `config.json` To connect Auto3dSeg to your AzureML workspace you will need to download the configuration file associated with that workspace by following the instructions [here](https://hi-ml.readthedocs.io/en/latest/azure_setup.html#accessing-the-workspace). Place this file in your new directory: ```shell . -└── your_new_dir/ +└── azure_work_dir/ └── config.json ``` @@ -40,7 +50,7 @@ Your directory should now look like so: ```shell . -└── your_new_dir/ +└── azure_work_dir/ ├── config.json └── datalist.json ``` @@ -70,7 +80,7 @@ Your directory should now look like this: ```dir . -└── your_new_dir/ +└── azure_work_dir/ ├── config.json ├── datalist.json └── task.yaml @@ -78,7 +88,7 @@ Your directory should now look like this: ## Running -Once the above configuration is completed, to run the Auto3dSeg AutoRunner in AzureML simply use the following command (in `your_new_dir/`): +Once the above configuration is completed, to run the Auto3dSeg AutoRunner in AzureML simply use the following command (in `azure_work_dir/`): ```python python -m monai.apps.auto3dseg AutoRunner run --input='./task.yaml` --azureml @@ -113,7 +123,7 @@ Under the outputs and logs tabs you will be able to see all of the outputs produ - `outputs/` contains all the files generated by Auto3dSeg for training the various architectures, as well as: - A copy of your `task.yaml` (renamed to `input.yaml`) - The results of the [Data Analyzer](./data_analyzer.md) in `datastats.yaml` -- `system_logs/` contains AzureML - related outputs such as Docker image building. It is unlikely you will ever need to access files in this folder. +- `system_logs/` contains AzureML-related outputs such as Docker image building. It is unlikely you will ever need to access files in this folder. - `user_logs/` contains all captured standard outputs (i.e. anything output by `print()` statements or loggers). All AutoRunner logs can be found in `std_log_process_0.txt`. ### Metrics @@ -131,4 +141,16 @@ Under the monitoring tab you can view metrics for all the hardware being used fo ## Additional Configuration Options -all other configuration options + some examples? +The [hi-ml package](https://hi-ml.readthedocs.io) allows for many additional configuration when deploying to AzureML. These can all be accessed via the `azureml_config` tag in your [`input.yaml`](#configure-auto3dseg---taskyaml) file. For a full list of options, please review the documentation [here](https://hi-ml.readthedocs.io/en/latest/api/health_azure.submit_to_azure_if_needed.html). To override the defaults on these values, simply specify the parameter name you wish to set under the `azureml_config` tag, followed by the value for that parameter, e.g.: + +```yaml +azureml_config: + compute_cluster_name: "" + input_datasets: + - "" # currently only 1 input dataset is supported + default_datastore: "" + experiment_name: "my new experiment" + wait_for_completion: True + display_name: "The_name_of_the_new_run + : +``` From 259dbf3d4db144a3e69d5db0b1cad3596152757e Mon Sep 17 00:00:00 2001 From: Peter Hessey Date: Thu, 23 Mar 2023 13:31:27 +0000 Subject: [PATCH 5/6] =?UTF-8?q?=F0=9F=93=9D=20Update=20outputs=20explanati?= =?UTF-8?q?on?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- auto3dseg/docs/running_in_azureml.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/auto3dseg/docs/running_in_azureml.md b/auto3dseg/docs/running_in_azureml.md index d94e1f8192..628b377a01 100644 --- a/auto3dseg/docs/running_in_azureml.md +++ b/auto3dseg/docs/running_in_azureml.md @@ -1,6 +1,6 @@ # Training Auto3dSeg in AzureML -It is possible to train Auto3dSeg using your the Azure Cloud. This page documents the minimal set up required to utilise Azure Machine Learning (AzureML) to run your Auto3dSeg training tasks. +It is possible to train Auto3dSeg using the Azure Cloud. This page documents the minimal set up required to utilise Azure Machine Learning (AzureML) to run your Auto3dSeg training tasks. ## AzureML Setup @@ -88,10 +88,10 @@ Your directory should now look like this: ## Running -Once the above configuration is completed, to run the Auto3dSeg AutoRunner in AzureML simply use the following command (in `azure_work_dir/`): +Once the above configuration is completed, to run the Auto3dSeg AutoRunner in AzureML simply use the following command (`cd azure_work_dir` first if you haven't already): ```python -python -m monai.apps.auto3dseg AutoRunner run --input='./task.yaml` --azureml +python -m monai.apps.auto3dseg AzureMLAutoRunner run --input='./task.yaml` ``` Make sure to follow any instructions in the terminal regarding authenticating on AzureML. After the job is successfully uploaded, you should see the following output: @@ -112,6 +112,7 @@ Under the outputs and logs tabs you will be able to see all of the outputs produ . ├── outputs/ │ ├── ... +│ ├── ensemble_output/ │ ├── datastats.yaml │ └── input.yaml ├── system_logs/ @@ -120,7 +121,8 @@ Under the outputs and logs tabs you will be able to see all of the outputs produ └── std_log_process_0.txt ``` -- `outputs/` contains all the files generated by Auto3dSeg for training the various architectures, as well as: +- `outputs/` contains all the files generated by Auto3dSeg for training the various architectures, the best checkpoints for each architecture and fold, as well as: + - The `ensemble_output/` folder, where the predictions from the final ensemble model on the test data are stored. - A copy of your `task.yaml` (renamed to `input.yaml`) - The results of the [Data Analyzer](./data_analyzer.md) in `datastats.yaml` - `system_logs/` contains AzureML-related outputs such as Docker image building. It is unlikely you will ever need to access files in this folder. From 0562e0ce78fd9c8ad728185fecdccdf9bb559dd8 Mon Sep 17 00:00:00 2001 From: Peter Hessey Date: Tue, 11 Apr 2023 12:25:33 +0000 Subject: [PATCH 6/6] DCO Remediation Commit for Peter Hessey I, Peter Hessey , hereby add my Signed-off-by to this commit: 259dbf3d4db144a3e69d5db0b1cad3596152757e Signed-off-by: Peter Hessey --- auto3dseg/docs/running_in_azureml.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/auto3dseg/docs/running_in_azureml.md b/auto3dseg/docs/running_in_azureml.md index 628b377a01..56efcb182d 100644 --- a/auto3dseg/docs/running_in_azureml.md +++ b/auto3dseg/docs/running_in_azureml.md @@ -20,7 +20,7 @@ As well as the requirements outlined in the [tutorials README](https://github.co ### Configure environment - `environment(-azure).yaml` -- To be discussed with Dong et. al. +- WIP -> hoping to have a Docker image integrated into ACR for this. ### Empty directory