From a5158aeec2fac9d80cef8c1d6fd49f37086ccf5b Mon Sep 17 00:00:00 2001 From: ErinWeisbart Date: Mon, 28 Oct 2024 18:44:16 +0000 Subject: [PATCH] deploy: f2d5c772c845078f879180ee7c586f85c625ca7e --- _images/sample_DCP_config_2.png | Bin 0 -> 25625 bytes _sources/config_examples.md | 1 + _sources/costs.md | 24 +++++++++++---- _sources/step_1_configuration.md | 24 +++++++++++---- config_examples.html | 50 ++++++++++++++++++------------- costs.html | 12 +++++++- searchindex.js | 2 +- step_1_configuration.html | 15 +++++++--- 8 files changed, 89 insertions(+), 39 deletions(-) create mode 100644 _images/sample_DCP_config_2.png diff --git a/_images/sample_DCP_config_2.png b/_images/sample_DCP_config_2.png new file mode 100644 index 0000000000000000000000000000000000000000..0dea4c989efc622597515e4813ff437df3825a0c GIT binary patch literal 25625 zcmce-^;cWn^9BkOmk^{lB{&och2kCv?owQe6)!HuA-FrmA-ETJcPLuirMQ>kE!?#4 z=X?KxyY3HJ>#TEuDjPuFrD%}v{M6@ZtV??CPh}){&%P|dmRue`^?y|U&0y@1$I($kxy9VBc z?CzQEK~(fI9fSo6=MT2FCyhKex2N_KbG|N#b|9d{`Dqse(Ez<)0X z`NQDPGNFGjqn&~9ukoS%y(HfUeS$v+Y{M^`qBcRXLsvUg&x7hYJpzj!Y#Xq;uTqe5Ki9jG< zUGGC!&{R_sj}5ozDL&rpftKC|;~Z2%ccxAgD~n^<{=CGVm^fo0JaasUagQ3q~Jrelgvn zzKR8a;j)c!Cfxx?ezA8qA?d$;!)Y60!-Q7zO#DheB}6;3jy~BoiWhCZ&N9zox&8CI zHV%roH(2`NL1O2~;)f2vbt+x9_E)i%wZ}_nAsr*lh$#Rnk`$Kvrld#;7(fQq8mlVFxF!Q?``-qF@?TUWP&5`RgTYDfIHClVx;U%yZGXl zj>J2W4`VcsAw@-1<9txXs4dJj`o14TddY_c2>kw`hJ)50m*$Zt zS0=@$XsNf4hf6?)_3fYY%Okvfj~T@2i>(lD58*oxRqdJU5u}KI(d15r1jjgt$uK6% zNX(sI0YPNmhaQ_j0{R4l+Qiys$wH|fR4k94@ZBuLZF3G#3$u1KLMV|f0;&_l7#c8TeiJWVzxyaez+VEXH!GG;gnmFg zCy^i57HiHn`-!Fu_A5jl9h`PR$QAo5CeKq786!e2GILRl8o@MEoj~i`V1}&kV`p>?avr2k(Eh&GP&m3s+A}vm+ zKS7`QNiiH_+RT%hgK_-h27Nl-Gu{K;6iXf{m3O@Zjvf)lM>;K2_1YGXg$H(??wY7eE!tRB`6vWKOKiZ z1ny5K{Kg+DcbH0Hr_|pRuYC*`zetWsb7(pDks7 z2OH5xU>mh~*vU{O)NUFvQ-dG6b__`b>p1K&0kF_?4&k6YACn&XH<$DUX$sWLh%-D)F$Gt=H8UP+c4$Z} zpciK&A+F)oa}d30QRZ2iQ#}IRAgSoSCeK&qfd=xZczy#lik3~wR5^8ol zaPZmU2WV+XM)P7Lvou-_Zh0$6wQVFN5+^CaQ3b{%{|#*`G8KnHo3F+_yaV@Fv?4k0 zHq9vcl1-C#Ib4ql%gvaJ-SyJ>Pd`y>1ZNh-wJ*#k+MAj?CTN6LUcSQLs z>g&p(f7kTS9*AdOZlE#t&l#1J`f)@~SOXF<0`Si!F3 z)^jKQRXT%%F7)Uv+mD>I@c?0ung{4 zQK*%HX*zoZAw|E4S?y_odL;2;LZ!b&2G&fYU4mPcb{t{LQ9qHCZNA}>+l=jAf}KhR ziSWpJtC{N+mjtz!Y3Zlr?r?@=^Bp-InnCG!B1N@mnd<003M)uXI&`rsWc-mqZ*m6h zal?2FVQJMbV)W3A->UKGwWOC#MSqeHtNC+SKSR;4(%e{NEZH(tmDz6*(3zf{@sk5~ zhQrTsF@Z8fN}?($6@fx+IC)Ntc7!eFDdw-x^5Wvrb997j-PiGNMMw;PPyQ82F17;IIIW z8{iP(A8>%f3HblQ@dO{5;=83oijU?l!&nrFKhAER`g)8VkZIgVrLUT=WRY9oKfIA} z0t!Yj#H6~BV+YdGuh9y8Kua@C{rf28k|M}7fsULdA?8{-w4(n~!jaHSzyeROmBg(< znkXN-GR+U7Uj>IEo9+`M$p*u)l}7>OgKT{_8fYViFAFsU>R$ow(aF`PQ};e>WkTUd zrcH|?h$HF5fkj5lUt1tbXct70lSC<=GY3wB9@Ky06AcW4E-t|wm<(I-LxBYmtZ9G* z2bJyjTbtiqf=q-^C~03d{phRvZ`>*opDC8CJRIMNaSGb8XVJbRMXQk(11AlGJ-9~R z)(&G0<9(G+r=V+jEoV{BmjqQI9q&QRJ0n1QN$i}X=gaOAXXc@R%#+oJHl2|Uy+G$` zpvxFChVl`zhC`*kf3bmwJ^hz$3A>eowA4OI3GP;iQDf2encFX-?xbWsR?zUgKkzScG?Qrj=mO48x$BKv1}yCHkK;Vjtg7O9Q6eON@z?3QS`Jy_E| zZ>_Sf#nmmAXD@%*vw9izOjXCUs@{_@P;jI7A+C3O{#oRt)8P$Zx`KVvU>~%s7Q$U2 z+Uw?%Fv_}Yf-d;k_?->cV)j!6_qK~l9%Hr#jn0Di2uk?3$({lsnwL9&DJVbGt7vUu`{nPK2b5(K)r}pq=}X$*{h_f3`}m>YClKxoYG<; z<}}9BAQ#W6gAgFXC_KvG4lQDLyv&P3O+ACx+n)*vbE<^u-VGh4*ud?I>QjuuN|IuiBa~9=i@~jp#B8%?eFDoljo{ zy>=IMId#Hkcs)}&GoYPUp`k2BzgUfUF{gWIuiwF@vBt(0!o|DKf^Yq-guSnY#52)@ zr02I&&dhK4M2@B9Px5o|h*lM z)-$eeN2w6)@qo5brv~~s(`itVSCxh*G~xmzcaufp9YLWZbMq@yLyzqHiB&r@?jb0( ziY=Hw5`!w+$HAi@{{zH_M>&8KL%WsRk3Y^2_}N5B#jHpS^H=Wvap6=x^xKH`+^*|l zJ!blviqFP~3^g}J*I=xIPKY_0A4~NdKLZ(NQq?~pP6#FBPzpfd(1IlfFomnn_#a%n ztA(Ly;?(S*O@$6W zGfb*4erk$;pNc2@u5_(sHH85Jrfnu(P7q5%NA50`ZgSHEqL{4*mYg;$IjMvYT|7Ae zSykdd9v6(rJOgic8CnrziXuotxCt-tktf2c%vNa8Y9v^THbJiqHwmhPc0oLaiubz{ zrX0FJY|||obN&M!RvW&9FHJ6|QNV%UVfpn>wpM@cCvVa6)$mc*I^qcp*#kW1G%E)tUIM^iK6-#EUM4)s3=WFVzS*-e znDKriHY-joVD^YrWLwwi4~L|0ReL)NRR<6njm@cNQoYH9ys+Yuh&rO3~ZfrXPQzzp&DJzS~&sRaLE7AvrJRWf`_C& z1~L_iQWUyP;;&rhRAL#uVgzR00Vq>?UTlS5hRgbSa8kq=F3x)&jSy4H+n&*z5ij7E z+u&LK;1pNUWpAatl%*ckAki@ezN7>Rz&Z-?giM0*yjx!qx#NfR7NbuuFr%5fBc7w6 z$sR3OnbC#oATaV3P|rKilFHsBM-6~OPjbzYXn=`K=;71T*YpyAkzT~u@L350z3{o; zwltZP>hX$yzIwjSYW*yG5MiOL30s;UFRsG;#sdiC8)*6F*?X~~;*zjUAOu7-Q;L#g z9HhPDKa5ohLh7j&jPW%5{Cu+W?D#Q!paJ5*P9zkGtfs7C>zb(SMFZK9GbV#{=(bTn z=p8Xjpn|)(*l$4g`NMW+2cVThwUrCM-!B7`zK9`si^m~-p<~MWCCPwLet{wKDvNF^ zS|MmC(1_|l7<8x?-eZPnL`O!q4M3n&(vO!g`NCOA|cz#e3?ejzH8=bMI}P z33b+jHv{#8O7yg10l@1LN$)_e<~fhaqfuUYvU7f?Z(o?eRE-_Jw03~e10*NPemf-x zV3E8MTOI$?6PU>f3+CTF3q-*UX6OUMXdx9+`CBE=tGpYwB3!>C7trJv_(30ZJ~+SB z$&@6L&Kpqu;P?{#3`y^sD^HWuMG1;32%V%1sHLr+Aa}efxSjj$4G4}Eg)${~V|kcn+#gg*Pf?@7i2N0^gA zDi57hiiqQ!fld59;HsI2Ew#r>{H@jb7kLM7!q&rS+o8E+G!PA1-1SPBqO|ngsy<#C4wPFEY%@aekNxu zIPh7~-^!&%f1je;WqpkM!Qh%wB5CA0qL*3-51~1P);G+OhQNRE14$?HZ(1WGk;B%n zO}E>P^VJ(Gu21Yat$uBe!>v!>%81dxkRNH3pHqwx7{y#Vca`{<>P1mcL2WJC_RY;C{CXu3U{oP@aK=|^OEafHV5%BsS>zpG>c~+a&^$;Lao4A6 z#6*T3*tbx^PJt6d)9JtmDl(ve`$H)|Qu+c>$s8f)(;68q?(F$X+vXMQ3fxTcg z<8GVV-}ga$l#Ort_fl(Z_|VoQ0W=*Y`eP>&^z*dzDK*Z-KGR>H&%j)gAPJMSA7?Mj z@zR^|!((`J__FLtC(51o(haVTyVvS=a=)={cI<&(+tE=#rn5h4dCZCs?mIB$SSDz? z@fG3L8a`0;8oW-$?~DQrM-ummbow;(^qX#xGeAu({6v)u@*HjIW_#Fz=S;A z?-O{y$FHRp784xqO#($=Bst@J+COYhfA`t7sRx{9*zMI1wvFq39oa9t^!)_WPyOVV z7$aG~FNTal0_t}c)CDg7ut5`B!5z5Br@8#Aydd=G)Ya2NvVQijnapU$T24bwkEQ1G zJx1+v%X{%rgShMjNOVr&$S(-;#9Ss5C#mL1l$k$;LODD9Z;jBVThx{|%?GP4Dm(*G z1bUaUoPol8SwsBsPzQ$hH-YNMxh}0XvaMf6{@QcPS28QoJiBeViUholPdci=r_uHj z4oIFaNI=&k4>c7XdGD(H4PDibN#Iq?g9HIFs(~I>+3j(eS5i@4rtWRKz`mV>${^8& zp)(N63G?kJ<)M3U*T^sP0kNp6mOkyMwJbyaF)*KZteKjXtcYbJ>H;(MYCm@-^ehSL zP#F3PR*Rdw9P@^N?FR4wBQQyD<-tQkWIPdX%|@f4HljO;lp5MBwj{I(!*?g8#YYhP zf(bK?w|q;|fsf(d`CypX*s7DiVf5ida}GfSvGCAA#f|Xdl_SvS)HIs}VbBRU8l>YI zTue&3e4dGuJ~JZwf@36v-=V=Jih7awB@vun&O-||PGnhRSG&o0lH@C)ai6{x@V!#m zCYu1eheBnf%gm8s7AZPtzKuL@UxDoJEL}8#I6=i#`r+9m7j8Vbl*>M4r&AZx`TJE- zcr6@K2sw1B_<VGm_ea!uWT++)X$ff4F6o z_wzJ=6EORnra=kPhX1m+)$#*p{hniobj+I5v5Ao<~a%;cx z@-n#JX!4#CCQ2v6k{)2o#nYf7$-9GujgAKSH@|9Z!Dk3h7@yxbQ_(z`)1aCn)pZ%< zX{a$!|6W|iZ|ZaUFn8>n=YJ~>r-nDV2OwL%o!2!vx-=g1U7`%)v%C^c$grq$gpv(I zOy3*d$9Hq*+FD&UI&(}yjjA3M@vRLf!ege`p*HZdWTC&m054(}(L86|pL4vpEJ@ z-o|J`n8C=}*vT3GQ1B_s;>S<>ocsP8k9=3|o`T>H zwT=QJ{yQ(chG$u%EjacYcBmtB}Zaaw~P6x--fnwT`@4umo{6 zJn^oa#M;+Zb_UDNQ`>St1rQuysdn`#+_PF8*K@(SMkhs^xQazpH2^f2a0ZWGl_3F^ z8rK6)WsL#WFdY(N)>bYn``?sHIsvX6>KHpBh$sDyK6VmQIIk`rG!2KS7FBdK;C>78 z6}$3{n(9wL(?m!Z4}%zC$_DDl#gLIbIW7Vu|0$_5^hM9XaN5>n%9lCw-%qp}z2M?{rdZ}HGM%-mo$)}A>rBXAP*{~C%AeKxZ=d)UMP zFi<-;uytpx$t8Uw6euGk5*>5>&v? zh<8P?Y|)BYfV*F;wXMV}Aw*a{p*v-1{W;iK8nv<}sB5w)2Fsb;(;K;i4PF`3{01 zw{doUg-h>Uc{yMC1+f8@r@;$|ZQ?cj;dllV64HfWB-lZoiy_Bxss0%uKdanxJ{ z?G3}bq8;o3bXA81lP#B;xjN%Wb^&fFG)bkiR5t0#Sows8f=8qv#_dN@#W0><*D=H5 zaIvL&qpWN(zX~~e>NU??qM7Um#n9z2GQ@TZgNh@89695Yd0f-MO_ZFViL zskJ}-rIcG0t)eLfm?3Rj>UK1D8hC|M$&8a=9i4yc-)@KvY7G-}f%EeW_xLqc)=UGY zDBpIkDC{2Y$V-Lin^0l#CBd3(O!Gx)YFA;rBAGC$x1!4+p@%a;KYlJEKn;)*S31`F zt7%4=Q&a0v-27q@hud2#U0s7UoIAoE46vMqNPs1kUH+F-lphrRZ?5=D7Dd#sH|zO{ zm!u+2B(UB{r&f1xxtX()0fJ_N4?MU$EpJKlY}J?X&!(F+=f!`7?bHtc7#{#5*PY}hpb)-m{JVm7p0 zlIL7Jqj+)eb)y&2@&qd(#q-@Pd(6{5Rx#Ez2WOF1eW$nS^2<4hG&)N+uQnBZ-srRb z?c`+*lP11u*sFWn+tksrtn@MoGKmbnkHN?=CT#W8?%)q#;z)E|42^M zJ_t9UWQ}&)*)s3?2hvu87acIWoh!T!Qjo@$MwS!SeqY>i$yaE+$7ha$hdwXeL*4z^ z>tMF2Q$>ZsIMij0utuP!czb}ID;aIhjA{4gka7rvPm-}g(we{{RD2`L{z!E`P~jj_ z>;Gb(E#N0Ft_RT~9!%+iW(!gS#6a=JyZ>yJ;BFokBPjtLnn=^K2Wo!8ALXHwP3@QL zmqS-ZhXlXGNlC_`;;Fw25vny6h--UJR}JAVx5dAQK(U700Tj z_-!DZf4l9GJ|e>S+@u@-nR>DXx)p!CK-wA>zZ$6P0~W`5+wuLeY6R7%VFu+>aW|%E zb=Vm$fnH3130$-^T(W!-0BXIk+&Rx)_x{YvIe}e*J=SyCy|>$#^;%8OMaCkR)}sc9 z!Z=1YNBGgd(_>*k&?19- z#GUm9syiJs$!SYyvq+FGTOPbii<-enN)LbXx;j=hv~G$>>TKUuTZ<8g^4djlC+gIw zdenopr!5Uzws!YL@V|lQ<0JDs+RWQ*Doc9X8U}5v*J3nqhD)_ibPGl^!L%3_N@V2K9 z+C!Rgl)faIkoB5b^k2&uUSLS)5{pfSIdjcK#&_U*I^21Z*A>&?IsxQpsb&?_8fmd? ztHgVB%qU;iW-}3by1FWgHUS#kBHxMie*`MBIN=TTv^eoU9Ql2!WLW_LRlf$5guJ&W z2&=NK@N$8E9zaD`4thz>D!TFOxZ&Ja@gZJ%G5phJLC;kGtMAVbY6dk%25YeCCYz`= ze1}1hbp!@-Aw~xl@z3Z#*O!?tjl>t(PC%^UfPerqXV~ZjS%}{Dw=fxPd{tV(e6C$k zW((`mx5AN$FU_wGQTGA)7X4{jR;1?DL2Dlx1uW|9$Bz*7d3p2b9qgSr9dnI*&S$8Ip;;K*Lc(8!*-h4z5erg^bw60%Ri+X#5Wh`gJW zhP7=_*mv3d`aFvyAnYXh11m$z%R`d)YH&mLa%ot^2vv{RxL(Lj9()_BZ}^j*{xRz^ zzxO9^s>bQ_(4$u8D=g&vaNzYoasK51A`yB58@k!7*>nFB<&dwrUz6KzhAYc@O&qeF z>Dnmo)7`d(ghdw_<5%nvF)rYG`@<7 zCWV+@Eka%dgyY>V#=o-HTDb@hxXRDIREfh*-HEi20>T{PJ~+^jgk|$j{b|uvNGg$q zLDEYH3c?3x6_;_yTTMBs&+Z4$ge0`8IyY`~qoB2LmK^{EE~4+PL=MCa(^<>#n9G}T zC>+yeP+TgRaC{^BP8WZY+-+zZ!t}?A?9xSt2Wv<8jNPqdqx4-HVrD-|)5Ot?PF8TE zahMJ!D`Uf+slYoi6EEXyMyyd$Hdgyt;p@ZJ)J}n&7K-o~Nrqv_;5%gs%wB4I?BEx< z=qe&XX8UtA>T>!qLAb)VM{JSs3qL-E-}Okf#VNIB7Ck_w$XfNz4POB4Vww-Qv|BE`7YuY zQ2O#bWg$}m*Q;iFRGr!o+c0B>k42wi<@nGF_1IBcyYuTlySP<1uIJO9&CRu8p0H(& z=e?xRJ^nhuQ&0mqu)bxTvNuw>msA}{>E+HLOtNQsy>b6jFO@y|@dcgaHbEZ_f0EP) zkQ9AFTZQ(;^yM&JKl6B({zd`o2AX4p+y;X+a*(BvcxexMA)Ew$o?GJ&S=U!LsQ65=TOQd`ocpRjP~OJb`i)ZbKpscVoOX=OG2rE4MnriN zv8LUdir*Dqw-W;+aJ6*XL>GD7th`w%6on%lt*LM&yI$S+mX$6RmhqL$H5DEeMY zCyR{a7+ga>3$A)~xs;>#PUYFt*0;u%eN%rk7qXT_`4H#~bei%fry<0H(ITieyxAkm zkd9`{ZJX7MRpZn7AfjQpY+z~mx@=ul`(Dy`pg4G!>L~lc;{M&f62G@oenatLr%b`~ zcx(pY%g3bmm2wiHql-PL%w#jhulYryh4}~9^-YZo6^%ZPy8uRi{?&!w?6}5}i@&TK zFK_VTc0(wZHb=P~Ttz*@TPtkjf8YCc*-1mo-4Z;WZL)LLdsGA~vqMQXBGmAO>0LG6 zj099Z`cT`35aTNJdlt@)5Y{}wB`3gX<>RTFt9c!)6=xgJfZ-MAK14&Pp2{VT*Yz5E z?S7E0+XA49=F>t5^UD)pYWs(?ixwg&9c@?*8yw_83hpXs`0iz_LzqXpMCkz5x2Fal zRy(?v6O0hlx2?`W=Kq(e*F)u_PGMZ zG!#Kc;Ja`~%x2_Fwo68_B0a=A!qs~u)=Z5IT;e$%_7Z+=%VjOIPLi*zg|?@JJ)=+_kABZGgZV%0|7I*jQy~*3c??< z0MG)RHp@#bvIE4q{vrjw;r%OPt@jmajoPe>t2#5kVosl81Wk55Yh#MSg{KcWYC82B zVFokVZ*?{v`gsIBNVUgejEpnjS$K&i?vz(~viBxgvHc?-Rz*(DhUh-%(n`<)85s7( zCE`pxOf8gKuDbwbHQ9n8XVd?E;iiT*LZMwScW6Wy6%*GP%iELJQ+}nDi)Ds&I@hx+ zk*_HXh6ZVBK0wIo`t``6QY6-A>sfjYMpl-ZWVkGdISeQ4A}i>HqC-=>M!0O^zaI3N z{uSe+NVYI@>cfF{W1+T4W6j-X*)W~2w#qKD8Md;C=G|`tGk0Ume2$m41mgrh4GKn1 z1%68*{3=XncZCxg_z1;G-8}h|i-4br<*)vaPJUh! zjf-cC-|27uI(TM3w?F%!lpmc(f~+@W8NVw2SEU6%Iv~&m+8;1r?De6*RNFz|2l|RW zLFyJ`D{5+u>5y$(%Y-Yy>H8u?@+}7FmjaYV=SE4Ix%FLV5A%D9H@BDIa!196;NqX5wNLL&tK+LS)#%?!x!K1H9FYXR8Q{r_|_63pL4P zgE);gGC)AjeR67XeokfF_2&yNz$iRqx&nG{dTK@2@&E~^;n}cLTs7=7^xu+R@^r^~ zlRU9SmE~5fs2a&1bJ)WZU~q+=yb4*u9AXdPKn<0LANzVXlIQsNh z{6TPjabrogK%CKB*;^MTg)FlIHi;%04snjKHEtZ?D1op-nLTn(#3l$W$!6WHvG5pp z$)Nl38<=p5N#WxFxMCvVl$&81oUlc-RCQYGb8PMkj6RGDU;Gk784`a=D_%{&tJDKt zY{5Q6Wk=OC(ZY=)(77Mvw?qZVeMd~&Mgo1u-gMc+Jc!Z9ucMex6JxEn#<8=Oh#Xw3 z;6RddFE59OW4$XDH=#dW(4LE?X>=Lr<;4#Y53`?gd^0~IMNh|BM001UG`|S^RNQXI zJJiGv2S?PzZq|{t?;8t0;2nEnpqbks`ju#7zo`+m-~`Cc69}WA)%=-uDA2-Kka?Jw z{|OFXplt(No7Iq>^=n57bH3(o_DV9ew?~kg0Ee^XMz$inwQ~D0J8+Rks}d$)y!(4O z@t!-YHQg<%RMEy3MTi?qZrW%YBR7r{)dFGRdd=cE?)#Sza1&=3m{mAHH&4JP3Q zYrp|G{wBhb8IZ9^APjH0zG3y}IE?J2)7kKB&6ZMQQaJDz;hR-W7YM5iPyA6(vm!js zVLYF?&gPkgpS?|yRPo8TI}ex;iB!ij`MMPpffPwQ6o>^O`ym!*@@k?q(yP+c_w;ae z8K9_$h49|STAdusUSEa-CrmzaK&<2yryOfmRx0bm%TSO^II<9NQ5_X~f_q0TkJd#Hz6Fki8?9lQ@s(C1`wvj~GM<2v<$u*(voX+QXLb~e% z+mz6o0%eZb5L@M6gsxW!&h$mmba1c13C0*FP~}WduIc+Af3$Zgu%*S*pHFS(7MANZ z0oDRwH(w$1;2UQ$AqDnPKt2k+rZyUfEsQ*bw=XMk2Uv#kSsAqrF2Z;yG)Qmuw0L*5 zeAA3BgXKpMigE$NDlhp5U!}>>nCg=O z4gA+{9j@`FV|AXrB~CQ6ui+IucvIMlPt<@?Ysj{77)kh7@$Oy*NYPH+Jaq)p{_j?Z z`*=C|xAIff1s5yp9>hXBGSl*z#T5J19HS0-osM_7{ttwAXYD`#e11Rv`N7TcK_$pC zmJ@Y)La2jJSJKraXhv{jQP58}C{XF*Ugf%5Vh(yN+8I!z4d-q>KS1l>vz%30CwI*} zwhxqZh*d2d_rgel+t?7(V@b0K>FO5_5qA}?h@k2HHK(S#&#$W+^$flu#*zI&VIdnx;5UV8+v7``w# z+taSQ(8u9EDSu>E7ix{_yt!+CqCzS#7Xo;%*w)&pXl4B#nAAA*x(rMk)`BnD9_~<; z$C3(R^5GBZ)UR>8@Fr-)XJBibI?iE~gNFXh zhj+UD)!Wm?!3!fRzXlu5t7C9l&??w@iAebHt+&F|PjNWM4%RG4iA+!2h)=TA=nMLIV8Mb&{Kb}iWe|_{msAusQ z#1lAerxm5$FV5Oh2ZV1#wGcmyYeIgq>Fz zbhw?8&0*;nb$jJO9gf(DoKC)(bR(UhgW20JkOT*`Ydh4wDKZEWp(UVu(Jb-fsgvBG z=TRVIu}ad2Zj&{WH2b^E!y5q@DO|1|6wy@%0oMX2)gD^UKYx$zPoxUJYdu<8xxu@r zwdgRP1R9ZE;oW)vI*iXzc0Qk+Z=urJzd!uGNgFF9{Q6<|XK_d2vC!CjpqCDN>21=i zSL&khzeGDEa^?5@%bMx8@F44g{kLE^g#z@33=r1T zAM>jrjDhP}bdKNypZ0uruT8OtO6C{%#$u&crWefjnw%o%lwDjInK*IcZ_PFb+S%pk zbM$!5-q|p4b>K`}c-%g=u>SHkX}(SJmi0xGm;A|k;Y~dK&Gf**b-w(OG*R_6VYMrJ zeH-y&$EM^>JWX5E6vG6Lz5B84r>wR7=zB|VIc#Nb5QGy|MVv8;I(VS+*=NybOP2!n zcba>^-L&-t48G<=y9q89#RZ8b?1bF4&T|_IB1CBKl@ojon})#6sA=fv!7+AH+*$%Q zwWh2wXp#Tlv}O+QtJ*7~g=_{XHwo}O<>@igxji}UX$xPy$+j>eF7KrVUCWZ*-Oz?5Rej=mED-43I z&XPZ}GLS|>)do*ppWrw65Ln3driX0?xrK<@9w`V#&YYg=*E2%Ix=D~>xCq%cX+CD9 zYpT}nDa!^V;Rpc4=t7RaO^-dKSMi0wFLCI>0S5x-FwZIU^Z?#mzR{}_&R6o6O z0Ty*^aV~nli_y*bpO^i%8pk`ksJpnle!ZssX)$r~k=G$n*JE_@sV zlj{Br67VAhITSA6^jh7d!c2VoRdxLu>z{t_KSHNVRerT?_G?(0M0iUT|K2)ieq0S6 zw6hpBqF}3}qWRlBW^%z}{no^TxguFGz)|8G3 z7IzsY%E09RuMP%p{2o>S4*UwupFPtt5qzq1-5T3Lh7oM!^>KQDgg8Vv4sn%3`lhi~ z(v^kpVW#2Z9YH7Cx+bn~XrlF{(B+OLn`lLtA!(SjHrO!O3y5@m6Bp71mQ>NM>TuMH zXy8jU1f;M|CI0{90C>9(btb!4{3rCWXS)}I$p4eW$wu^~SU{2!G0fY(oyLG2z zKlyf2`p(QA4}&itXA^?%ES ztcT2#LsriJj^^Of6zc$)>d!y>@0epsNa?lTr7?1=RfW$tW@(l%kkT6t}N*P<+DqBs{1 z!m-_9ZWQa^Y`QcFE>OJS6cqb4%ek_Yx{fJjkHU`SwV%0r zanLJ<;vIwBRH$9`hD-*Ov-O{u7=T`vkr8vAp$$=V$xq-}X$DRu6x9nAOHK+p0r7ro zF0z=1NuR6nJ&C?951+3tSuc=MP7vFuPoqQ^E$6>r=T`mnIF?OuCz~{ zB?uj0l;7#V1&p}s-SYq979OQ9)MRyALqCbLi^b_2}A8w*Q$ajV|xf@gaS@Q=G-jr4yK*8 z2E$fKX#JHFm@vR^6Yyadq*30-{7o$dpF%bL#ER|bNc6aie9hm41;bASVf{N@!!u;E zJ$%Ze6>jH{<~sb7v-B;da*VY4SWuWK{;OCiZwPS^yZ%MjqttQhDC*b|PRe`o%2>Pl z3hS>zH#e<6yX^0Cy;!-piDDuQ#Ar)I-ilv_1Cx_65FLeWo#_?u=`$+3*77PA%S%Oa z-_;*P*;$*m6%{GJcOHCLzApdE-2W1hkCZ_;7W(uz>hj2w54S^*u!oN_F=9!VlnB== z@;#I5_I^bVl*dQzZ_3Gnhj=h!xz(CdIxId{T%v4sWXBl$IlNwvW6%04%>cvIQ9BpA zB;QXLR4rD|@!H^2>h2gvm#V9OFD&J6{qzTZvgm&=bpz;XifWh$s+tHkl`B#-nm7rW zxAqg`#PkS0%l+Cl@@1%4xAZzOU_sm;DzHnIex1WigBFL9Tp2m&doxQG2{-ii{ z$y2@}Rd^p-6Y)wX49U01yWsrJJv)A^%V&>ZWhZ)m4mpu1aAffC&h~4P9cQW?*9#S` zPlTFX)3wjo$b2T)__A50*YV3RjA!#_o~?zI+1<8Tc#VColC00zzhrQcS-f6d@B8hE z=g1v+N%vplKo-?4HkaKpxF-FX$G>v=8WK;N2-KvAx=)D7nyK|uX+OW{?ehdu%f1hxNl0R1)e1y`%Qse187I-d*tNLkHVO~0FU>E}Cl zdt{pUH0wwNSbSG!{rA@ddA4C#$o-6LTypx__Y*~3ls`j@``3Sj>m-CbPN4_LT#ezZ zBquATkNUJ)Poal446=>C37E>Ov|5<0FSqepc-n2%woCd?#D{mB1Z4?OTCg>GjnwPF z_wRA3$<=UYj0018>-AT5d=XJH{3ko8Z3HrH(k!qcspATEVf>bao|NEo8j$pm>i=o( zEdQeF;&u-M(hNh_07EDUNJ$LM07^(Yq|zZRAR%2tcM8&s(v2Vu(ug9BLkZG?bclqU zjrV;&=lloHd9mNjXJ+ru{_VBaUTa;~^_@#)WQT%rvnFy1&AMYf>mlM8Dz8;0+E8xq z^HmU1)VY{*M1u86 zKdy%hk9=+FN7j@m!_a}nKc^0Pnta9}9)9Dq^@+4oejNu-yK=DI$#6codl6VrwE32{ zx_a@@WR-vloGa!ZDaLJ7TT?|^o@x?%d7^!A>)!j^yQ~adCjm$`xQYx=D8qh7+8KFy#KnyCi(GK+*33Ss$jjE-jV?O+imdv<%={alaQvhDwICEindNHaHbuSzz7 zvM@!FFnDv+;ycD-uCdsdKx0u2a&`LHz{~HA#}wWjdAnY!sEju`A3rH)WwxX-#^=E` zpHK`Oh%?8BUQ%utZ^-INpA;6osD8f{q_poU) zvq0edU{_EG_bYGz{i`{aQv&O~ULWpMbC$5l6U7WRpb=i$qV_ji_b%5)4dA`)LzvLV1bs_?`zs~^+#*3QnuLu?6&%ZYn8=Ql3ezdp|5A5+oijeLs>Wx=3{bt5ar zyLDvu>Y>IRr{kEkT3a|ol@`4+wpp-X6I!Qkr^Ho@C$gSKI!zt-iaKtcI$LqinXW7_&TDvE6oq_btNwx14&40E|FYqDFOdGLN zu0>WkX4lYG@|~8;x%^p|aKJ#stA1hfa#6C9>CwB#;p)gH*uZ5K1XY{gYf(LLcI6`I9cg)o@L5TW2`yB@k@R3RN1@^%D=*% z7uG6~PY_^DvTs|#0eS9bvcB#HWT$8-BEdAosIc|-(Wd?@aIuG4N)m#&%WfyXAFI2f zfb7_XLZEMRY1L14!DlmOCR0ItVCPvnhdb{q-t60=YS3eCFM2+WICprBi8+NLg-U#n zdrwhwho63xl#|(!L`>n$dc4c>h%>OPWCH~+AS|ur`toK1d3FSurl+h zIb1Xk<2LhxnMo~nH>G(W-lq|o?jS?=V<4 z;${m*W?Fy{cJh1OVSbi@{!*Md|M)B?#7h>DTwcVL`-EH`R8KT$pTz&zl5iK9YnYlF z?)-siIXdin1VP~*Fy-QwxX)I&COeV-My2>>#b5g__{!6lpzw}{;@0cb0iWPHHK_*= zgM)d2W6y=Nh-ovF2A?`xbVd(G#UwM|X^3prH}=>gM}LwVainJej;y`UXh&?CjOCww+_kUDv~yj?C$nu|Va)vwKTQ(h(`Y`i z3_C5UNiiFvIilkWDqVcxbD9Qt>=*aQXuH`T<1p}8m7nx7WsvHBHj9<*J~>)1chQ6| zyHnG??!E0TKKj86h3~}cLlhjCrf0&1r?HK9a&MjQ5Oa5Om0=z?t*3+>l0I0IG!583 zEOZSFdaLYb(Zm`t&PC6L(?w5gZv~TJejN%g{GGm?mAZBT)6o7grx+ImiBsYdp({8Z zlozgUksKJ!a@hJLLMm`#@$dMYhaw_S#D0aPUCpj5xkJ%ii@SxU3_=LB=)L)9~$_H~%T4a%4ww(WD zv4{=zTH{kP2RT76b^miM{AQnCo51J8a#!26zxx2w+f9>{z$a~Dl_zJF06}!;EUId~ zCpSSYkDN}p!cuOnD})s-goPKOx0u2JUyq*O6N5hmyQP7zWZiaCdpO0O_-5|LD4s7~ zHmPXWY8plXGC3xqS(%EGS0*qimOqC@PS>W6*QV7r{>!9Pqvj!HwoWFM3+1&q zaA|IyCb4x!+X}Uy!B9CFp`C!%UmD1~~Ci{T+LBWhMUQSb>FO=3b`{o;*x##s z(10JKhEF^F^QqbVoAPf}<@S$t1K#={=7bcV%g%^1#~Q^a>+E`M@e+Enkts=D*VyR? zOqn#~_TIIp4CT(@rd&pfDm}uL&t=Vh(;ha`0Y&clvCAE|{HNK`SD~=b_8@JtZp~Qa zhucf;QL@-V3q&7^m%h-NeSbUa*HrS}8!_A^KheQ9VUiON@xDw2*{qBFqdB!A8E`OA zt<-TCqu}}5RykR#P<`DnBFx!8D`wdDsQgK4xM&(VE0Iva(>qF!*i+#$ZgjzkIHu~- zM7A}iqzby%E;HbzCCe7IkB;UvHi{2^WKoYw~V@TP?FN$COTGeIN(rH9w%NLER z>ON9G=ZOSiqo^2w_}s|EuiX43HP)?B|o zax$d?_WIFC+tTF8$2@Ty@~^A+)FA}zM{$q>=<8zqy+AL9n}q$%-i+7(mR;s=twT}c z?s7#!NIlDaNhUYQFcYw7>hX;1zQqK}f3d2RGO4Et{9C`plop zpVQuwx5=&L36xO$s5K;#BOXN_UCYl&Q=qly-PCezE`1hsA`)u*&x{YSCh*|n&3l6w zqXaalu)EB`Z1!&1iVRpR4vH6SKH^XU}yjT-E;9K{(Ryg)_tVELTo`fMOe%vocecv?(Y zc>a_ag(jEmwQ)oK zaBe^kY#l`4;NaB#l>3S!h`j(aY}@Z9-lu##t{HGhX>jT#bpy;A&IIIc&zc1zX=-C^^vA2s8XMFQ_bHt4|{ zj8Q$R%l)2-c*~CHoZsQ`Qte!nvC`f}-`X;t<~sa*ttp4OZ(6~cX|oxwWi5lSlU+Ju zeeC9b+}vr>>PdPtHEPR)hPI(hgA>f?MH8iOxlmr75Ajqlea(w}XjM9*|M+nT*}aAI z7m!a5HT=xj8WAldBLsP8l)irRf_*10zAoaMTXEZqfN>#UXU_Dh^Mw7H8~(rISJCFp z-_9uti2LmrV0=owFS8AMOCg|o+e{8SiW`rL^10vs5@J4_8Jmg{N+~Kgq#S|L3jD4b zOL+iV(rTgzL7Xbi3x%VqQW}564W1jXT#rbMB-3UQljM|TimE%c_EWh<3MtA~M0>Xa zR%)--A=p8W>KJuC=-7#<$D|=Z32~C%GQ{67wKON9J2$1fKZEE6>eJ}^tGc?LUfh6rX$<`R-4jqSSnrAcIl`b zMZtEGA?BR{FU=~y)hiUb3shlaNOGzgfikmV{wks0?6vCR{}MDj_X|N zVD#r(5ryJ+ZuoZqm`^1WR5(b<$^=ti>Q9!7U5~XTdL8b3cdM@i4$AFL1F*2ZIW>}G zy*FoW2+zRKT^-~I66I8mukDyoK% z%!O2d_`SB8Q4a-VuAYHWVLY#fl71n!kpw6r(Vw}+AXLYm=i%H{5T^5z_K0em@i`wWWN4(T~K!Dje45myfO7>5G~HZ33^d&gT5 zKJOsvu1^=|KOo4YkGFkY_g4lI+yJH`G1Of@cEG~z-faND3~28^ibDUanV#h=-4ucB z^V#jALi#I6r}xHq8100IZ6G)+QM5;aEn z!GLUsHzO0ZiC6H?EP;D;e`(IOnn6aY1mclk_h3h%rwli(47jP+>Yldxg!sY1ROl2! z5Pg?EW{Fd_`GH^!mZ{;GqWI<#qAsYt+E1n~s5jz@#SG_C_r*h?^t+kZ*ujeg8vAos zSeZXZtD>N|xS>5>tkz4bdo|d+J^fA6btJ~Xr;0d|AnES{p*5s_Ctia-R$Njr3MdOR z%57A76&6sDB--%fLOUb@r z8~y{-zD?(eYwoaVcja$kubD z<(JVX3UGAW{@puoyEqrsB-GcYx{hiP$AMy9FEHnPKb9CGyHDLsT>`rTyPw(R(t47m zbs|QH^A>lc`g6nPP?jHH%EA~5iq>})Ec%WTz2n#wy)<(eY$vlIzrxRbo@%I*WEop! z1^4$lm(lo24Ct>o7ZwVtHy+BaybhvX*Qr3%suvf!$$`6RZo!))jov_L6NP^vnsgBHb=)(T@eNp}D0($a2w%xCh2ZtE2V^7PaS5h+JNX35IhAn`#w9Tx)fXg` zXv+1Ymjd_TB=xzRU*S4z#vze(v1vifa^l0g=sq*0Aram_tSDtFiw8gU>tsGpr$^q_~nrH$Tt6IJ8;DXUz`|#s~?3hgB@U8mgSoT;hIt zq0WZi#kn&e`xG2vJ=>i3lx7{Jg;J*A`_XGLnTWDwnsM5OeH3v4pJNk7s*jpcy?VAF z^)x3BF2PWpl{W!(M^@0{*Z>KP7_m8A^Xf!wOrU;|@Enk4n zujeFDjGQ5`=wz}a2q?)Boc$5o4!QJ@ARA7JMG;8}{hXeAcCM2UM|!$Dd(p}3Aho;u zLvSvkEVpV)X!B##gQ$UGZ10fs)AOCA?F<9*4t_6qASHT3G_ZWQ`U}qsytw_z1wD7V z+bIa>9M!B9M0AWmx_l^KDFts|_x>iw)!F`!Y7&Y3y_b8;K?ujq79r(H3~gILGSk~5 zD@7jEP=3tj#=wY0^!32A$sZ?cYfPU?nf|EO5Rh2p922B|HMlz8QZI6{S`a<37Ud}^ zPsSZSAMfdYAy+{gQ*54W$4qF#uA7C7B#GbjoD8H1+s;Yh9--%D2kDJ_38F63aBSnKwwrXCs!~@kd6_%)@G( zhr07(J*kY^_sY)XtdAWwDLSgkMw`G7+p0)=9AD3cBDNTFKDSEFJBEMpfJu)jQ{=dxl;t zsKcO0Q4z^QzA^J(94cLLtJt+o-J~}2XUbpNf1^08C*!{q^Y_30bcL+)OpIiAf&12f zi_1+zyw5UwMYOU!^OZ6v^Hl{&3l)k2dd`wH)@^X5i8q5?kkcD2GhR14Mtw3xhb|Vf z{Uw<}6CYcYO}&JLswr@p&sPz$WkDn(G#WuC%;^ZTL2s#ErpvpM7B~u0@TYDs&sHzO zInog}>1ud!r2T}uX$3t-D$j(*l@1%93q^9nO@+fAp)GU9^tqi))i^zj6AZ@b;&Np5 z@Jg|07px$5Y8jR<(8?`N&XH=E3%a-c10?B;`uCoNOm<_Nl8g|?kdS@(p`|CJr%PXd zUsX!w=jTxdJGFFyuvojL?$7VyZAQ5BExgC05Ok-xT}*EEpP_N_YdZDCpYK=~p!}XMejHTL_huwLCm9HR_y#jvDwAHAiHK zG-_t{Pkw(gVpA)xi??tGWz}3kc*$box$H6%@1C>T;UW2^jHCs$HgTKGnfZg$?%Pu? z&qStLOC!$Y$GPCI5pLna@kR_| z=L6gO6Hp~@aiP#&CREa5h*#UdS~LV3V`yjftRMmDTHab|DK%ey_!Xp}!l(DF_^#q9 zngL<0<+0+KsB{roMRzeg(z!0{Ut93mO;k#RHJrFsyFH2>Pn>VDZ__{*Pgm><49VW3 zg^z;vBaG(|`BrfVxj2LpQLr(>8T-JP0~cbjjjhw_C=eE3K}JnU2fXBK5myJBRy zcy~X;{MwJ3$9acxKft(?v3l{I0B39*fj#uqAQYUb(64)Do%qeVITjB==qC4B#ZAqi zqK{IzqkBv6q9(z8Ui=4eOLwiU+VC$U{WVk1X8dI884|uJE%=F&6+_izXI-^An1q?U zx(7=S9fR_JostXU`FF&wU9%v|5$(x1zA9q^4 AGXMYp literal 0 HcmV?d00001 diff --git a/_sources/config_examples.md b/_sources/config_examples.md index 5d3b5bc..8ffc4bc 100644 --- a/_sources/config_examples.md +++ b/_sources/config_examples.md @@ -46,6 +46,7 @@ Our internal configurations for each pipeline are as follows: | EBS_VOL_SIZE (if using S3 mounted as a file system) | 22 | 22 | 22 | 22 | 22 | Files are read directly off of S3, mounted as a file system when `DOWNLOAD_FILES = False`. | | EBS_VOL_SIZE (if downloading files) | 22 | 200 | 22 | 22 | 40 | Files are downloaded to the EBS volume when `DOWNLOAD_FILES = True`. | | DOWNLOAD_FILES | 'False' | 'False' | 'False' | 'False' | 'False' | | +| ASSIGN_IP | 'False' | 'False' | 'False' | 'False' | 'False' | | | DOCKER_CORES | 4 | 4 | 4 | 4 | 3 | If using c class machines and large images (2k + pixels) then you might need to reduce this number. | | CPU_SHARES | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | We never change this. | | MEMORY | 7500 | 7500 | 7500 | 7500 | 7500 | This must match your machine type. m class use 15000, c class use 7500. | diff --git a/_sources/costs.md b/_sources/costs.md index 1673ab8..de7645d 100644 --- a/_sources/costs.md +++ b/_sources/costs.md @@ -2,18 +2,28 @@ Distributed-CellProfiler is run by a series of three commands, only one of which incurs costs at typical scale of usage: -[`setup`](step_1_configuration.md) creates a queue in SQS and a cluster, service, and task definition in ECS. -ECS is entirely free. +[`setup`](step_1_configuration.md) creates a queue in SQS and a cluster, service, and task definition in ECS. +ECS is entirely free. SQS queues are free to create and use up to 1 million requests/month. [`submitJobs`](step_2_submit_jobs.md) places messages in the SQS queue which is free (under 1 million requests/month). -[`startCluster`](step_3_start_cluster.md) is the only command that incurs costs with initiation of your spot fleet request, creating machine alarms, and optionally creating a run dashboard. +[`startCluster`](step_3_start_cluster.md) is the only command that incurs costs with initiation of your spot fleet request, creating machine alarms, and optionally creating a run dashboard. -The spot fleet is the major cost of running Distributed-CellProfiler, exact pricing of which depends on the number of machines, type of machines, and duration of use. +The spot fleet is the major cost of running Distributed-CellProfiler, exact pricing of which depends on the number of machines, type of machines, and duration of use. Your bid is configured in the [config file](step_1_configuration.md). +Simple spot fleet configurations can be minimized by: + +1) Optimize `MACHINE_TYPE` and `EBS_VOL_SIZE` based on the actual memory and harddrive needs of your run. +2) When possible, mount your S3 bucket using S3FS so that you can set `DOWNLOAD_FILES = 'False'` to not incur file egress costs. +See [Step 1 Configuration](step_1_configuration.md) for more information. +Data egress charges range for various reasons including traversing AWS regions and/or AWS availability zones but are [often $0.08–$0.12 per GB](https://aws.amazon.com/blogs/apn/aws-data-transfer-charges-for-server-and-serverless-architectures/). +3) Set `ASSIGN_IP = 'False'` so that you don't pay for IPv4 addresses per EC2 instance in your spot fleet. +Public IPv4 costs are minimal ([$0.005/IP/hour as of February 1, 2024](https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/)) but there is no need to incur even this minimal cost unless you have a specific need for it. +See [Step 1 Configuration](step_1_configuration.md) for more information. Spot fleet costs can be minimized/stopped in multiple ways: + 1) We encourage the use of [`monitor`](step_4_monitor.md) during your job to help minimize the spot fleet cost as it automatically scales down your spot fleet request as your job queue empties and cancels your spot fleet request when you have no more jobs in the queue. Note that you can also perform a more aggressive downscaling of your fleet by monitor by engaging Cheapest mode (see [`more information here`](step_4_monitor.md)). 2) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to perform the same cleanup (without the automatic scaling). @@ -23,14 +33,16 @@ Note that you can also perform a more aggressive downscaling of your fleet by mo After the spot fleet has started, a Cloudwatch instance alarm is automatically placed on each instance in the fleet. Cloudwatch instance alarms [are currently $0.10/alarm/month](https://aws.amazon.com/cloudwatch/pricing/). Cloudwatch instance alarm costs can be minimized/stopped in multiple ways: + 1) If you run monitor during your job, it will automatically delete Cloudwatch alarms for any instance that is no longer in use once an hour while running and at the end of a run. 2) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to delete Cloudwatch alarms for any instance that is no longer in use. 3) In [AWS Cloudwatch console](https://console.aws.amazon.com/cloudwatch/) you can select unused alarms by going to Alarms => All alarms. Change Any State to Insufficient Data, select all alarms, and then Actions => Delete. 4) We provide a [hygiene script](hygiene.md) that will clean up old alarms for you. -Cloudwatch Dashboards [are currently free](https://aws.amazon.com/cloudwatch/pricing/) for 3 Dashboards with up to 50 metrics per month and are $3 per dashboard per month after that. +Cloudwatch Dashboards [are currently free](https://aws.amazon.com/cloudwatch/pricing/) for 3 Dashboards with up to 50 metrics per month and are $3 per dashboard per month after that. Cloudwatch Dashboard costs can be minimized/prevented in multiple ways: + 1) You can choose not to have Distributed-CellProfiler create a Dashboard by setting `CREATE_DASHBOARD = 'False'` in your [config file](step_1_configuration.md). 2) We encourage the use of [`monitor`](step_4_monitor.md) during your job as if you have set `CLEAN_DASHBOARD = 'True'` in your [config file](step_1_configuration.md) it will automatically delete your Dashboard when your job is done. 3) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to perform the same cleanup (without the automatic scaling). -4) You can manually delete Dashboards in the [Cloudwatch Console]((https://console.aws.amazon.com/cloudwatch/)) by going to Dashboards, selecting your Dashboard, and selecting Delete. \ No newline at end of file +4) You can manually delete Dashboards in the [Cloudwatch Console]((https://console.aws.amazon.com/cloudwatch/)) by going to Dashboards, selecting your Dashboard, and selecting Delete. diff --git a/_sources/step_1_configuration.md b/_sources/step_1_configuration.md index fc2aa65..3dc89a2 100644 --- a/_sources/step_1_configuration.md +++ b/_sources/step_1_configuration.md @@ -17,6 +17,7 @@ It need not be unique, but it should be descriptive enough that you can tell job *** ### AWS GENERAL SETTINGS + These are settings that will allow your instances to be configured correctly and access the resources they need- see [Step 0: Prep](step_0_prep.md) for more information. Bucket configurations allow you to read/write from/to different bucket in different accounts from where you are running DCP. @@ -48,15 +49,23 @@ Distinct clusters for each job are not necessary, but if you're running multiple * **MACHINE_PRICE:** How much you're willing to pay per hour for each machine launched. AWS has a handy [price history tracker](https://console.aws.amazon.com/ec2sp/v1/spot/home) you can use to make a reasonable estimate of how much to bid. If your jobs complete quickly and/or you don't need the data immediately you can reduce your bid accordingly; jobs that may take many hours to finish or that you need results from immediately may justify a higher bid. +See also [AWS on-demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/) to compare the cost savings of using spot fleets. * **EBS_VOL_SIZE:** The size of the temporary hard drive associated with each EC2 instance in GB. The minimum allowed is 22. If you have multiple Dockers running per machine, each Docker will have access to (EBS_VOL_SIZE/TASKS_PER_MACHINE)- 2 GB of space. * **DOWNLOAD_FILES:** Whether or not to download the image files to the EBS volume before processing, as opposed to accessing them all from S3FS. This typically requires a larger EBS volume (depending on the size of your image sets, and how many sets are processed per group), but avoids occasional issues with S3FS that can crop up on longer runs. +By default, DCP uses S3FS to mount the S3 `SOURCE_BUCKET` as a pseudo-file system on each EC2 instance in your spot fleet to avoid file download. +If you are unable to mount the `SOURCE_BUCKET` (perhaps because of a permissions issue) you should proceed with `DOWNLOAD_FILES = 'True'`. +* **ASSIGN_IP:** Whether or not to assign an a public IPv4 address to each instance in the spot fleet. +If set to 'False' will overwrite whatever is in the Fleet file. +If set to 'True' will respect whatever is in the Fleet file. +Distributed-CellProfiler originally defaulted to assign an IP address to each instance so that one could connect to the instance for troubleshooting but that need has been mostly obviated by the level of logging currently in DCP. *** ### DOCKER INSTANCE RUNNING ENVIRONMENT + * **DOCKER_CORES:** How many copies of your script to run in each Docker container. * **CPU_SHARES:** How many CPUs each Docker container may have. * **MEMORY:** How much memory each Docker container may have. @@ -83,8 +92,9 @@ See [Step 0: Prep](step_0_prep.med) for more information. *** - ### MONITORING - * **AUTO_MONITOR:** Whether or not to have Auto-Monitor automatically monitor your jobs. +### MONITORING + +* **AUTO_MONITOR:** Whether or not to have Auto-Monitor automatically monitor your jobs. *** @@ -111,6 +121,7 @@ Useful when trying to detect jobs that may have exported smaller corrupted files *** ### CELLPROFILER SETTINGS + * **ALWAYS CONTINUE:** Whether or not to run CellProfiler with the --always-continue flag, which will keep CellProfiler from crashing if it errors. Use with caution. This can be particularly helpful in jobs where a large number of files are loaded in a single run (such as during illumination correction) so that a corrupted or missing file doesn't prevent the whole job completing. @@ -120,6 +131,7 @@ We suggest using this setting in conjunction with a small number of JOB_RETRIES. *** ### PLUGINS + * **USE_PLUGINS:** Whether or not you will be using external plugins from the CellProfiler-plugins repository. When True, passes the `--plugins-directory` flag to CellProfiler. Defaults to the current v1.0 `CellProfiler-plugins/active_plugins` location for plugins but will revert to the historical location of plugins in the `CellProfiler-plugins` root directory if the `active_plugins` folder is not present. @@ -147,7 +159,7 @@ If you need to use deprecated plugin organization you can access previous commit ### EXAMPLE CONFIGURATIONS -!(Sample_Distributed-CellProfiler_Configuration_1)[images/sample_DCP_config_1.png] +![Sample_Distributed-CellProfiler_Configuration_1](images/sample_DCP_config_1.png) This is an example of one possible configuration. It's a fairly large machine that is able to process 64 jobs at the same time. @@ -159,9 +171,9 @@ The Config settings for this example are: **DOCKER_CORES** = 4 (copies of CellProfiler to run inside a docker) **CPU_SHARES** = 4096 (number of cores for each Docker * 1024) -**MEMORY** = 15000 (MB for each Docker) +**MEMORY** = 15000 (MB for each Docker) -!(Sample_Distributed-CellProfiler_Configuration_2)[images/sample_DCP_config_2.png] +![Sample_Distributed-CellProfiler_Configuration_2](images/sample_DCP_config_2.png) This is an example of another possible configuration. When we run Distributed CellProfiler we tend to prefer running a larger number of smaller machine. @@ -175,4 +187,4 @@ The Config settings for this example are: **DOCKER_CORES** = 4 (copies of CellProfiler to run inside a docker) **CPU_SHARES** = 4096 (number of cores for each Docker * 1024) -**MEMORY** = 15000 (MB for each Docker) +**MEMORY** = 15000 (MB for each Docker) diff --git a/config_examples.html b/config_examples.html index 7ada4af..d30cd70 100644 --- a/config_examples.html +++ b/config_examples.html @@ -594,7 +594,15 @@

config.py configuration examples

DOCKER_CORES

+

ASSIGN_IP

+

‘False’

+

‘False’

+

‘False’

+

‘False’

+

‘False’

+

+ +

DOCKER_CORES

4

4

4

@@ -602,7 +610,7 @@

config.py configuration examples

CPU_SHARES

+

CPU_SHARES

DOCKER_CORES * 1024

DOCKER_CORES * 1024

DOCKER_CORES * 1024

@@ -610,7 +618,7 @@

config.py configuration examples

MEMORY

+

MEMORY

7500

7500

7500

@@ -618,7 +626,7 @@

config.py configuration examples

SECONDS_TO_START

+

SECONDS_TO_START

60

3*60

60

@@ -626,7 +634,7 @@

config.py configuration examples

SQS_QUEUE_NAME

+

SQS_QUEUE_NAME

APP_NAME + ‘Queue’

APP_NAME + ‘Queue’

APP_NAME + ‘Queue’

@@ -634,7 +642,7 @@

config.py configuration examples

SQS_MESSAGE_VISIBILITY

+

SQS_MESSAGE_VISIBILITY

3*60

240*60

15*60

@@ -642,7 +650,7 @@

config.py configuration examples

SQS_DEAD_LETTER_QUEUE

+

SQS_DEAD_LETTER_QUEUE

‘YOURNAME_DEADMESSAGES’

‘YOURNAME_DEADMESSAGES’

‘YOURNAME_DEADMESSAGES’

@@ -650,7 +658,7 @@

config.py configuration examples

JOB_RETRIES

+

JOB_RETRIES

3

3

3

@@ -658,7 +666,7 @@

config.py configuration examples

AUTO_MONITOR

+

AUTO_MONITOR

‘True’

‘True’

‘True’

@@ -666,7 +674,7 @@

config.py configuration examples

CREATE_DASHBOARD

+

CREATE_DASHBOARD

‘True’

‘True’

‘True’

@@ -674,7 +682,7 @@

config.py configuration examples

CLEAN_DASHBOARD

+

CLEAN_DASHBOARD

‘True’

‘True’

‘True’

@@ -682,7 +690,7 @@

config.py configuration examples

CHECK_IF_DONE_BOOL

+

CHECK_IF_DONE_BOOL

‘True’

‘True’

‘True’

@@ -690,7 +698,7 @@

config.py configuration examples

EXPECTED_NUMBER_FILES

+

EXPECTED_NUMBER_FILES

1 (an image)

number channels + 1 (an .npy for each channel and isdone)

3 (Experiment.csv, Image.csv, and isdone)

@@ -698,7 +706,7 @@

config.py configuration examples

MIN_FILE_SIZE_BYTES

+

MIN_FILE_SIZE_BYTES

1

1

1

@@ -706,7 +714,7 @@

config.py configuration examples

NECESSARY_STRING

+

NECESSARY_STRING

‘’

‘’

‘’

@@ -714,7 +722,7 @@

config.py configuration examples

ALWAYS_CONTINUE

+

ALWAYS_CONTINUE

‘False’

‘False’

‘False’

@@ -722,7 +730,7 @@

config.py configuration examples

USE_PLUGINS

+

USE_PLUGINS

‘False’

‘False’

‘False’

@@ -730,7 +738,7 @@

config.py configuration examples

UPDATE_PLUGINS

+

UPDATE_PLUGINS

‘False’

‘False’

‘False’

@@ -738,7 +746,7 @@

config.py configuration examples

PLUGINS_COMMIT

+

PLUGINS_COMMIT

‘’

‘’

‘’

@@ -746,7 +754,7 @@

config.py configuration examples

INSTALL_REQUIREMENTS

+

INSTALL_REQUIREMENTS

‘False’

‘False’

‘False’

@@ -754,7 +762,7 @@

config.py configuration examples

REQUIREMENTS_FILE

+

REQUIREMENTS_FILE

‘’

‘’

‘’

diff --git a/costs.html b/costs.html index 60b584f..e2c5c08 100644 --- a/costs.html +++ b/costs.html @@ -424,7 +424,17 @@

What does Distributed-CellProfiler cost?submitJobs places messages in the SQS queue which is free (under 1 million requests/month).

startCluster is the only command that incurs costs with initiation of your spot fleet request, creating machine alarms, and optionally creating a run dashboard.

The spot fleet is the major cost of running Distributed-CellProfiler, exact pricing of which depends on the number of machines, type of machines, and duration of use. -Your bid is configured in the config file.

+Your bid is configured in the config file. +Simple spot fleet configurations can be minimized by:

+
    +

  1. Optimize MACHINE_TYPE and EBS_VOL_SIZE based on the actual memory and harddrive needs of your run.

    2. +

  2. When possible, mount your S3 bucket using S3FS so that you can set DOWNLOAD_FILES = 'False' to not incur file egress costs. +See Step 1 Configuration for more information. +Data egress charges range for various reasons including traversing AWS regions and/or AWS availability zones but are often $0.08–$0.12 per GB.

    3. +

  3. Set ASSIGN_IP = 'False' so that you don’t pay for IPv4 addresses per EC2 instance in your spot fleet. +Public IPv4 costs are minimal ($0.005/IP/hour as of February 1, 2024) but there is no need to incur even this minimal cost unless you have a specific need for it. +See Step 1 Configuration for more information.

    4. +

Spot fleet costs can be minimized/stopped in multiple ways:

  1. We encourage the use of monitor during your job to help minimize the spot fleet cost as it automatically scales down your spot fleet request as your job queue empties and cancels your spot fleet request when you have no more jobs in the queue. diff --git a/searchindex.js b/searchindex.js index 833e6ea..5b9fbdc 100644 --- a/searchindex.js +++ b/searchindex.js @@ -1 +1 @@ -Search.setIndex({"docnames": ["AWS_hygiene_scripts", "SQS_QUEUE_information", "advanced_configuration", "config_examples", "costs", "dashboard", "external_buckets", "overview", "overview_2", "passing_files_to_DCP", "step_0_prep", "step_1_configuration", "step_2_submit_jobs", "step_3_start_cluster", "step_4_monitor", "troubleshooting_runs", "troubleshooting_start_cluster", "versions"], "filenames": ["AWS_hygiene_scripts.md", "SQS_QUEUE_information.md", "advanced_configuration.md", "config_examples.md", "costs.md", "dashboard.md", "external_buckets.md", "overview.md", "overview_2.md", "passing_files_to_DCP.md", "step_0_prep.md", "step_1_configuration.md", "step_2_submit_jobs.md", "step_3_start_cluster.md", "step_4_monitor.md", "troubleshooting_runs.md", "troubleshooting_start_cluster.md", "versions.md"], "titles": ["AWS Hygiene Scripts", "SQS QUEUE Information", "Advanced Configuration of DCP", "config.py configuration examples", "What does Distributed-CellProfiler cost?", "AWS Cloudwatch Dashboard", "Using External Buckets", "What is Distributed-CellProfiler?", "What happens in AWS when I run Distributed-CellProfiler?", "Passing Files to DCP", "Step 0: Prep", "Step 1: Configuration", "Step 2: Submit Jobs", "Step 3: Start Cluster", "Step 4: Monitor", "Troubleshooting", "Troubleshooting startCluster", "Versions"], "terms": {"see": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 17], "also": [0, 1, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 17], "auspic": 0, "set": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17], "up": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 13, 14, 15], "variou": 0, "automat": [0, 4, 8, 11, 13, 14, 15], "run": [0, 1, 2, 3, 4, 5, 6, 9, 10, 12, 15, 16, 17], "your": [0, 1, 3, 4, 5, 7, 8, 9, 10, 11, 15, 16], "account": [0, 3, 7, 8, 10, 11, 13, 15, 17], "python": [0, 11, 12, 13, 14, 17], "import": [0, 1, 9], "boto3": [0, 17], "time": [0, 1, 3, 5, 7, 8, 10, 11, 12, 13, 14, 15], "filterstr": 0, "myprojectnam": 0, "client": 0, "cloudwatch": [0, 1, 4, 8, 14, 15], "describe_alarm": 0, "alarmtyp": 0, "metricalarm": 0, "statevalu": 0, "insufficient_data": 0, "while": [0, 1, 4, 7, 8, 10, 11, 12], "true": [0, 1, 3, 4, 9, 11, 14, 15], "eachalarm": 0, "alarmnam": 0, "delete_alarm": 0, "sleep": 0, "1": [0, 1, 2, 3, 4, 5, 6, 8, 12, 14, 15], "avoid": [0, 2, 11, 15], "throttl": 0, "token": 0, "nexttoken": 0, "print": [0, 12, 15, 17], "bash": 0, "describ": [0, 3, 6, 10], "in2csv": 0, "f": [0, 12], "json": [0, 6, 10, 12, 13, 14, 15, 16], "kei": [0, 6, 10, 13, 15], "loggroup": 0, "csv": [0, 3, 11, 12, 15], "r": [0, 10, 11, 15], "requir": [0, 1, 3, 7, 8, 9, 11, 15, 17], "dplyr": 0, "readr": 0, "librari": [0, 7], "read_csv": 0, "col_typ": 0, "cols_onli": 0, "storedbyt": 0, "col_integ": 0, "creationtim": 0, "col_doubl": 0, "loggroupnam": 0, "col_charact": 0, "mutat": 0, "posixct": 0, "1000": 0, "origin": [0, 1], "1970": 0, "01": [0, 12], "filter": [0, 15, 17], "0": [0, 3, 4, 6, 11, 15], "select": [0, 1, 2, 4, 5, 9, 10, 13], "write_tsv": 0, "logs_clear": 0, "txt": [0, 9, 10, 12], "col_nam": 0, "parallel": [0, 12], "delet": [0, 1, 4, 14], "name": [0, 1, 2, 5, 6, 8, 9, 10, 11, 14, 15, 16, 17], "thi": [1, 2, 3, 5, 6, 9, 11, 12, 13, 14, 15, 16, 17], "depth": 1, "about": [1, 3, 5, 6, 7, 8, 9, 12, 14, 15], "configur": [1, 4, 6, 7, 14, 15, 16], "compon": [1, 12], "section": [1, 10, 14, 15], "step": [1, 3, 5, 6, 7, 8, 9, 15, 16], "distribut": [1, 3, 5, 6, 9, 10, 11, 12, 13, 14, 16, 17], "cellprofil": [1, 3, 5, 6, 9, 10, 12, 13, 14, 15, 16, 17], "where": [1, 2, 3, 5, 6, 7, 9, 11, 12, 14, 15], "all": [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17], "job": [1, 3, 4, 5, 8, 9, 10, 11, 13, 14, 15, 17], "ar": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17], "sent": [1, 11, 13, 14], "A": [1, 3, 5, 6, 8, 9, 11, 12, 15], "exactli": [1, 11], "what": [1, 5, 9, 11, 12, 15], "sound": 1, "like": [1, 3, 5, 6, 9, 10, 11, 12, 13, 15], "list": [1, 2, 3, 8, 10, 11, 12, 13, 15, 16, 17], "thing": [1, 5, 8, 12, 13, 14], "wait": [1, 5, 11, 15], "turn": [1, 3], "repres": [1, 13], "one": [1, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 15], "complet": [1, 5, 8, 10, 11, 13, 15, 17], "through": [1, 5, 6, 8, 9, 10, 11, 13, 15], "pipelin": [1, 3, 8, 9, 11, 12, 15, 17], "though": [1, 8, 10, 12, 13], "each": [1, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14], "mai": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16], "involv": [1, 10, 14], "ani": [1, 2, 3, 4, 5, 8, 9, 10, 11, 13, 14, 15], "number": [1, 2, 3, 4, 5, 8, 10, 11, 12, 13, 14, 15, 17], "imag": [1, 3, 7, 8, 9, 11, 12, 15, 17], "e": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17], "g": [1, 2, 3, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17], "analysi": [1, 3, 11, 12], "thousand": 1, "singl": [1, 3, 5, 8, 9, 11, 15], "make": [1, 3, 5, 8, 9, 10, 11, 12, 13, 15], "an": [1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 14, 15, 16, 17], "illumin": [1, 3, 11, 12], "correct": [1, 3, 11, 12, 15], "iter": 1, "produc": 1, "output": [1, 3, 6, 8, 11, 12, 13, 14, 15, 17], "file": [1, 2, 3, 4, 5, 6, 7, 8, 10, 13, 15, 16, 17], "you": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], "want": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15], "descript": [1, 11], "enough": [1, 3, 5, 11, 14], "distinguish": 1, "from": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17], "other": [1, 3, 5, 6, 8, 9, 10], "we": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13], "usual": [1, 3, 12, 15], "our": [1, 2, 3, 5, 7, 9, 10, 12, 13, 17], "base": [1, 3, 11, 12, 13, 15], "project": [1, 2, 3, 7, 9, 15], "goal": 1, "someth": [1, 15], "hepatocyte_differentiation_illum": 1, "lipid_droplet_analysi": 1, "fail": [1, 5, 11, 15], "If": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16], "everyth": [1, 7, 10], "goe": 1, "perfectli": 1, "alwai": [1, 7, 11, 14, 17], "remain": [1, 5, 14, 15], "empti": [1, 4, 5, 8, 15], "multipl": [1, 3, 4, 8, 11, 14, 15, 17], "default": [1, 2, 3, 6, 9, 10, 11, 12, 14, 15, 17], "10": [1, 3, 4, 6, 12, 14], "thei": [1, 3, 5, 8, 11, 12, 13, 14, 15, 17], "move": [1, 5, 11, 15, 17], "dead": [1, 8, 11], "letter": [1, 8, 11], "which": [1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17], "us": [1, 3, 4, 5, 8, 10, 11, 12, 13, 15, 16, 17], "initi": [1, 4, 5, 7, 10, 14, 15], "The": [1, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17], "therefor": [1, 5, 14], "function": [1, 3, 10, 12, 15], "effect": [1, 3, 6], "log": [1, 2, 6, 8, 10, 13, 14, 15, 17], "so": [1, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15], "can": [1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 17], "It": [1, 3, 5, 7, 8, 9, 10, 11, 12, 15], "differ": [1, 2, 3, 5, 8, 11, 12, 14, 15, 16, 17], "machin": [1, 3, 4, 5, 8, 9, 10, 11, 13, 14, 15, 17], "do": [1, 5, 6, 9, 10, 11, 12, 13, 14, 15], "try": [1, 2, 5, 10, 11, 12, 15, 16], "pull": [1, 5, 10, 13], "protip": 1, "member": 1, "team": [1, 3], "maintain": [1, 5, 9], "own": [1, 3, 5, 8, 10, 12, 15], "don": [1, 3, 6, 7, 8, 10, 11, 12, 13, 15, 17], "t": [1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 15, 17], "have": [1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16], "worri": [1, 7, 15], "find": [1, 9, 12, 13, 14, 15, 16], "messag": [1, 2, 4, 5, 8, 15], "peopl": 1, "same": [1, 4, 5, 6, 9, 11, 13, 15], "deadmessages_erin": 1, "end": [1, 4, 5, 8, 9, 11], "mani": [1, 5, 7, 8, 11, 12, 13, 15, 17], "place": [1, 2, 4, 8, 10, 11, 12, 13, 14, 15], "could": [1, 5, 8], "problem": [1, 7, 11, 15, 16], "hopefulli": 1, "ll": [1, 5, 7, 8, 13], "keep": [1, 2, 6, 8, 10, 11, 14], "ey": [1, 8, 14], "part": [1, 5, 11, 15, 16], "aw": [1, 2, 3, 4, 7, 13, 14, 15, 16], "monitor": [1, 2, 3, 4, 5, 8, 13, 17], "servic": [1, 4, 6, 7, 8, 11, 14], "after": [1, 2, 4, 5, 8, 9, 10, 11, 13, 14, 15], "start": [1, 4, 5, 8, 11, 12, 14, 15, 16], "catch": 1, "issu": [1, 9, 11, 15, 17], "befor": [1, 10, 11, 12, 15], "rest": 1, "successfulli": [1, 5, 10, 12], "corrupt": [1, 11, 15], "ha": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15], "save": [1, 5, 6, 11, 13, 14, 15], "properli": [1, 10, 12], "been": [1, 3, 5, 10, 11, 13, 14, 15], "damag": 1, "open": [1, 3, 8, 10, 15], "whether": [1, 5, 8, 11], "process": [1, 3, 7, 9, 10, 11, 12, 14, 15], "re": [1, 5, 7, 8, 11, 12, 13, 15], "interest": 1, "cellular": 1, "measur": [1, 3, 12], "per": [1, 2, 4, 5, 9, 10, 11, 12, 13], "basi": [1, 12], "when": [1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 15, 16, 17], "plate": [1, 3, 8, 12, 15, 17], "major": [1, 4, 8], "reason": [1, 11, 13], "why": [1, 5, 8], "certainli": [1, 10], "pai": [1, 7, 11, 13, 15], "cluster": [1, 3, 4, 5, 9, 11, 14, 16, 17], "indefinit": [1, 11, 15], "attempt": 1, "wouldn": 1, "necessarili": [1, 5, 15], "help": [1, 3, 4, 5, 7, 8, 11, 12, 14, 16, 17], "kind": [1, 9, 12], "error": [1, 8, 11, 12, 14, 15, 16, 17], "becaus": [1, 5, 13, 14, 15], "ten": 1, "hundr": 1, "success": [1, 9, 15], "instanc": [1, 2, 4, 7, 10, 13, 14, 15, 16], "etc": [1, 11, 12, 14, 15], "control": [1, 3, 6, 12], "how": [1, 3, 5, 7, 9, 10, 11, 12, 13, 15], "long": [1, 3, 5, 8, 11, 14, 15], "hidden": [1, 5, 11], "being": [1, 3, 5, 7, 8, 9, 11, 12, 15, 16], "must": [1, 3, 6, 9, 11, 12, 15, 17], "visibl": [1, 5, 8], "i": [1, 5, 6, 10, 11, 15], "order": [1, 9, 11, 12, 13, 17], "docker": [1, 3, 5, 8, 9, 13, 14, 15], "In": [1, 2, 4, 5, 7, 8, 13, 15, 16], "word": 1, "enter": [1, 2, 3, 6, 9, 10, 11, 13, 14], "allow": [1, 5, 6, 7, 10, 11, 12, 13, 17], "chanc": 1, "unhidden": 1, "made": [1, 8, 10, 11, 15], "avail": [1, 7, 10, 14], "copi": [1, 3, 6, 8, 9, 11, 13, 15], "s": [1, 5, 7, 8, 10, 11, 12, 13, 15], "quit": [1, 8, 14], "correctli": [1, 3, 10, 11, 15, 16], "typic": [1, 4, 6, 11, 15], "sai": [1, 14], "estim": [1, 11, 13], "5x": 1, "take": [1, 3, 5, 7, 8, 11, 12, 13, 14], "best": [1, 8, 12], "guess": 1, "sure": [1, 3, 5, 9, 10, 11, 12, 13, 15], "To": [1, 5, 6, 8, 9, 10], "understand": [1, 5, 12, 14], "consequ": 1, "incorrect": [1, 15], "let": [1, 13], "look": [1, 3, 5, 7, 10, 11, 13, 14, 15], "more": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17], "carefulli": 1, "two": [1, 5, 10, 17], "categori": 1, "flight": [1, 15], "regardless": 1, "mean": [1, 5, 9, 14, 15], "current": [1, 4, 10, 11, 14, 17], "hide": [1, 5], "submit": [1, 8, 10, 13, 14, 15], "config": [1, 2, 4, 5, 8, 9, 12, 14, 15, 17], "creat": [1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 14, 15, 16, 17], "out": [1, 5, 8, 11, 14, 15], "put": [1, 5, 8, 9, 12], "under": [1, 4, 9, 13, 15], "fleet": [1, 2, 4, 5, 7, 8, 10, 11, 14, 15, 16, 17], "ec2": [1, 2, 4, 6, 8, 13, 14, 15], "2": [1, 2, 3, 5, 8, 11, 13, 15], "ec": [1, 2, 4, 6, 8, 10, 13, 14, 15], "contain": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15], "those": [1, 7], "3": [1, 3, 4, 5, 8, 12, 15, 16, 17], "onc": [1, 3, 4, 5, 8, 10, 11, 12, 13, 14], "durat": [1, 4], "becom": [1, 14, 15], "again": [1, 10, 11, 13, 16], "tell": [1, 8, 9, 11], "anoth": [1, 8, 9, 11, 14], "too": [1, 8, 10, 11, 13, 15], "short": [1, 5, 11, 15], "even": [1, 5, 13, 14], "still": [1, 4, 17], "pick": 1, "come": [1, 5, 11], "along": [1, 8], "unnecessari": 1, "comput": [1, 6, 7, 8, 10, 11], "both": [1, 5, 13, 14], "continu": [1, 11, 13], "until": [1, 5, 8], "finish": [1, 2, 4, 5, 8, 11, 13, 15], "wast": [1, 11], "monei": [1, 5, 14], "crash": [1, 5, 8, 11, 14], "done": [1, 2, 4, 5, 9, 11, 15], "anyth": [1, 7, 8, 14], "caus": [1, 15, 16], "stop": [1, 4], "mid": 1, "remov": [1, 5, 8, 14, 15, 17], "outbid": 1, "stai": 1, "doesn": [1, 5, 7, 8, 11, 15], "doe": [1, 10, 15, 16], "limit": [1, 2, 3, 6, 13, 14], "power": [1, 7], "some": [1, 2, 3, 5, 8, 9, 11, 13, 14], "hang": [1, 5], "around": [1, 5, 10, 11], "noth": [1, 5, 15], "cost": 1, "doubt": 1, "better": [1, 3, 5, 11, 14, 15, 17], "than": [1, 3, 5, 7, 8, 11, 13, 14, 15, 17], "happen": [1, 5, 13, 14, 15], "rare": 1, "small": [1, 5, 8, 10, 11], "notic": [1, 11], "larger": [1, 5, 8, 11, 15], "There": [1, 10, 15], "easi": [1, 9], "wai": [1, 3, 4, 7, 9, 12, 14, 17], "appropri": [1, 5, 6, 10, 15], "amount": [1, 5, 8, 11], "first": [1, 5, 8, 9, 10, 11, 13, 14], "brand": 1, "new": [1, 3, 10, 11, 12, 14], "confirm": [1, 3, 10, 15, 16], "didn": 1, "need": [1, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15], "manual": [1, 3, 4, 16], "go": [1, 3, 4, 8, 11, 13], "figur": [1, 5], "got": [1, 15], "individu": [1, 5, 8, 11, 12, 13, 14], "never": [1, 3, 10, 15], "likewis": 1, "timeout": 1, "seem": [1, 13], "few": [1, 8, 13, 14], "cpu": [1, 3, 8, 11, 14], "usag": [1, 4, 5, 12, 14, 15], "high": 1, "check": [1, 5, 15, 16, 17], "execut": [1, 8, 9, 11], "either": [1, 5, 9, 15, 16], "begin": [1, 9, 12, 13], "note": [1, 3, 4, 5, 9, 11, 12, 13, 14, 15], "accur": 1, "idea": 1, "roughli": [1, 8, 11], "accordingli": [1, 11, 13, 15], "fly": 1, "updat": [1, 5, 11, 13, 15], "won": [1, 5, 9, 13], "affect": [1, 7, 8], "alreadi": [1, 11, 13, 15], "wa": [1, 5, 7, 9, 11, 12, 15, 16], "hour": [1, 4, 5, 11, 13, 14], "chang": [1, 3, 4, 5, 15, 17], "retri": [1, 11, 15], "count": [1, 3, 11], "everi": [1, 5, 10, 12, 13, 14], "instead": [1, 2, 9, 14, 15], "recommend": [1, 10, 11, 12, 13], "stochast": [1, 15], "failur": [1, 15], "possibl": [1, 5, 8, 11, 12, 17], "unavail": 1, "larg": [1, 3, 5, 7, 8, 11, 12, 15], "tend": [1, 11], "most": [1, 3, 5, 8, 10, 11, 14, 15, 17], "mode": [1, 2, 4, 10, 15, 17], "sample_sqs_queu": 1, "png": [1, 11], "alt": 1, "activ": [1, 10, 11], "task": [1, 4, 5, 6, 8, 11, 14], "64": [1, 8, 11], "32": 1, "moment": 1, "right": [1, 5], "lab": [1, 7, 12], "fortun": 1, "ve": [2, 10, 11, 12, 13], "tri": [2, 7, 11], "veri": [2, 3, 5, 15], "hard": [2, 3, 5, 8, 10, 11], "light": [2, 3], "adapt": 2, "manag": [2, 6], "assumpt": 2, "below": [2, 3, 6, 12, 13, 14], "non": [2, 6, 11, 12], "comprehens": 2, "code": [2, 8, 10], "purpos": 2, "locat": [2, 8, 9, 11, 12, 15], "By": [2, 12], "bucket": [2, 3, 7, 9, 10, 11, 14, 15, 17], "prefix": 2, "ecsconfig": [2, 15], "altern": [2, 15], "design": [2, 7, 11, 12, 17], "script": [2, 4, 7, 8, 9, 10, 11, 12, 14, 15], "export": [2, 9, 10, 11, 14], "group": [2, 5, 8, 9, 10, 12, 13, 15, 17], "retent": 2, "60": [2, 3], "dai": 2, "hit": 2, "250": 2, "them": [2, 5, 6, 8, 11, 12, 13, 14, 15], "exportedlog": 2, "log_group_nam": [2, 3, 11], "These": [2, 5, 8, 9, 10, 11, 13], "modifi": [2, 12], "addit": [2, 6, 9, 11, 12, 17], "spot": [2, 4, 5, 7, 8, 11, 14, 16], "instal": [2, 6, 7, 11, 13], "packag": [2, 7, 9], "startup": [2, 15], "userdata": 2, "paramet": [2, 3, 8, 14, 15, 16, 17], "sq": [2, 4, 5, 8, 12, 14, 15], "queue": [2, 3, 4, 5, 8, 12, 13], "detail": [2, 8, 9, 12, 13], "unprocess": 2, "expir": 2, "14": 2, "maximum": [2, 3, 13, 14, 15], "valu": [2, 3, 11, 13, 14, 15, 16], "py": [2, 6, 8, 9, 10, 11, 13, 14, 15, 17], "version": [2, 3, 6, 7, 11, 14, 15], "date": [2, 11], "latest": [2, 14], "stabl": [2, 5], "releas": 2, "case": [2, 5, 11, 15], "build": 2, "edit": [2, 6, 9, 10, 11, 14, 15], "dockerfil": [2, 9], "call": [2, 9, 12, 15, 16], "alarm": [2, 4, 5, 8, 11, 14], "threshold": 2, "worker": [2, 10, 15], "frequenc": 2, "type": [2, 3, 4, 5, 8, 9, 10, 11, 12, 13, 17], "inform": [2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16], "includ": [2, 5, 7, 8, 10, 11, 12, 13, 15], "adjust": [2, 5], "command": [2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 16], "line": [2, 3, 5, 9, 10, 12, 16], "flag": [2, 6, 11], "cp": [2, 3, 15], "stream": [2, 13], "level": 2, "At": [2, 9], "least": [2, 5], "4": [2, 3, 5, 8, 11, 15, 17], "dockerhub_tag": [2, 3, 11, 15], "bethcimini": [2, 17], "0_4": [2, 3, 11], "4_plugin": 2, "custom": [2, 5], "model": 2, "user": [2, 3, 6, 7, 9, 10, 12, 15], "train": 2, "gener": [2, 3, 6, 9, 10, 13, 15], "cellpos": [2, 11], "add": [2, 6, 8, 9, 10, 11, 12, 13, 16, 17], "s3": [2, 3, 6, 7, 8, 9, 10, 14, 15, 17], "follow": [2, 3, 5, 6, 8, 9, 10, 12, 13, 14, 16], "structur": [2, 9, 11, 12, 13, 15, 17], "organ": [2, 9, 11], "project_nam": [2, 3], "workspac": [2, 11], "custom_model_filenam": 2, "modul": [2, 9, 12, 15], "insid": [2, 11, 14], "detect": [2, 11], "pre": [2, 13], "mount": [2, 3, 8, 9, 10, 15], "path": [2, 3, 9, 11, 12, 15], "home": [2, 5, 7, 9, 15], "ubuntu": [2, 9, 15], "hand": 3, "standard": [3, 9], "workflow": [3, 7, 8, 9, 10, 11, 14, 15], "stereotyp": [3, 12], "fashion": 3, "cell": 3, "paint": 3, "assai": 3, "read": [3, 5, 8, 11, 12, 14, 15, 17], "context": 3, "here": [3, 4, 6, 8, 9, 12, 13], "z": 3, "pixel": 3, "plane": 3, "condens": 3, "content": 3, "focal": 3, "perform": [3, 4, 10, 11, 12, 14, 15], "downstream": 3, "batch": [3, 8, 12, 15, 17], "irregular": 3, "depend": [3, 4, 7, 10, 11, 12, 13, 15], "upon": [3, 15], "found": [3, 8, 9, 17], "qualiti": [3, 12], "provid": [3, 4, 10, 12, 13, 16], "metric": [3, 4, 5, 11, 14], "input": [3, 8, 9, 12, 15, 17], "necessari": [3, 6, 8, 10, 11, 12, 14], "particularli": [3, 11, 14], "improv": [3, 7, 11, 17], "wetlab": 3, "compar": [3, 5, 14], "across": [3, 5], "dataset": [3, 7], "dev": 3, "segment": 3, "quick": [3, 5], "outlin": [3, 8], "overlaid": 3, "multichannel": 3, "rescal": 3, "visual": 3, "inspect": 3, "often": [3, 5, 6, 11, 15], "stitch": 3, "pseudo": [3, 10], "view": [3, 5, 8, 11], "chosen": 3, "work": [3, 10, 12, 13], "appli": 3, "actual": [3, 5, 8, 9, 11, 15], "occur": [3, 5, 15, 16], "profil": [3, 6, 16], "taken": 3, "memori": [3, 5, 8, 11, 12, 14, 15], "reduc": [3, 11, 13, 15], "decreas": [3, 5, 8, 12, 15], "docker_cor": [3, 8, 11], "intern": 3, "app_nam": [3, 11, 14], "project_name_zproj": 3, "project_name_illum": 3, "project_name_qc": 3, "project_name_assaydev": 3, "project_name_analysi": 3, "excess": 3, "truncat": 3, "care": 3, "subsequ": [3, 13], "protocol": 3, "2021_06_08_wcpc_zproj": 3, "ensur": [3, 8, 9], "tag": [3, 10, 11, 17], "match": [3, 5, 12, 14, 15], "easili": [3, 10, 14], "text": [3, 6, 9, 12], "editor": 3, "3rd": 3, "daterevis": 3, "413": 3, "aws_region": [3, 6], "east": [3, 6, 10, 15], "aws_profil": [3, 6, 15], "ssh_key_nam": [3, 6, 15], "yourpem": 3, "pem": [3, 6, 10, 13, 15], "aws_bucket": [3, 6, 11], "dcp": [3, 11, 15, 17], "source_bucket": [3, 6, 11], "public": [3, 6], "cellpaint": [3, 6, 12], "galleri": 3, "destination_bucket": [3, 6, 11], "upload_flag": [3, 6, 11], "ecs_clust": [3, 8, 11], "just": [3, 5, 13, 15], "yournam": 3, "cluster_machin": [3, 11], "100": [3, 5, 11], "200": [3, 14], "round": [3, 5], "25": 3, "request": [3, 4, 5, 10, 14, 15, 16], "largest": 3, "capac": [3, 5, 13, 14, 17], "tasks_per_machin": [3, 11], "machine_typ": [3, 11, 15], "c5": [3, 16], "xlarg": [3, 11, 16], "histor": [3, 11], "m4": [3, 8, 11], "m5": 3, "howev": [3, 5, 8, 11, 13], "recent": [3, 14], "get": [3, 5, 7, 8, 10, 14, 15, 16], "m": 3, "class": [3, 10], "switch": 3, "c": [3, 5, 12, 15], "size": [3, 5, 8, 11, 15, 17], "between": [3, 5, 10, 13, 15, 17], "machine_pric": [3, 11], "20": 3, "Will": 3, "ebs_vol_s": [3, 11, 15], "system": [3, 7, 9, 10, 17], "22": [3, 11], "directli": [3, 4, 5, 8], "off": 3, "download_fil": [3, 9, 11, 15], "fals": [3, 4, 9, 11, 15], "download": [3, 9, 10, 11, 13, 15, 17], "40": 3, "eb": [3, 8, 11, 15, 17], "volum": [3, 8, 11, 13, 15, 17], "2k": 3, "might": [3, 11, 14], "cpu_shar": [3, 11], "1024": [3, 11], "7500": 3, "15000": [3, 11], "seconds_to_start": [3, 11, 15], "sqs_queue_nam": [3, 11], "sqs_message_vis": [3, 5, 11, 15], "240": 3, "15": [3, 8, 14], "120": 3, "expect": [3, 6, 15], "5": [3, 5, 15], "second": [3, 12, 13], "sqs_dead_letter_queu": [3, 11], "yourname_deadmessag": 3, "job_retri": [3, 11], "auto_monitor": [3, 11, 14], "create_dashboard": [3, 4, 11], "clean_dashboard": [3, 4, 11, 14], "check_if_done_bool": [3, 8, 11, 15], "overwrit": [3, 11, 14, 15], "old": [3, 4, 11, 15], "data": [3, 4, 7, 8, 10, 11, 12, 13, 15], "expected_number_fil": [3, 8, 11, 15], "channel": [3, 9, 15], "npy": 3, "isdon": 3, "experi": [3, 12], "nuclei": 3, "cytoplasm": 3, "underestim": 3, "overestim": 3, "min_file_size_byt": [3, 11], "necessary_str": [3, 11], "Not": [3, 5, 11], "always_continu": 3, "caution": [3, 11], "use_plugin": [3, 11], "update_plugin": [3, 11], "plugins_commit": [3, 11], "install_requir": [3, 11], "requirements_fil": [3, 11], "seri": [4, 7, 12], "three": [4, 8], "onli": [4, 5, 6, 7, 10, 11, 12, 13, 14, 17], "incur": [4, 14], "scale": [4, 5, 10, 14], "setup": [4, 5, 8, 10, 11, 13, 15], "definit": [4, 8, 13, 14], "entir": [4, 8, 11, 12], "free": [4, 7, 10, 13], "million": 4, "month": 4, "submitjob": [4, 8, 9, 12], "startclust": [4, 8, 13, 14], "option": [4, 5, 8, 9, 11, 12, 13, 14, 15, 17], "dashboard": [4, 8, 14], "exact": 4, "price": [4, 8, 11, 13, 14, 17], "bid": [4, 5, 8, 11, 13, 14], "minim": [4, 7, 11], "encourag": [4, 13], "dure": [4, 10, 11, 15], "down": [4, 5, 8, 12, 13, 14], "cancel": [4, 16, 17], "aggress": 4, "downscal": [4, 14, 15, 17], "engag": [4, 14], "cheapest": [4, 17], "cleanup": 4, "without": [4, 5, 10, 13, 14, 15], "abort": 4, "clean": [4, 5, 11, 14], "purg": 4, "consol": [4, 5, 8, 10, 16], "press": [4, 9], "action": [4, 6], "longer": [4, 5, 11, 14], "unus": 4, "state": [4, 10, 17], "insuffici": [4, 5], "hygien": 4, "50": [4, 14], "prevent": [4, 5, 11], "choos": [4, 13, 14], "customiz": 5, "page": [5, 7, 8, 12, 13], "resourc": [5, 6, 7, 8, 11, 16], "full": [5, 6, 8, 11], "widget": 5, "report": [5, 9, 15], "separ": [5, 7, 8, 12, 15], "auto": [5, 10, 11], "ds": [5, 8, 10], "popul": 5, "helfpul": 5, "post": 5, "mortem": 5, "previou": [5, 8, 11, 12, 13, 14, 15, 17], "consist": [5, 15], "manner": [5, 15], "told": [5, 9], "optim": [5, 13, 14], "thu": 5, "fulfil": 5, "given": [5, 8], "point": [5, 8, 9, 14, 15], "spin": 5, "hope": [5, 12], "straight": 5, "steadi": 5, "dip": [5, 14], "inevit": 5, "replac": [5, 6, 9, 14], "give": [5, 6, 7, 8, 11], "higher": [5, 11, 13], "bidder": 5, "progress": [5, 11, 13, 15], "lost": 5, "reach": [5, 15], "For": [5, 11, 12, 14], "much": [5, 8, 11, 13, 15], "impact": 5, "frustrat": 5, "potenti": 5, "expens": [5, 17], "lot": 5, "abl": [5, 11, 15], "futur": [5, 10, 12], "region": [5, 6, 10, 15], "blip": 5, "coincid": 5, "outag": [5, 11], "bad": [5, 11, 15], "luck": 5, "exampl": [5, 8, 12, 14, 15, 17], "abov": [5, 6, 10, 14, 15], "bulk": 5, "numberofmessagesdelet": 5, "well": [5, 6, 7, 8, 9, 12, 13, 15, 17], "back": [5, 8], "track": [5, 9], "top": [5, 10, 12, 16], "gap": 5, "fraction": 5, "insight": 5, "encount": [5, 15], "smallest": 5, "economi": 5, "sake": 5, "util": [5, 8], "intens": [5, 8, 11, 12], "averag": [5, 11], "minut": [5, 8, 13, 14], "window": 5, "max": [5, 15], "percentag": 5, "less": [5, 17], "aka": [5, 9, 14], "messagesnotvis": 5, "whichev": 5, "linear": 5, "declin": 5, "total": [5, 11, 15], "equal": 5, "core": [5, 11], "sometim": [5, 15], "rapid": 5, "increas": [5, 8, 9, 12, 15, 17], "pictur": 5, "disk": [5, 8, 14], "rapidli": [5, 15], "caught": 5, "reboot": [5, 8], "lose": 5, "appear": 5, "loss": 5, "were": [5, 11, 13], "return": [5, 12, 16, 17], "statu": 5, "rel": [5, 9], "slope": 5, "graph": 5, "discov": 5, "midwai": 5, "7": 5, "12": [5, 10, 12], "greater": [5, 8], "fewer": [5, 15], "specif": [5, 6, 7, 8, 9, 11, 13, 15, 17], "within": [5, 6, 8, 11, 15], "plot": [5, 11], "its": [5, 7, 8, 13, 14], "ideal": 5, "softwar": [5, 7, 8, 9, 10, 11], "entri": [5, 12], "toggl": 5, "row": [5, 9, 12, 15], "logstream": 5, "timescal": 5, "absolut": [5, 9], "upper": 5, "corner": 5, "screen": [5, 14], "zoom": 5, "particular": [5, 12, 15], "draw": 5, "box": 5, "itself": [5, 14, 15], "deselect": 5, "certain": [5, 13, 15, 17], "perman": 5, "simpli": [5, 9, 11, 12, 13], "click": [5, 13], "x": 5, "next": [5, 11, 13], "legend": 5, "hover": 5, "left": 5, "direct": [5, 6], "arrow": 5, "icon": 5, "drag": 5, "drop": [5, 15], "lower": 5, "diagon": 5, "desir": [5, 11, 17], "menu": 5, "refresh": 5, "ident": [6, 15], "access": [6, 7, 10, 11, 15, 17], "iam": [6, 16], "would": [6, 8, 9, 10, 11, 12, 14], "cli": [6, 11, 16], "upload": [6, 10, 11, 15, 17], "ssh": [6, 10], "role": [6, 13, 15, 16, 17], "acl": 6, "owner": 6, "metadata": [6, 12, 15, 17], "wil": 6, "further": [6, 10, 15], "someon": 6, "els": [6, 14], "handl": [6, 15, 17], "assum": [6, 10, 13], "learn": 6, "defin": [6, 8], "trust": 6, "relationship": 6, "templat": [6, 13], "2012": 6, "17": 6, "statement": 6, "princip": 6, "arn": [6, 13, 16], "123456789123": 6, "image_analyst": 6, "image_expert": 6, "amazonaw": [6, 15], "com": [6, 10, 15], "st": 6, "assumerol": 6, "node": [6, 12], "block": [6, 10], "bottom": [6, 9], "role_arn": 6, "source_profil": 6, "my": [6, 10], "credenti": [6, 10], "fill": 6, "info": 6, "aws_access_key_id": 6, "access_kei": 6, "aws_secret_access_kei": 6, "secret_access_kei": 6, "amazon": [7, 10, 13], "web": [7, 10, 13, 14], "storag": [7, 10, 13], "store": [7, 10], "cloud": [7, 9], "platform": [7, 10], "sourc": [7, 15], "oper": [7, 10, 15, 16], "benefit": 7, "eas": 7, "themselv": [7, 8], "reproduc": 7, "result": [7, 11, 13, 14], "fix": [7, 15], "flexibl": 7, "demand": 7, "infrastructur": [7, 14], "far": 7, "institut": 7, "great": 7, "piec": 7, "ad": [7, 14, 17], "consum": 7, "confus": 7, "leverag": 7, "former": 7, "latter": 7, "essenti": [7, 12], "termin": [7, 10, 11, 13, 14], "program": [7, 13], "feel": [7, 13], "cimini": 7, "broad": 7, "cambridg": 7, "ma": 7, "usa": 7, "develop": 7, "now": [7, 14, 16], "carpent": 7, "singh": 7, "repositori": [8, 11], "readm": 8, "respect": 8, "document": [8, 10, 11, 13, 15], "overview": 8, "explain": 8, "python3": 8, "gave": 8, "addition": [8, 14], "onto": [8, 11, 13, 14], "mismatch": [8, 15, 17], "accident": [8, 15], "intend": 8, "blindli": 8, "sit": 8, "idl": 8, "hook": 8, "_perinst": 8, "shut": [8, 14], "send": [8, 11, 12], "16xlarg": [8, 11], "250gb": 8, "ram": 8, "165": [8, 11], "16": [8, 11], "10gb": 8, "space": [8, 11, 15], "assign": 8, "gb": [8, 11], "share": 8, "among": [8, 13], "384": [8, 15], "timelaps": 8, "movi": 8, "stagger": 8, "aren": 8, "simultan": 8, "likelihood": 8, "degre": 8, "trial": 8, "local": [8, 9, 10, 13, 15], "sens": [8, 15], "drive": [8, 10, 11], "fluctuat": 8, "choic": [8, 12], "quickli": [8, 11, 13, 15], "willing": [8, 11, 13], "oppos": [8, 11], "ones": [8, 11], "risk": [8, 14], "tweak": 8, "five": 8, "primari": 8, "interact": [8, 10], "prepar": [8, 15], "outsid": 8, "granular": 8, "tab": 8, "browser": 8, "watch": 8, "behavior": 8, "especi": 8, "split": 9, "unlik": 9, "inclus": 9, "NOT": [9, 12, 16], "wish": [9, 13], "metdata": 9, "ye": 9, "enabl": [9, 10], "extract": [9, 12], "folder": [9, 11, 12, 15, 17], "present": [9, 11, 13, 17], "remot": 9, "method": 9, "should": [9, 10, 11, 13, 14], "pars": [9, 11], "minimum": [9, 11], "pathname_": 9, "nameofchannel": 9, "filename_": 9, "column": [9, 12, 13, 15], "metadata_": [9, 15], "pieceofmetadata": 9, "pathnam": [9, 15], "relativepath": 9, "odd": 9, "yourself": [9, 12], "via": [9, 17], "favorit": [9, 12], "languag": [9, 12], "phenix": 9, "xml": 9, "pe2loaddata": [9, 15], "namesandtyp": 9, "written": [9, 10], "video": 9, "hardcod": 9, "nest": [9, 15], "simpl": [9, 14], "eweisbar": 9, "desktop": 9, "run_work": 9, "sh": [9, 12, 15], "s3f": [9, 11, 15, 17], "bypass": [9, 15, 17], "data_fil": [9, 12], "run_batch_gener": [9, 15, 17], "stuff": 9, "argument": [9, 15], "makeanalysisjob": [9, 15], "unless": [9, 14, 15], "deviat": 9, "datafilepath": 9, "transit": 9, "h5": [9, 12, 15], "captur": 9, "togeth": [9, 11], "createbatchfil": [9, 12, 15], "map": 9, "root": [9, 11, 13], "analyz": [9, 12], "button": 9, "batchpipenam": 9, "batchpath": 9, "format": [9, 12], "implement": 10, "destroi": 10, "creation": [10, 16], "setup_aw": 10, "proce": 10, "login": 10, "secur": [10, 13, 16], "safe": [10, 11], "later": 10, "probabl": 10, "d": [10, 11, 12], "rather": [10, 15, 17], "pair": [10, 13, 15], "chmod": 10, "600": 10, "privat": 10, "connect": [10, 15], "vpc": [10, 13, 16], "subnet": [10, 13, 14, 16], "inbound": 10, "ip": 10, "address": 10, "lambda_funct": 10, "bucket_nam": 10, "quot": 10, "ecsinstancerol": [10, 15, 16], "permiss": [10, 15], "coordin": 10, "grant": 10, "launch": [10, 11, 13], "topic": 10, "trigger": [10, 14, 15], "link": [10, 13], "prefer": [10, 11], "dedic": 10, "simplic": 10, "special": 10, "micro": 10, "basic": 10, "guid": 10, "tool": [10, 12], "instruct": [10, 13, 14], "linux": [10, 13, 16], "support": [10, 11, 12, 17], "9": 10, "sudo": 10, "apt": 10, "git": [10, 11], "http": 10, "github": [10, 15], "distributedsci": 10, "cd": 10, "interfac": [10, 13, 14], "main": 10, "awscli": 10, "ignor": 10, "six": 10, "upgrad": 10, "last": [10, 14, 15], "west": 10, "eu": 10, "2a": 10, "real": 10, "gotten": 10, "know": [10, 15], "replic": 10, "repeat": 10, "decid": [10, 12], "specifi": [11, 12, 15], "tie": 11, "uniqu": 11, "apart": [11, 13], "analys": 11, "nuclearsegmentation_drosophila": 11, "encapsul": 11, "prep": 11, "write": 11, "extern": 11, "load_data": [11, 15], "distinct": 11, "wrong": [11, 15], "nuclearsegmentation_hela": 11, "handi": [11, 13], "histori": [11, 13], "tracker": [11, 13], "immedi": [11, 13], "justifi": [11, 13], "temporari": 11, "associ": [11, 14], "occasion": 11, "crop": 11, "slower": 11, "advis": 11, "length": [11, 15], "due": [11, 12, 14, 15], "lack": [11, 17], "slightli": 11, "stall": [11, 14, 15], "sinc": 11, "exist": [11, 13, 15, 16], "fact": 11, "proceed": [11, 15], "insensit": 11, "partwai": 11, "resubmit": 11, "whole": [11, 14, 15], "reprocess": [11, 15], "haven": [11, 15], "succeed": 11, "vs": [11, 15], "rerun": 11, "determin": 11, "previous": [11, 14], "mark": 11, "byte": 11, "object": 11, "smaller": [11, 12, 15], "string": 11, "toward": 11, "load": [11, 12, 15], "miss": [11, 15, 16], "harder": 11, "succesffulli": 11, "suggest": 11, "conjunct": 11, "pass": [11, 15], "directori": [11, 12, 15], "v1": 11, "active_plugin": 11, "revert": 11, "fetch": 11, "commit": 11, "checkout": 11, "deprec": [11, 13], "pip": 11, "older": [11, 14], "clone": 11, "newer": 11, "sample_distribut": 11, "cellprofiler_configuration_1": 11, "sample_dcp_config_1": 11, "fairli": 11, "4096": 11, "mb": 11, "cellprofiler_configuration_2": 11, "sample_dcp_config_2": 11, "break": 12, "site": [12, 15, 17], "readi": [12, 13], "yourjobfil": 12, "loaddata": [12, 15], "OR": [12, 15], "headless": 12, "versu": 12, "mechan": 12, "000": 12, "suffer": 12, "throughput": 12, "none": 12, "rule": 12, "flagimag": [12, 15], "scan": 12, "output_structur": [12, 17], "subfold": 12, "hyphen": [12, 17], "metadata_pl": [12, 15], "plate1": [12, 15], "metadata_wel": [12, 15], "a01": [12, 15], "output_top_directori": [12, 17], "append": 12, "gnu": 12, "combin": 12, "approach": 12, "uniform": 12, "96": 12, "echo": 12, "plate2": 12, "plate3": 12, "b": 12, "h": 12, "02": 12, "03": 12, "04": [12, 15], "05": 12, "06": 12, "07": 12, "08": 12, "09": 12, "11": [12, 15], "sort": 12, "variabl": [12, 15, 17], "besid": 12, "serv": 12, "shortcut": 12, "common": [12, 15], "regular": 12, "n": 12, "pathwai": 12, "comfort": 12, "faster": 12, "yourfleetfil": 13, "anywher": 13, "coupl": 13, "sever": 13, "app_namespotfleetrequestid": [13, 14], "walk": 13, "awai": 13, "ask": 13, "shortli": 13, "term": 13, "explan": 13, "cover": 13, "review": 13, "test": 13, "ami": [13, 16], "obtain": 13, "id": 13, "iamfleetrol": 13, "iaminstanceprofil": 13, "keynam": [13, 15], "subnetid": 13, "did": 13, "extens": [13, 15], "imageid": 13, "snapshotid": 13, "refer": [13, 15], "os": 13, "yourselv": 13, "good": 13, "had": 13, "pleas": [13, 15, 16], "repo": [13, 15], "offici": 13, "strongli": 13, "validfrom": 13, "validto": 13, "year": 13, "targetcapac": 13, "align": 13, "step_1_configur": 13, "md": 13, "instancetyp": 13, "spotpric": 13, "search": 13, "kill": [13, 14], "network": [13, 14], "On": 13, "snapshot": 13, "label": 14, "statist": 14, "relat": 14, "consecut": 14, "almost": 14, "show": [14, 15], "soon": 14, "whatev": 14, "tmux": 14, "disconnect": 14, "critic": 14, "lambda": 14, "forc": 14, "24": 14, "rise": 14, "charg": 14, "over": [14, 17], "rid": 14, "final": 14, "multi": [14, 15], "inher": 14, "overal": 14, "alloc": 14, "normal": 14, "monitor_fleet_id": 14, "sfr": 14, "9999ef99": 14, "99fc": 14, "9d9d": 14, "9999": 14, "9999999e99ab": 14, "monitor_app_nam": 14, "2021_12_13_project_analysi": 14, "monitor_ecs_clust": 14, "monitor_queue_nam": 14, "2021_12_13_project_analysisqueu": 14, "monitor_bucket_nam": 14, "monitor_log_group_nam": 14, "monitor_start_tim": 14, "1649187798951": 14, "troubl": 14, "kept": 14, "solut": 15, "beyond": 15, "throw": 15, "bug": 15, "expected_number_of_fil": 15, "6": 15, "batch_data": 15, "uncheck": 15, "valueerror": 15, "dictionari": 15, "sequenc": 15, "element": 15, "syntax": 15, "consid": 15, "Or": 15, "patient": 15, "seen": 15, "sqs_queue_inform": 15, "invalid": [15, 16], "routin": 15, "unrecover": 15, "No": 15, "extra": 15, "comma": 15, "somewher": 15, "invis": 15, "excel": 15, "panda": 15, "to_csv": 15, "index": 15, "datafram": 15, "indexerror": 15, "bound": 15, "axi": 15, "lead": 15, "zero": 15, "ie": [15, 17], "metadata_sit": 15, "isn": 15, "pad": 15, "batchfil": 15, "rang": 15, "mistak": 15, "exporttospreadsheet": 15, "sub": 15, "unusu": 15, "criteria": 15, "cppipe": 15, "outcom": 15, "typo": 15, "a1": 15, "attent": 15, "reflect": 15, "perinst": 15, "ioerror": 15, "indic": 15, "nameless": 15, "regist": 15, "cannot": 15, "err": 15, "slowdown": 15, "putobject": 15, "rate": 15, "preval": 15, "backlog": 15, "transport": 15, "endpoint": 15, "ssl": 15, "certif": 15, "subject": 15, "target": 15, "host": 15, "xxx": 15, "yyi": 15, "dot": 15, "use_path_request_styl": 15, "silent": 15, "mito": 15, "referenc": 15, "filenam": 15, "arrai": 15, "reduct": 15, "header": 15, "went": 15, "hint": 15, "field": 15, "mykeypair": 15, "examplefleet": 15, "With": 15, "examplejob_plateid": 15, "filterobject": 15, "saveimag": 15, "botocor": 16, "except": 16, "clienterror": 16, "invalidspotfleetrequestconfig": 16, "requestspotfleet": 16, "spotfleetrequestconfig": 16, "spotfleetrequestconfigurationinvalid": 16, "0f161e6034a6262d8": 16, "unix": 16, "fleetfil": 16, "012345678901": 16, "instanceprofil": 16, "One": 16, "valid": 16, "sg": 16, "01234567890123451atest": 16, "0123456789012345": 16, "dockerhub": 17, "dcpversion_cellprofilervers": 17, "0_3": 17, "8": 17, "versionnumb": 17, "plugin": 17, "abil": 17, "boto": 17, "fabric": 17, "elimin": 17, "check_if_don": 17, "cheapli": 17, "slowli": 17, "launchdata": 17, "submiss": 17, "discount": 17, "compat": 17, "item": 17, "discuss": 17}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"aw": [0, 5, 6, 8, 10, 11], "hygien": 0, "script": 0, "clean": 0, "out": [0, 13], "old": 0, "alarm": 0, "log": [0, 5, 11], "group": [0, 11, 16], "sq": [1, 11], "queue": [1, 11, 14], "inform": [1, 11], "sqs_queue_nam": 1, "sqs_dead_letter_queu": 1, "sqs_message_vis": 1, "job_retri": 1, "exampl": [1, 3, 6, 11], "advanc": 2, "configur": [2, 3, 8, 10, 11, 12, 13], "dcp": [2, 6, 9, 13], "chang": 2, "you": 2, "can": [2, 7], "make": 2, "distribut": [2, 4, 7, 8], "cellprofil": [2, 4, 7, 8, 11], "outsid": 2, "docker": [2, 7, 11], "contain": 2, "requir": [2, 10], "your": [2, 6, 12, 13, 14], "own": 2, "pipelin": 2, "us": [2, 6, 7, 9, 14], "runcellpos": 2, "plugin": [2, 11], "config": [3, 6, 11, 13], "py": [3, 12], "what": [4, 7, 8], "doe": [4, 8], "cost": 4, "cloudwatch": [5, 11], "dashboard": [5, 11], "fulfilledcapac": 5, "numberofmessagesreceiv": 5, "delet": 5, "memoryutil": 5, "messagesvis": 5, "notvis": 5, "distinct": 5, "all": 5, "show": 5, "error": 5, "interact": 5, "extern": 6, "bucket": 6, "setup": 6, "read": 6, "from": [6, 10], "cell": 6, "paint": 6, "galleri": 6, "write": 6, "collabor": 6, "s": 6, "permiss": 6, "In": 6, "account": 6, "instanc": [6, 8, 11], "why": 7, "would": 7, "i": [7, 8], "want": 7, "thi": [7, 8, 10], "do": [7, 8], "need": 7, "have": 7, "run": [7, 8, 11, 13, 14], "contribut": 7, "code": 7, "who": 7, "made": 7, "happen": 8, "when": [8, 14], "an": 8, "look": 8, "like": 8, "how": 8, "determin": 8, "my": 8, "pass": 9, "file": [9, 11, 12, 14], "metadata": 9, "load": 9, "data": 9, "creat": [9, 10], "loaddata": 9, "csv": 9, "batch": 9, "list": 9, "step": [10, 11, 12, 13, 14], "0": [10, 17], "prep": 10, "1": [10, 11, 13, 17], "manual": [10, 14], "resourc": 10, "2": [10, 12, 17], "automat": 10, "3": [10, 13], "auxiliari": 10, "4": [10, 14], "increas": 10, "spot": [10, 13], "limit": 10, "The": 10, "control": 10, "node": 10, "scratch": 10, "instal": 10, "python": 10, "8": 10, "higher": 10, "pip": 10, "clone": 10, "repositori": 10, "cli": 10, "s3f": 10, "fuse": 10, "option": 10, "ami": 10, "remov": 10, "long": 10, "term": 10, "infrastructur": 10, "role": 10, "lambda": 10, "monitor": [10, 11, 14], "sn": 10, "ec2": [10, 11], "compon": 11, "gener": 11, "set": 11, "AND": 11, "ec": 11, "environ": 11, "creation": 11, "redund": 11, "check": 11, "submit": 12, "job": 12, "overview": 12, "altern": 12, "submiss": 12, "run_batch_gener": 12, "start": 13, "cluster": 13, "fleet": 13, "request": 13, "paramet": 13, "must": 13, "current": 13, "version": [13, 17], "To": 13, "region": 13, "where": 13, "isn": 13, "t": 13, "avail": 13, "date": 13, "auto": 14, "function": 14, "while": 14, "analysi": 14, "total": 14, "empti": 14, "ar": 14, "visibl": 14, "Not": 14, "messag": 14, "cheapest": 14, "mode": 14, "troubleshoot": [15, 16], "startclust": 16, "iamfleetrol": 16, "iaminstanceprofil": 16, "subnetid": 16, "histori": 17, "forthcom": 17, "0rc2": 17, "releas": 17, "20201110": 17, "0rc1": 17, "20201105": 17, "20201103": 17, "20200109": 17, "updat": 17, "through": 17, "20191002": 17, "20181108": 17, "20170217": 17, "20170221": 17, "bugfix": 17, "20181018": 17, "20170213": 17}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 6, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinx": 56}}) \ No newline at end of file +Search.setIndex({"docnames": ["AWS_hygiene_scripts", "SQS_QUEUE_information", "advanced_configuration", "config_examples", "costs", "dashboard", "external_buckets", "overview", "overview_2", "passing_files_to_DCP", "step_0_prep", "step_1_configuration", "step_2_submit_jobs", "step_3_start_cluster", "step_4_monitor", "troubleshooting_runs", "troubleshooting_start_cluster", "versions"], "filenames": ["AWS_hygiene_scripts.md", "SQS_QUEUE_information.md", "advanced_configuration.md", "config_examples.md", "costs.md", "dashboard.md", "external_buckets.md", "overview.md", "overview_2.md", "passing_files_to_DCP.md", "step_0_prep.md", "step_1_configuration.md", "step_2_submit_jobs.md", "step_3_start_cluster.md", "step_4_monitor.md", "troubleshooting_runs.md", "troubleshooting_start_cluster.md", "versions.md"], "titles": ["AWS Hygiene Scripts", "SQS QUEUE Information", "Advanced Configuration of DCP", "config.py configuration examples", "What does Distributed-CellProfiler cost?", "AWS Cloudwatch Dashboard", "Using External Buckets", "What is Distributed-CellProfiler?", "What happens in AWS when I run Distributed-CellProfiler?", "Passing Files to DCP", "Step 0: Prep", "Step 1: Configuration", "Step 2: Submit Jobs", "Step 3: Start Cluster", "Step 4: Monitor", "Troubleshooting", "Troubleshooting startCluster", "Versions"], "terms": {"see": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 17], "also": [0, 1, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 17], "auspic": 0, "set": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17], "up": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 13, 14, 15], "variou": [0, 4], "automat": [0, 4, 8, 11, 13, 14, 15], "run": [0, 1, 2, 3, 4, 5, 6, 9, 10, 12, 15, 16, 17], "your": [0, 1, 3, 4, 5, 7, 8, 9, 10, 11, 15, 16], "account": [0, 3, 7, 8, 10, 11, 13, 15, 17], "python": [0, 11, 12, 13, 14, 17], "import": [0, 1, 9], "boto3": [0, 17], "time": [0, 1, 3, 5, 7, 8, 10, 11, 12, 13, 14, 15], "filterstr": 0, "myprojectnam": 0, "client": 0, "cloudwatch": [0, 1, 4, 8, 14, 15], "describe_alarm": 0, "alarmtyp": 0, "metricalarm": 0, "statevalu": 0, "insufficient_data": 0, "while": [0, 1, 4, 7, 8, 10, 11, 12], "true": [0, 1, 3, 4, 9, 11, 14, 15], "eachalarm": 0, "alarmnam": 0, "delete_alarm": 0, "sleep": 0, "1": [0, 1, 2, 3, 4, 5, 6, 8, 12, 14, 15], "avoid": [0, 2, 11, 15], "throttl": 0, "token": 0, "nexttoken": 0, "print": [0, 12, 15, 17], "bash": 0, "describ": [0, 3, 6, 10], "in2csv": 0, "f": [0, 12], "json": [0, 6, 10, 12, 13, 14, 15, 16], "kei": [0, 6, 10, 13, 15], "loggroup": 0, "csv": [0, 3, 11, 12, 15], "r": [0, 10, 11, 15], "requir": [0, 1, 3, 7, 8, 9, 11, 15, 17], "dplyr": 0, "readr": 0, "librari": [0, 7], "read_csv": 0, "col_typ": 0, "cols_onli": 0, "storedbyt": 0, "col_integ": 0, "creationtim": 0, "col_doubl": 0, "loggroupnam": 0, "col_charact": 0, "mutat": 0, "posixct": 0, "1000": 0, "origin": [0, 1, 11], "1970": 0, "01": [0, 12], "filter": [0, 15, 17], "0": [0, 3, 4, 6, 11, 15], "select": [0, 1, 2, 4, 5, 9, 10, 13], "write_tsv": 0, "logs_clear": 0, "txt": [0, 9, 10, 12], "col_nam": 0, "parallel": [0, 12], "delet": [0, 1, 4, 14], "name": [0, 1, 2, 5, 6, 8, 9, 10, 11, 14, 15, 16, 17], "thi": [1, 2, 3, 4, 5, 6, 9, 11, 12, 13, 14, 15, 16, 17], "depth": 1, "about": [1, 3, 5, 6, 7, 8, 9, 12, 14, 15], "configur": [1, 4, 6, 7, 14, 15, 16], "compon": [1, 12], "section": [1, 10, 14, 15], "step": [1, 3, 4, 5, 6, 7, 8, 9, 15, 16], "distribut": [1, 3, 5, 6, 9, 10, 11, 12, 13, 14, 16, 17], "cellprofil": [1, 3, 5, 6, 9, 10, 12, 13, 14, 15, 16, 17], "where": [1, 2, 3, 5, 6, 7, 9, 11, 12, 14, 15], "all": [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17], "job": [1, 3, 4, 5, 8, 9, 10, 11, 13, 14, 15, 17], "ar": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17], "sent": [1, 11, 13, 14], "A": [1, 3, 5, 6, 8, 9, 11, 12, 15], "exactli": [1, 11], "what": [1, 5, 9, 11, 12, 15], "sound": 1, "like": [1, 3, 5, 6, 9, 10, 11, 12, 13, 15], "list": [1, 2, 3, 8, 10, 11, 12, 13, 15, 16, 17], "thing": [1, 5, 8, 12, 13, 14], "wait": [1, 5, 11, 15], "turn": [1, 3], "repres": [1, 13], "one": [1, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 15], "complet": [1, 5, 8, 10, 11, 13, 15, 17], "through": [1, 5, 6, 8, 9, 10, 11, 13, 15], "pipelin": [1, 3, 8, 9, 11, 12, 15, 17], "though": [1, 8, 10, 12, 13], "each": [1, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14], "mai": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16], "involv": [1, 10, 14], "ani": [1, 2, 3, 4, 5, 8, 9, 10, 11, 13, 14, 15], "number": [1, 2, 3, 4, 5, 8, 10, 11, 12, 13, 14, 15, 17], "imag": [1, 3, 7, 8, 9, 11, 12, 15, 17], "e": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17], "g": [1, 2, 3, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17], "analysi": [1, 3, 11, 12], "thousand": 1, "singl": [1, 3, 5, 8, 9, 11, 15], "make": [1, 3, 5, 8, 9, 10, 11, 12, 13, 15], "an": [1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 14, 15, 16, 17], "illumin": [1, 3, 11, 12], "correct": [1, 3, 11, 12, 15], "iter": 1, "produc": 1, "output": [1, 3, 6, 8, 11, 12, 13, 14, 15, 17], "file": [1, 2, 3, 4, 5, 6, 7, 8, 10, 13, 15, 16, 17], "you": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], "want": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15], "descript": [1, 11], "enough": [1, 3, 5, 11, 14], "distinguish": 1, "from": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17], "other": [1, 3, 5, 6, 8, 9, 10], "we": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13], "usual": [1, 3, 12, 15], "our": [1, 2, 3, 5, 7, 9, 10, 12, 13, 17], "base": [1, 3, 4, 11, 12, 13, 15], "project": [1, 2, 3, 7, 9, 15], "goal": 1, "someth": [1, 15], "hepatocyte_differentiation_illum": 1, "lipid_droplet_analysi": 1, "fail": [1, 5, 11, 15], "If": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16], "everyth": [1, 7, 10], "goe": 1, "perfectli": 1, "alwai": [1, 7, 11, 14, 17], "remain": [1, 5, 14, 15], "empti": [1, 4, 5, 8, 15], "multipl": [1, 3, 4, 8, 11, 14, 15, 17], "default": [1, 2, 3, 6, 9, 10, 11, 12, 14, 15, 17], "10": [1, 3, 4, 6, 12, 14], "thei": [1, 3, 5, 8, 11, 12, 13, 14, 15, 17], "move": [1, 5, 11, 15, 17], "dead": [1, 8, 11], "letter": [1, 8, 11], "which": [1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17], "us": [1, 3, 4, 5, 8, 10, 11, 12, 13, 15, 16, 17], "initi": [1, 4, 5, 7, 10, 14, 15], "The": [1, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17], "therefor": [1, 5, 14], "function": [1, 3, 10, 12, 15], "effect": [1, 3, 6], "log": [1, 2, 6, 8, 10, 13, 14, 15, 17], "so": [1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15], "can": [1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 17], "It": [1, 3, 5, 7, 8, 9, 10, 11, 12, 15], "differ": [1, 2, 3, 5, 8, 11, 12, 14, 15, 16, 17], "machin": [1, 3, 4, 5, 8, 9, 10, 11, 13, 14, 15, 17], "do": [1, 5, 6, 9, 10, 11, 12, 13, 14, 15], "try": [1, 2, 5, 10, 11, 12, 15, 16], "pull": [1, 5, 10, 13], "protip": 1, "member": 1, "team": [1, 3], "maintain": [1, 5, 9], "own": [1, 3, 5, 8, 10, 12, 15], "don": [1, 3, 4, 6, 7, 8, 10, 11, 12, 13, 15, 17], "t": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 17], "have": [1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16], "worri": [1, 7, 15], "find": [1, 9, 12, 13, 14, 15, 16], "messag": [1, 2, 4, 5, 8, 15], "peopl": 1, "same": [1, 4, 5, 6, 9, 11, 13, 15], "deadmessages_erin": 1, "end": [1, 4, 5, 8, 9, 11], "mani": [1, 5, 7, 8, 11, 12, 13, 15, 17], "place": [1, 2, 4, 8, 10, 11, 12, 13, 14, 15], "could": [1, 5, 8, 11], "problem": [1, 7, 11, 15, 16], "hopefulli": 1, "ll": [1, 5, 7, 8, 13], "keep": [1, 2, 6, 8, 10, 11, 14], "ey": [1, 8, 14], "part": [1, 5, 11, 15, 16], "aw": [1, 2, 3, 4, 7, 13, 14, 15, 16], "monitor": [1, 2, 3, 4, 5, 8, 13, 17], "servic": [1, 4, 6, 7, 8, 11, 14], "after": [1, 2, 4, 5, 8, 9, 10, 11, 13, 14, 15], "start": [1, 4, 5, 8, 11, 12, 14, 15, 16], "catch": 1, "issu": [1, 9, 11, 15, 17], "befor": [1, 10, 11, 12, 15], "rest": 1, "successfulli": [1, 5, 10, 12], "corrupt": [1, 11, 15], "ha": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15], "save": [1, 5, 6, 11, 13, 14, 15], "properli": [1, 10, 12], "been": [1, 3, 5, 10, 11, 13, 14, 15], "damag": 1, "open": [1, 3, 8, 10, 15], "whether": [1, 5, 8, 11], "process": [1, 3, 7, 9, 10, 11, 12, 14, 15], "re": [1, 5, 7, 8, 11, 12, 13, 15], "interest": 1, "cellular": 1, "measur": [1, 3, 12], "per": [1, 2, 4, 5, 9, 10, 11, 12, 13], "basi": [1, 12], "when": [1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 15, 16, 17], "plate": [1, 3, 8, 12, 15, 17], "major": [1, 4, 8], "reason": [1, 4, 11, 13], "why": [1, 5, 8], "certainli": [1, 10], "pai": [1, 4, 7, 11, 13, 15], "cluster": [1, 3, 4, 5, 9, 11, 14, 16, 17], "indefinit": [1, 11, 15], "attempt": 1, "wouldn": 1, "necessarili": [1, 5, 15], "help": [1, 3, 4, 5, 7, 8, 11, 12, 14, 16, 17], "kind": [1, 9, 12], "error": [1, 8, 11, 12, 14, 15, 16, 17], "becaus": [1, 5, 11, 13, 14, 15], "ten": 1, "hundr": 1, "success": [1, 9, 15], "instanc": [1, 2, 4, 7, 10, 13, 14, 15, 16], "etc": [1, 11, 12, 14, 15], "control": [1, 3, 6, 12], "how": [1, 3, 5, 7, 9, 10, 11, 12, 13, 15], "long": [1, 3, 5, 8, 11, 14, 15], "hidden": [1, 5, 11], "being": [1, 3, 5, 7, 8, 9, 11, 12, 15, 16], "must": [1, 3, 6, 9, 11, 12, 15, 17], "visibl": [1, 5, 8], "i": [1, 5, 6, 10, 11, 15], "order": [1, 9, 11, 12, 13, 17], "docker": [1, 3, 5, 8, 9, 13, 14, 15], "In": [1, 2, 4, 5, 7, 8, 13, 15, 16], "word": 1, "enter": [1, 2, 3, 6, 9, 10, 11, 13, 14], "allow": [1, 5, 6, 7, 10, 11, 12, 13, 17], "chanc": 1, "unhidden": 1, "made": [1, 8, 10, 11, 15], "avail": [1, 4, 7, 10, 14], "copi": [1, 3, 6, 8, 9, 11, 13, 15], "s": [1, 5, 7, 8, 10, 11, 12, 13, 15], "quit": [1, 8, 14], "correctli": [1, 3, 10, 11, 15, 16], "typic": [1, 4, 6, 11, 15], "sai": [1, 14], "estim": [1, 11, 13], "5x": 1, "take": [1, 3, 5, 7, 8, 11, 12, 13, 14], "best": [1, 8, 12], "guess": 1, "sure": [1, 3, 5, 9, 10, 11, 12, 13, 15], "To": [1, 5, 6, 8, 9, 10], "understand": [1, 5, 12, 14], "consequ": 1, "incorrect": [1, 15], "let": [1, 13], "look": [1, 3, 5, 7, 10, 11, 13, 14, 15], "more": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17], "carefulli": 1, "two": [1, 5, 10, 17], "categori": 1, "flight": [1, 15], "regardless": 1, "mean": [1, 5, 9, 14, 15], "current": [1, 4, 10, 11, 14, 17], "hide": [1, 5], "submit": [1, 8, 10, 13, 14, 15], "config": [1, 2, 4, 5, 8, 9, 12, 14, 15, 17], "creat": [1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 14, 15, 16, 17], "out": [1, 5, 8, 11, 14, 15], "put": [1, 5, 8, 9, 12], "under": [1, 4, 9, 13, 15], "fleet": [1, 2, 4, 5, 7, 8, 10, 11, 14, 15, 16, 17], "ec2": [1, 2, 4, 6, 8, 13, 14, 15], "2": [1, 2, 3, 5, 8, 11, 13, 15], "ec": [1, 2, 4, 6, 8, 10, 13, 14, 15], "contain": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15], "those": [1, 7], "3": [1, 3, 4, 5, 8, 12, 15, 16, 17], "onc": [1, 3, 4, 5, 8, 10, 11, 12, 13, 14], "durat": [1, 4], "becom": [1, 14, 15], "again": [1, 10, 11, 13, 16], "tell": [1, 8, 9, 11], "anoth": [1, 8, 9, 11, 14], "too": [1, 8, 10, 11, 13, 15], "short": [1, 5, 11, 15], "even": [1, 4, 5, 13, 14], "still": [1, 4, 17], "pick": 1, "come": [1, 5, 11], "along": [1, 8], "unnecessari": 1, "comput": [1, 6, 7, 8, 10, 11], "both": [1, 5, 13, 14], "continu": [1, 11, 13], "until": [1, 5, 8], "finish": [1, 2, 4, 5, 8, 11, 13, 15], "wast": [1, 11], "monei": [1, 5, 14], "crash": [1, 5, 8, 11, 14], "done": [1, 2, 4, 5, 9, 11, 15], "anyth": [1, 7, 8, 14], "caus": [1, 15, 16], "stop": [1, 4], "mid": 1, "remov": [1, 5, 8, 14, 15, 17], "outbid": 1, "stai": 1, "doesn": [1, 5, 7, 8, 11, 15], "doe": [1, 10, 15, 16], "limit": [1, 2, 3, 6, 13, 14], "power": [1, 7], "some": [1, 2, 3, 5, 8, 9, 11, 13, 14], "hang": [1, 5], "around": [1, 5, 10, 11], "noth": [1, 5, 15], "cost": [1, 11], "doubt": 1, "better": [1, 3, 5, 11, 14, 15, 17], "than": [1, 3, 5, 7, 8, 11, 13, 14, 15, 17], "happen": [1, 5, 13, 14, 15], "rare": 1, "small": [1, 5, 8, 10, 11], "notic": [1, 11], "larger": [1, 5, 8, 11, 15], "There": [1, 10, 15], "easi": [1, 9], "wai": [1, 3, 4, 7, 9, 12, 14, 17], "appropri": [1, 5, 6, 10, 15], "amount": [1, 5, 8, 11], "first": [1, 5, 8, 9, 10, 11, 13, 14], "brand": 1, "new": [1, 3, 10, 11, 12, 14], "confirm": [1, 3, 10, 15, 16], "didn": 1, "need": [1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15], "manual": [1, 3, 4, 16], "go": [1, 3, 4, 8, 11, 13], "figur": [1, 5], "got": [1, 15], "individu": [1, 5, 8, 11, 12, 13, 14], "never": [1, 3, 10, 15], "likewis": 1, "timeout": 1, "seem": [1, 13], "few": [1, 8, 13, 14], "cpu": [1, 3, 8, 11, 14], "usag": [1, 4, 5, 12, 14, 15], "high": 1, "check": [1, 5, 15, 16, 17], "execut": [1, 8, 9, 11], "either": [1, 5, 9, 15, 16], "begin": [1, 9, 12, 13], "note": [1, 3, 4, 5, 9, 11, 12, 13, 14, 15], "accur": 1, "idea": 1, "roughli": [1, 8, 11], "accordingli": [1, 11, 13, 15], "fly": 1, "updat": [1, 5, 11, 13, 15], "won": [1, 5, 9, 13], "affect": [1, 7, 8], "alreadi": [1, 11, 13, 15], "wa": [1, 5, 7, 9, 11, 12, 15, 16], "hour": [1, 4, 5, 11, 13, 14], "chang": [1, 3, 4, 5, 15, 17], "retri": [1, 11, 15], "count": [1, 3, 11], "everi": [1, 5, 10, 12, 13, 14], "instead": [1, 2, 9, 14, 15], "recommend": [1, 10, 11, 12, 13], "stochast": [1, 15], "failur": [1, 15], "possibl": [1, 4, 5, 8, 11, 12, 17], "unavail": 1, "larg": [1, 3, 5, 7, 8, 11, 12, 15], "tend": [1, 11], "most": [1, 3, 5, 8, 10, 11, 14, 15, 17], "mode": [1, 2, 4, 10, 15, 17], "sample_sqs_queu": 1, "png": 1, "alt": 1, "activ": [1, 10, 11], "task": [1, 4, 5, 6, 8, 11, 14], "64": [1, 8, 11], "32": 1, "moment": 1, "right": [1, 5], "lab": [1, 7, 12], "fortun": 1, "ve": [2, 10, 11, 12, 13], "tri": [2, 7, 11], "veri": [2, 3, 5, 15], "hard": [2, 3, 5, 8, 10, 11], "light": [2, 3], "adapt": 2, "manag": [2, 6], "assumpt": 2, "below": [2, 3, 6, 12, 13, 14], "non": [2, 6, 11, 12], "comprehens": 2, "code": [2, 8, 10], "purpos": 2, "locat": [2, 8, 9, 11, 12, 15], "By": [2, 11, 12], "bucket": [2, 3, 4, 7, 9, 10, 11, 14, 15, 17], "prefix": 2, "ecsconfig": [2, 15], "altern": [2, 15], "design": [2, 7, 11, 12, 17], "script": [2, 4, 7, 8, 9, 10, 11, 12, 14, 15], "export": [2, 9, 10, 11, 14], "group": [2, 5, 8, 9, 10, 12, 13, 15, 17], "retent": 2, "60": [2, 3], "dai": 2, "hit": 2, "250": 2, "them": [2, 5, 6, 8, 11, 12, 13, 14, 15], "exportedlog": 2, "log_group_nam": [2, 3, 11], "These": [2, 5, 8, 9, 10, 11, 13], "modifi": [2, 12], "addit": [2, 6, 9, 11, 12, 17], "spot": [2, 4, 5, 7, 8, 11, 14, 16], "instal": [2, 6, 7, 11, 13], "packag": [2, 7, 9], "startup": [2, 15], "userdata": 2, "paramet": [2, 3, 8, 14, 15, 16, 17], "sq": [2, 4, 5, 8, 12, 14, 15], "queue": [2, 3, 4, 5, 8, 12, 13], "detail": [2, 8, 9, 12, 13], "unprocess": 2, "expir": 2, "14": 2, "maximum": [2, 3, 13, 14, 15], "valu": [2, 3, 11, 13, 14, 15, 16], "py": [2, 6, 8, 9, 10, 11, 13, 14, 15, 17], "version": [2, 3, 6, 7, 11, 14, 15], "date": [2, 11], "latest": [2, 14], "stabl": [2, 5], "releas": 2, "case": [2, 5, 11, 15], "build": 2, "edit": [2, 6, 9, 10, 11, 14, 15], "dockerfil": [2, 9], "call": [2, 9, 12, 15, 16], "alarm": [2, 4, 5, 8, 11, 14], "threshold": 2, "worker": [2, 10, 15], "frequenc": 2, "type": [2, 3, 4, 5, 8, 9, 10, 11, 12, 13, 17], "inform": [2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16], "includ": [2, 4, 5, 7, 8, 10, 11, 12, 13, 15], "adjust": [2, 5], "command": [2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 16], "line": [2, 3, 5, 9, 10, 12, 16], "flag": [2, 6, 11], "cp": [2, 3, 15], "stream": [2, 13], "level": [2, 11], "At": [2, 9], "least": [2, 5], "4": [2, 3, 5, 8, 11, 15, 17], "dockerhub_tag": [2, 3, 11, 15], "bethcimini": [2, 17], "0_4": [2, 3, 11], "4_plugin": 2, "custom": [2, 5], "model": 2, "user": [2, 3, 6, 7, 9, 10, 12, 15], "train": 2, "gener": [2, 3, 6, 9, 10, 13, 15], "cellpos": [2, 11], "add": [2, 6, 8, 9, 10, 11, 12, 13, 16, 17], "s3": [2, 3, 4, 6, 7, 8, 9, 10, 11, 14, 15, 17], "follow": [2, 3, 5, 6, 8, 9, 10, 12, 13, 14, 16], "structur": [2, 9, 11, 12, 13, 15, 17], "organ": [2, 9, 11], "project_nam": [2, 3], "workspac": [2, 11], "custom_model_filenam": 2, "modul": [2, 9, 12, 15], "insid": [2, 11, 14], "detect": [2, 11], "pre": [2, 13], "mount": [2, 3, 4, 8, 9, 10, 11, 15], "path": [2, 3, 9, 11, 12, 15], "home": [2, 5, 7, 9, 15], "ubuntu": [2, 9, 15], "hand": 3, "standard": [3, 9], "workflow": [3, 7, 8, 9, 10, 11, 14, 15], "stereotyp": [3, 12], "fashion": 3, "cell": 3, "paint": 3, "assai": 3, "read": [3, 5, 8, 11, 12, 14, 15, 17], "context": 3, "here": [3, 4, 6, 8, 9, 12, 13], "z": 3, "pixel": 3, "plane": 3, "condens": 3, "content": 3, "focal": 3, "perform": [3, 4, 10, 11, 12, 14, 15], "downstream": 3, "batch": [3, 8, 12, 15, 17], "irregular": 3, "depend": [3, 4, 7, 10, 11, 12, 13, 15], "upon": [3, 15], "found": [3, 8, 9, 17], "qualiti": [3, 12], "provid": [3, 4, 10, 12, 13, 16], "metric": [3, 4, 5, 11, 14], "input": [3, 8, 9, 12, 15, 17], "necessari": [3, 6, 8, 10, 11, 12, 14], "particularli": [3, 11, 14], "improv": [3, 7, 11, 17], "wetlab": 3, "compar": [3, 5, 11, 14], "across": [3, 5], "dataset": [3, 7], "dev": 3, "segment": 3, "quick": [3, 5], "outlin": [3, 8], "overlaid": 3, "multichannel": 3, "rescal": 3, "visual": 3, "inspect": 3, "often": [3, 4, 5, 6, 11, 15], "stitch": 3, "pseudo": [3, 10, 11], "view": [3, 5, 8, 11], "chosen": 3, "work": [3, 10, 12, 13], "appli": 3, "actual": [3, 4, 5, 8, 9, 11, 15], "occur": [3, 5, 15, 16], "profil": [3, 6, 16], "taken": 3, "memori": [3, 4, 5, 8, 11, 12, 14, 15], "reduc": [3, 11, 13, 15], "decreas": [3, 5, 8, 12, 15], "docker_cor": [3, 8, 11], "intern": 3, "app_nam": [3, 11, 14], "project_name_zproj": 3, "project_name_illum": 3, "project_name_qc": 3, "project_name_assaydev": 3, "project_name_analysi": 3, "excess": 3, "truncat": 3, "care": 3, "subsequ": [3, 13], "protocol": 3, "2021_06_08_wcpc_zproj": 3, "ensur": [3, 8, 9], "tag": [3, 10, 11, 17], "match": [3, 5, 12, 14, 15], "easili": [3, 10, 14], "text": [3, 6, 9, 12], "editor": 3, "3rd": 3, "daterevis": 3, "413": 3, "aws_region": [3, 6], "east": [3, 6, 10, 15], "aws_profil": [3, 6, 15], "ssh_key_nam": [3, 6, 15], "yourpem": 3, "pem": [3, 6, 10, 13, 15], "aws_bucket": [3, 6, 11], "dcp": [3, 11, 15, 17], "source_bucket": [3, 6, 11], "public": [3, 4, 6, 11], "cellpaint": [3, 6, 12], "galleri": 3, "destination_bucket": [3, 6, 11], "upload_flag": [3, 6, 11], "ecs_clust": [3, 8, 11], "just": [3, 5, 13, 15], "yournam": 3, "cluster_machin": [3, 11], "100": [3, 5, 11], "200": [3, 14], "round": [3, 5], "25": 3, "request": [3, 4, 5, 10, 14, 15, 16], "largest": 3, "capac": [3, 5, 13, 14, 17], "tasks_per_machin": [3, 11], "machine_typ": [3, 4, 11, 15], "c5": [3, 16], "xlarg": [3, 11, 16], "histor": [3, 11], "m4": [3, 8, 11], "m5": 3, "howev": [3, 5, 8, 11, 13], "recent": [3, 14], "get": [3, 5, 7, 8, 10, 14, 15, 16], "m": 3, "class": [3, 10], "switch": 3, "c": [3, 5, 12, 15], "size": [3, 5, 8, 11, 15, 17], "between": [3, 5, 10, 13, 15, 17], "machine_pric": [3, 11], "20": 3, "Will": 3, "ebs_vol_s": [3, 4, 11, 15], "system": [3, 7, 9, 10, 11, 17], "22": [3, 11], "directli": [3, 4, 5, 8], "off": 3, "download_fil": [3, 4, 9, 11, 15], "fals": [3, 4, 9, 11, 15], "download": [3, 9, 10, 11, 13, 15, 17], "40": 3, "eb": [3, 8, 11, 15, 17], "volum": [3, 8, 11, 13, 15, 17], "assign_ip": [3, 4, 11], "2k": 3, "might": [3, 11, 14], "cpu_shar": [3, 11], "1024": [3, 11], "7500": 3, "15000": [3, 11], "seconds_to_start": [3, 11, 15], "sqs_queue_nam": [3, 11], "sqs_message_vis": [3, 5, 11, 15], "240": 3, "15": [3, 8, 14], "120": 3, "expect": [3, 6, 15], "5": [3, 5, 15], "second": [3, 12, 13], "sqs_dead_letter_queu": [3, 11], "yourname_deadmessag": 3, "job_retri": [3, 11], "auto_monitor": [3, 11, 14], "create_dashboard": [3, 4, 11], "clean_dashboard": [3, 4, 11, 14], "check_if_done_bool": [3, 8, 11, 15], "overwrit": [3, 11, 14, 15], "old": [3, 4, 11, 15], "data": [3, 4, 7, 8, 10, 11, 12, 13, 15], "expected_number_fil": [3, 8, 11, 15], "channel": [3, 9, 15], "npy": 3, "isdon": 3, "experi": [3, 12], "nuclei": 3, "cytoplasm": 3, "underestim": 3, "overestim": 3, "min_file_size_byt": [3, 11], "necessary_str": [3, 11], "Not": [3, 5, 11], "always_continu": 3, "caution": [3, 11], "use_plugin": [3, 11], "update_plugin": [3, 11], "plugins_commit": [3, 11], "install_requir": [3, 11], "requirements_fil": [3, 11], "seri": [4, 7, 12], "three": [4, 8], "onli": [4, 5, 6, 7, 10, 11, 12, 13, 14, 17], "incur": [4, 14], "scale": [4, 5, 10, 14], "setup": [4, 5, 8, 10, 11, 13, 15], "definit": [4, 8, 13, 14], "entir": [4, 8, 11, 12], "free": [4, 7, 10, 13], "million": 4, "month": 4, "submitjob": [4, 8, 9, 12], "startclust": [4, 8, 13, 14], "option": [4, 5, 8, 9, 11, 12, 13, 14, 15, 17], "dashboard": [4, 8, 14], "exact": 4, "price": [4, 8, 11, 13, 14, 17], "bid": [4, 5, 8, 11, 13, 14], "simpl": [4, 9, 14], "minim": [4, 7, 11], "optim": [4, 5, 13, 14], "harddriv": 4, "s3f": [4, 9, 11, 15, 17], "egress": 4, "charg": [4, 14], "rang": [4, 15], "travers": 4, "region": [4, 5, 6, 10, 15], "zone": 4, "08": [4, 12], "12": [4, 5, 10, 12], "gb": [4, 8, 11], "ipv4": [4, 11], "address": [4, 10, 11], "005": 4, "ip": [4, 10, 11], "februari": 4, "2024": 4, "unless": [4, 9, 14, 15], "specif": [4, 5, 6, 7, 8, 9, 11, 13, 15, 17], "encourag": [4, 13], "dure": [4, 10, 11, 15], "down": [4, 5, 8, 12, 13, 14], "cancel": [4, 16, 17], "aggress": 4, "downscal": [4, 14, 15, 17], "engag": [4, 14], "cheapest": [4, 17], "cleanup": 4, "without": [4, 5, 10, 13, 14, 15], "abort": 4, "clean": [4, 5, 11, 14], "purg": 4, "consol": [4, 5, 8, 10, 16], "press": [4, 9], "action": [4, 6], "longer": [4, 5, 11, 14], "unus": 4, "state": [4, 10, 17], "insuffici": [4, 5], "hygien": 4, "50": [4, 14], "prevent": [4, 5, 11], "choos": [4, 13, 14], "customiz": 5, "page": [5, 7, 8, 12, 13], "resourc": [5, 6, 7, 8, 11, 16], "full": [5, 6, 8, 11], "widget": 5, "report": [5, 9, 15], "separ": [5, 7, 8, 12, 15], "auto": [5, 10, 11], "ds": [5, 8, 10], "popul": 5, "helfpul": 5, "post": 5, "mortem": 5, "previou": [5, 8, 11, 12, 13, 14, 15, 17], "consist": [5, 15], "manner": [5, 15], "told": [5, 9], "thu": 5, "fulfil": 5, "given": [5, 8], "point": [5, 8, 9, 14, 15], "spin": 5, "hope": [5, 12], "straight": 5, "steadi": 5, "dip": [5, 14], "inevit": 5, "replac": [5, 6, 9, 14], "give": [5, 6, 7, 8, 11], "higher": [5, 11, 13], "bidder": 5, "progress": [5, 11, 13, 15], "lost": 5, "reach": [5, 15], "For": [5, 11, 12, 14], "much": [5, 8, 11, 13, 15], "impact": 5, "frustrat": 5, "potenti": 5, "expens": [5, 17], "lot": 5, "abl": [5, 11, 15], "futur": [5, 10, 12], "blip": 5, "coincid": 5, "outag": [5, 11], "bad": [5, 11, 15], "luck": 5, "exampl": [5, 8, 12, 14, 15, 17], "abov": [5, 6, 10, 14, 15], "bulk": 5, "numberofmessagesdelet": 5, "well": [5, 6, 7, 8, 9, 12, 13, 15, 17], "back": [5, 8], "track": [5, 9], "top": [5, 10, 12, 16], "gap": 5, "fraction": 5, "insight": 5, "encount": [5, 15], "smallest": 5, "economi": 5, "sake": 5, "util": [5, 8], "intens": [5, 8, 11, 12], "averag": [5, 11], "minut": [5, 8, 13, 14], "window": 5, "max": [5, 15], "percentag": 5, "less": [5, 17], "aka": [5, 9, 14], "messagesnotvis": 5, "whichev": 5, "linear": 5, "declin": 5, "total": [5, 11, 15], "equal": 5, "core": [5, 11], "sometim": [5, 15], "rapid": 5, "increas": [5, 8, 9, 12, 15, 17], "pictur": 5, "disk": [5, 8, 14], "rapidli": [5, 15], "caught": 5, "reboot": [5, 8], "lose": 5, "appear": 5, "loss": 5, "were": [5, 11, 13], "return": [5, 12, 16, 17], "statu": 5, "rel": [5, 9], "slope": 5, "graph": 5, "discov": 5, "midwai": 5, "7": 5, "greater": [5, 8], "fewer": [5, 15], "within": [5, 6, 8, 11, 15], "plot": [5, 11], "its": [5, 7, 8, 13, 14], "ideal": 5, "softwar": [5, 7, 8, 9, 10, 11], "entri": [5, 12], "toggl": 5, "row": [5, 9, 12, 15], "logstream": 5, "timescal": 5, "absolut": [5, 9], "upper": 5, "corner": 5, "screen": [5, 14], "zoom": 5, "particular": [5, 12, 15], "draw": 5, "box": 5, "itself": [5, 14, 15], "deselect": 5, "certain": [5, 13, 15, 17], "perman": 5, "simpli": [5, 9, 11, 12, 13], "click": [5, 13], "x": 5, "next": [5, 11, 13], "legend": 5, "hover": 5, "left": 5, "direct": [5, 6], "arrow": 5, "icon": 5, "drag": 5, "drop": [5, 15], "lower": 5, "diagon": 5, "desir": [5, 11, 17], "menu": 5, "refresh": 5, "ident": [6, 15], "access": [6, 7, 10, 11, 15, 17], "iam": [6, 16], "would": [6, 8, 9, 10, 11, 12, 14], "cli": [6, 11, 16], "upload": [6, 10, 11, 15, 17], "ssh": [6, 10], "role": [6, 13, 15, 16, 17], "acl": 6, "owner": 6, "metadata": [6, 12, 15, 17], "wil": 6, "further": [6, 10, 15], "someon": 6, "els": [6, 14], "handl": [6, 15, 17], "assum": [6, 10, 13], "learn": 6, "defin": [6, 8], "trust": 6, "relationship": 6, "templat": [6, 13], "2012": 6, "17": 6, "statement": 6, "princip": 6, "arn": [6, 13, 16], "123456789123": 6, "image_analyst": 6, "image_expert": 6, "amazonaw": [6, 15], "com": [6, 10, 15], "st": 6, "assumerol": 6, "node": [6, 12], "block": [6, 10], "bottom": [6, 9], "role_arn": 6, "source_profil": 6, "my": [6, 10], "credenti": [6, 10], "fill": 6, "info": 6, "aws_access_key_id": 6, "access_kei": 6, "aws_secret_access_kei": 6, "secret_access_kei": 6, "amazon": [7, 10, 13], "web": [7, 10, 13, 14], "storag": [7, 10, 13], "store": [7, 10], "cloud": [7, 9], "platform": [7, 10], "sourc": [7, 15], "oper": [7, 10, 15, 16], "benefit": 7, "eas": 7, "themselv": [7, 8], "reproduc": 7, "result": [7, 11, 13, 14], "fix": [7, 15], "flexibl": 7, "demand": [7, 11], "infrastructur": [7, 14], "far": 7, "institut": 7, "great": 7, "piec": 7, "ad": [7, 14, 17], "consum": 7, "confus": 7, "leverag": 7, "former": 7, "latter": 7, "essenti": [7, 12], "termin": [7, 10, 11, 13, 14], "program": [7, 13], "feel": [7, 13], "cimini": 7, "broad": 7, "cambridg": 7, "ma": 7, "usa": 7, "develop": 7, "now": [7, 14, 16], "carpent": 7, "singh": 7, "repositori": [8, 11], "readm": 8, "respect": [8, 11], "document": [8, 10, 11, 13, 15], "overview": 8, "explain": 8, "python3": 8, "gave": 8, "addition": [8, 14], "onto": [8, 11, 13, 14], "mismatch": [8, 15, 17], "accident": [8, 15], "intend": 8, "blindli": 8, "sit": 8, "idl": 8, "hook": 8, "_perinst": 8, "shut": [8, 14], "send": [8, 11, 12], "16xlarg": [8, 11], "250gb": 8, "ram": 8, "165": [8, 11], "16": [8, 11], "10gb": 8, "space": [8, 11, 15], "assign": [8, 11], "share": 8, "among": [8, 13], "384": [8, 15], "timelaps": 8, "movi": 8, "stagger": 8, "aren": 8, "simultan": 8, "likelihood": 8, "degre": 8, "trial": 8, "local": [8, 9, 10, 13, 15], "sens": [8, 15], "drive": [8, 10, 11], "fluctuat": 8, "choic": [8, 12], "quickli": [8, 11, 13, 15], "willing": [8, 11, 13], "oppos": [8, 11], "ones": [8, 11], "risk": [8, 14], "tweak": 8, "five": 8, "primari": 8, "interact": [8, 10], "prepar": [8, 15], "outsid": 8, "granular": 8, "tab": 8, "browser": 8, "watch": 8, "behavior": 8, "especi": 8, "split": 9, "unlik": 9, "inclus": 9, "NOT": [9, 12, 16], "wish": [9, 13], "metdata": 9, "ye": 9, "enabl": [9, 10], "extract": [9, 12], "folder": [9, 11, 12, 15, 17], "present": [9, 11, 13, 17], "remot": 9, "method": 9, "should": [9, 10, 11, 13, 14], "pars": [9, 11], "minimum": [9, 11], "pathname_": 9, "nameofchannel": 9, "filename_": 9, "column": [9, 12, 13, 15], "metadata_": [9, 15], "pieceofmetadata": 9, "pathnam": [9, 15], "relativepath": 9, "odd": 9, "yourself": [9, 12], "via": [9, 17], "favorit": [9, 12], "languag": [9, 12], "phenix": 9, "xml": 9, "pe2loaddata": [9, 15], "namesandtyp": 9, "written": [9, 10], "video": 9, "hardcod": 9, "nest": [9, 15], "eweisbar": 9, "desktop": 9, "run_work": 9, "sh": [9, 12, 15], "bypass": [9, 15, 17], "data_fil": [9, 12], "run_batch_gener": [9, 15, 17], "stuff": 9, "argument": [9, 15], "makeanalysisjob": [9, 15], "deviat": 9, "datafilepath": 9, "transit": 9, "h5": [9, 12, 15], "captur": 9, "togeth": [9, 11], "createbatchfil": [9, 12, 15], "map": 9, "root": [9, 11, 13], "analyz": [9, 12], "button": 9, "batchpipenam": 9, "batchpath": 9, "format": [9, 12], "implement": 10, "destroi": 10, "creation": [10, 16], "setup_aw": 10, "proce": [10, 11], "login": 10, "secur": [10, 13, 16], "safe": [10, 11], "later": 10, "probabl": 10, "d": [10, 11, 12], "rather": [10, 15, 17], "pair": [10, 13, 15], "chmod": 10, "600": 10, "privat": 10, "connect": [10, 11, 15], "vpc": [10, 13, 16], "subnet": [10, 13, 14, 16], "inbound": 10, "lambda_funct": 10, "bucket_nam": 10, "quot": 10, "ecsinstancerol": [10, 15, 16], "permiss": [10, 11, 15], "coordin": 10, "grant": 10, "launch": [10, 11, 13], "topic": 10, "trigger": [10, 14, 15], "link": [10, 13], "prefer": [10, 11], "dedic": 10, "simplic": 10, "special": 10, "micro": 10, "basic": 10, "guid": 10, "tool": [10, 12], "instruct": [10, 13, 14], "linux": [10, 13, 16], "support": [10, 11, 12, 17], "9": 10, "sudo": 10, "apt": 10, "git": [10, 11], "http": 10, "github": [10, 15], "distributedsci": 10, "cd": 10, "interfac": [10, 13, 14], "main": 10, "awscli": 10, "ignor": 10, "six": 10, "upgrad": 10, "last": [10, 14, 15], "west": 10, "eu": 10, "2a": 10, "real": 10, "gotten": 10, "know": [10, 15], "replic": 10, "repeat": 10, "decid": [10, 12], "specifi": [11, 12, 15], "tie": 11, "uniqu": 11, "apart": [11, 13], "analys": 11, "nuclearsegmentation_drosophila": 11, "encapsul": 11, "prep": 11, "write": 11, "extern": 11, "load_data": [11, 15], "distinct": 11, "wrong": [11, 15], "nuclearsegmentation_hela": 11, "handi": [11, 13], "histori": [11, 13], "tracker": [11, 13], "immedi": [11, 13], "justifi": [11, 13], "temporari": 11, "associ": [11, 14], "occasion": 11, "crop": 11, "unabl": 11, "perhap": 11, "whatev": [11, 14], "troubleshoot": 11, "mostli": 11, "obviat": 11, "slower": 11, "advis": 11, "length": [11, 15], "due": [11, 12, 14, 15], "lack": [11, 17], "slightli": 11, "stall": [11, 14, 15], "sinc": 11, "exist": [11, 13, 15, 16], "fact": 11, "proceed": [11, 15], "insensit": 11, "partwai": 11, "resubmit": 11, "whole": [11, 14, 15], "reprocess": [11, 15], "haven": [11, 15], "succeed": 11, "vs": [11, 15], "rerun": 11, "determin": 11, "previous": [11, 14], "mark": 11, "byte": 11, "object": 11, "smaller": [11, 12, 15], "string": 11, "toward": 11, "load": [11, 12, 15], "miss": [11, 15, 16], "harder": 11, "succesffulli": 11, "suggest": 11, "conjunct": 11, "pass": [11, 15], "directori": [11, 12, 15], "v1": 11, "active_plugin": 11, "revert": 11, "fetch": 11, "commit": 11, "checkout": 11, "deprec": [11, 13], "pip": 11, "older": [11, 14], "clone": 11, "newer": 11, "fairli": 11, "4096": 11, "mb": 11, "break": 12, "site": [12, 15, 17], "readi": [12, 13], "yourjobfil": 12, "loaddata": [12, 15], "OR": [12, 15], "headless": 12, "versu": 12, "mechan": 12, "000": 12, "suffer": 12, "throughput": 12, "none": 12, "rule": 12, "flagimag": [12, 15], "scan": 12, "output_structur": [12, 17], "subfold": 12, "hyphen": [12, 17], "metadata_pl": [12, 15], "plate1": [12, 15], "metadata_wel": [12, 15], "a01": [12, 15], "output_top_directori": [12, 17], "append": 12, "gnu": 12, "combin": 12, "approach": 12, "uniform": 12, "96": 12, "echo": 12, "plate2": 12, "plate3": 12, "b": 12, "h": 12, "02": 12, "03": 12, "04": [12, 15], "05": 12, "06": 12, "07": 12, "09": 12, "11": [12, 15], "sort": 12, "variabl": [12, 15, 17], "besid": 12, "serv": 12, "shortcut": 12, "common": [12, 15], "regular": 12, "n": 12, "pathwai": 12, "comfort": 12, "faster": 12, "yourfleetfil": 13, "anywher": 13, "coupl": 13, "sever": 13, "app_namespotfleetrequestid": [13, 14], "walk": 13, "awai": 13, "ask": 13, "shortli": 13, "term": 13, "explan": 13, "cover": 13, "review": 13, "test": 13, "ami": [13, 16], "obtain": 13, "id": 13, "iamfleetrol": 13, "iaminstanceprofil": 13, "keynam": [13, 15], "subnetid": 13, "did": 13, "extens": [13, 15], "imageid": 13, "snapshotid": 13, "refer": [13, 15], "os": 13, "yourselv": 13, "good": 13, "had": 13, "pleas": [13, 15, 16], "repo": [13, 15], "offici": 13, "strongli": 13, "validfrom": 13, "validto": 13, "year": 13, "targetcapac": 13, "align": 13, "step_1_configur": 13, "md": 13, "instancetyp": 13, "spotpric": 13, "search": 13, "kill": [13, 14], "network": [13, 14], "On": 13, "snapshot": 13, "label": 14, "statist": 14, "relat": 14, "consecut": 14, "almost": 14, "show": [14, 15], "soon": 14, "tmux": 14, "disconnect": 14, "critic": 14, "lambda": 14, "forc": 14, "24": 14, "rise": 14, "over": [14, 17], "rid": 14, "final": 14, "multi": [14, 15], "inher": 14, "overal": 14, "alloc": 14, "normal": 14, "monitor_fleet_id": 14, "sfr": 14, "9999ef99": 14, "99fc": 14, "9d9d": 14, "9999": 14, "9999999e99ab": 14, "monitor_app_nam": 14, "2021_12_13_project_analysi": 14, "monitor_ecs_clust": 14, "monitor_queue_nam": 14, "2021_12_13_project_analysisqueu": 14, "monitor_bucket_nam": 14, "monitor_log_group_nam": 14, "monitor_start_tim": 14, "1649187798951": 14, "troubl": 14, "kept": 14, "solut": 15, "beyond": 15, "throw": 15, "bug": 15, "expected_number_of_fil": 15, "6": 15, "batch_data": 15, "uncheck": 15, "valueerror": 15, "dictionari": 15, "sequenc": 15, "element": 15, "syntax": 15, "consid": 15, "Or": 15, "patient": 15, "seen": 15, "sqs_queue_inform": 15, "invalid": [15, 16], "routin": 15, "unrecover": 15, "No": 15, "extra": 15, "comma": 15, "somewher": 15, "invis": 15, "excel": 15, "panda": 15, "to_csv": 15, "index": 15, "datafram": 15, "indexerror": 15, "bound": 15, "axi": 15, "lead": 15, "zero": 15, "ie": [15, 17], "metadata_sit": 15, "isn": 15, "pad": 15, "batchfil": 15, "mistak": 15, "exporttospreadsheet": 15, "sub": 15, "unusu": 15, "criteria": 15, "cppipe": 15, "outcom": 15, "typo": 15, "a1": 15, "attent": 15, "reflect": 15, "perinst": 15, "ioerror": 15, "indic": 15, "nameless": 15, "regist": 15, "cannot": 15, "err": 15, "slowdown": 15, "putobject": 15, "rate": 15, "preval": 15, "backlog": 15, "transport": 15, "endpoint": 15, "ssl": 15, "certif": 15, "subject": 15, "target": 15, "host": 15, "xxx": 15, "yyi": 15, "dot": 15, "use_path_request_styl": 15, "silent": 15, "mito": 15, "referenc": 15, "filenam": 15, "arrai": 15, "reduct": 15, "header": 15, "went": 15, "hint": 15, "field": 15, "mykeypair": 15, "examplefleet": 15, "With": 15, "examplejob_plateid": 15, "filterobject": 15, "saveimag": 15, "botocor": 16, "except": 16, "clienterror": 16, "invalidspotfleetrequestconfig": 16, "requestspotfleet": 16, "spotfleetrequestconfig": 16, "spotfleetrequestconfigurationinvalid": 16, "0f161e6034a6262d8": 16, "unix": 16, "fleetfil": 16, "012345678901": 16, "instanceprofil": 16, "One": 16, "valid": 16, "sg": 16, "01234567890123451atest": 16, "0123456789012345": 16, "dockerhub": 17, "dcpversion_cellprofilervers": 17, "0_3": 17, "8": 17, "versionnumb": 17, "plugin": 17, "abil": 17, "boto": 17, "fabric": 17, "elimin": 17, "check_if_don": 17, "cheapli": 17, "slowli": 17, "launchdata": 17, "submiss": 17, "discount": 17, "compat": 17, "item": 17, "discuss": 17}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"aw": [0, 5, 6, 8, 10, 11], "hygien": 0, "script": 0, "clean": 0, "out": [0, 13], "old": 0, "alarm": 0, "log": [0, 5, 11], "group": [0, 11, 16], "sq": [1, 11], "queue": [1, 11, 14], "inform": [1, 11], "sqs_queue_nam": 1, "sqs_dead_letter_queu": 1, "sqs_message_vis": 1, "job_retri": 1, "exampl": [1, 3, 6, 11], "advanc": 2, "configur": [2, 3, 8, 10, 11, 12, 13], "dcp": [2, 6, 9, 13], "chang": 2, "you": 2, "can": [2, 7], "make": 2, "distribut": [2, 4, 7, 8], "cellprofil": [2, 4, 7, 8, 11], "outsid": 2, "docker": [2, 7, 11], "contain": 2, "requir": [2, 10], "your": [2, 6, 12, 13, 14], "own": 2, "pipelin": 2, "us": [2, 6, 7, 9, 14], "runcellpos": 2, "plugin": [2, 11], "config": [3, 6, 11, 13], "py": [3, 12], "what": [4, 7, 8], "doe": [4, 8], "cost": 4, "cloudwatch": [5, 11], "dashboard": [5, 11], "fulfilledcapac": 5, "numberofmessagesreceiv": 5, "delet": 5, "memoryutil": 5, "messagesvis": 5, "notvis": 5, "distinct": 5, "all": 5, "show": 5, "error": 5, "interact": 5, "extern": 6, "bucket": 6, "setup": 6, "read": 6, "from": [6, 10], "cell": 6, "paint": 6, "galleri": 6, "write": 6, "collabor": 6, "s": 6, "permiss": 6, "In": 6, "account": 6, "instanc": [6, 8, 11], "why": 7, "would": 7, "i": [7, 8], "want": 7, "thi": [7, 8, 10], "do": [7, 8], "need": 7, "have": 7, "run": [7, 8, 11, 13, 14], "contribut": 7, "code": 7, "who": 7, "made": 7, "happen": 8, "when": [8, 14], "an": 8, "look": 8, "like": 8, "how": 8, "determin": 8, "my": 8, "pass": 9, "file": [9, 11, 12, 14], "metadata": 9, "load": 9, "data": 9, "creat": [9, 10], "loaddata": 9, "csv": 9, "batch": 9, "list": 9, "step": [10, 11, 12, 13, 14], "0": [10, 17], "prep": 10, "1": [10, 11, 13, 17], "manual": [10, 14], "resourc": 10, "2": [10, 12, 17], "automat": 10, "3": [10, 13], "auxiliari": 10, "4": [10, 14], "increas": 10, "spot": [10, 13], "limit": 10, "The": 10, "control": 10, "node": 10, "scratch": 10, "instal": 10, "python": 10, "8": 10, "higher": 10, "pip": 10, "clone": 10, "repositori": 10, "cli": 10, "s3f": 10, "fuse": 10, "option": 10, "ami": 10, "remov": 10, "long": 10, "term": 10, "infrastructur": 10, "role": 10, "lambda": 10, "monitor": [10, 11, 14], "sn": 10, "ec2": [10, 11], "compon": 11, "gener": 11, "set": 11, "AND": 11, "ec": 11, "environ": 11, "creation": 11, "redund": 11, "check": 11, "submit": 12, "job": 12, "overview": 12, "altern": 12, "submiss": 12, "run_batch_gener": 12, "start": 13, "cluster": 13, "fleet": 13, "request": 13, "paramet": 13, "must": 13, "current": 13, "version": [13, 17], "To": 13, "region": 13, "where": 13, "isn": 13, "t": 13, "avail": 13, "date": 13, "auto": 14, "function": 14, "while": 14, "analysi": 14, "total": 14, "empti": 14, "ar": 14, "visibl": 14, "Not": 14, "messag": 14, "cheapest": 14, "mode": 14, "troubleshoot": [15, 16], "startclust": 16, "iamfleetrol": 16, "iaminstanceprofil": 16, "subnetid": 16, "histori": 17, "forthcom": 17, "0rc2": 17, "releas": 17, "20201110": 17, "0rc1": 17, "20201105": 17, "20201103": 17, "20200109": 17, "updat": 17, "through": 17, "20191002": 17, "20181108": 17, "20170217": 17, "20170221": 17, "bugfix": 17, "20181018": 17, "20170213": 17}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 6, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinx": 56}}) \ No newline at end of file diff --git a/step_1_configuration.html b/step_1_configuration.html index 2ad1c86..e5ee78c 100644 --- a/step_1_configuration.html +++ b/step_1_configuration.html @@ -486,12 +486,19 @@

    EC2 AND ECS INFORMATION

    MACHINE_TYPE: A list of what type(s) of machines your spot fleet should contain.

  2. MACHINE_PRICE: How much you’re willing to pay per hour for each machine launched. AWS has a handy price history tracker you can use to make a reasonable estimate of how much to bid. -If your jobs complete quickly and/or you don’t need the data immediately you can reduce your bid accordingly; jobs that may take many hours to finish or that you need results from immediately may justify a higher bid.

    3. +If your jobs complete quickly and/or you don’t need the data immediately you can reduce your bid accordingly; jobs that may take many hours to finish or that you need results from immediately may justify a higher bid. +See also AWS on-demand pricing to compare the cost savings of using spot fleets.

  3. EBS_VOL_SIZE: The size of the temporary hard drive associated with each EC2 instance in GB. The minimum allowed is 22. If you have multiple Dockers running per machine, each Docker will have access to (EBS_VOL_SIZE/TASKS_PER_MACHINE)- 2 GB of space.

  4. DOWNLOAD_FILES: Whether or not to download the image files to the EBS volume before processing, as opposed to accessing them all from S3FS. -This typically requires a larger EBS volume (depending on the size of your image sets, and how many sets are processed per group), but avoids occasional issues with S3FS that can crop up on longer runs.

    5. +This typically requires a larger EBS volume (depending on the size of your image sets, and how many sets are processed per group), but avoids occasional issues with S3FS that can crop up on longer runs. +By default, DCP uses S3FS to mount the S3 SOURCE_BUCKET as a pseudo-file system on each EC2 instance in your spot fleet to avoid file download. +If you are unable to mount the SOURCE_BUCKET (perhaps because of a permissions issue) you should proceed with DOWNLOAD_FILES = 'True'.

    +

  5. ASSIGN_IP: Whether or not to assign an a public IPv4 address to each instance in the spot fleet. +If set to ‘False’ will overwrite whatever is in the Fleet file. +If set to ‘True’ will respect whatever is in the Fleet file. +Distributed-CellProfiler originally defaulted to assign an IP address to each instance so that one could connect to the instance for troubleshooting but that need has been mostly obviated by the level of logging currently in DCP.

    6. @@ -596,7 +603,7 @@

    PLUGINS

    EXAMPLE CONFIGURATIONS#

    -

    !(Sample_Distributed-CellProfiler_Configuration_1)[images/sample_DCP_config_1.png]

    +

    Sample_Distributed-CellProfiler_Configuration_1

    This is an example of one possible configuration. It’s a fairly large machine that is able to process 64 jobs at the same time.

    The Config settings for this example are:
    @@ -606,7 +613,7 @@

    EXAMPLE CONFIGURATIONSDOCKER_CORES = 4 (copies of CellProfiler to run inside a docker)
    CPU_SHARES = 4096 (number of cores for each Docker * 1024)
    MEMORY = 15000 (MB for each Docker)

    -

    !(Sample_Distributed-CellProfiler_Configuration_2)[images/sample_DCP_config_2.png]

    +

    Sample_Distributed-CellProfiler_Configuration_2

    This is an example of another possible configuration. When we run Distributed CellProfiler we tend to prefer running a larger number of smaller machine. This is an example of a configuration we often use.