From b04cc80071e7eb0d52155d1c99a485a87e86e158 Mon Sep 17 00:00:00 2001 From: mayorao Date: Wed, 12 Oct 2022 15:58:26 -0700 Subject: [PATCH] Update docs for distributed training --- docs/source/index.rst | 7 +- docs/source/user_guide/ADSString/index.rst | 11 - docs/source/user_guide/apachespark/spark.rst | 6 +- .../user_guide/cli/opctl/localdev/vscode.rst | 2 +- .../source/user_guide/data_labeling/index.rst | 6 +- .../data_transformation.rst | 12 +- .../data_visualization/visualization.rst | 6 +- .../user_guide/loading_data/connect.rst | 6 +- .../model_registration/introduction.rst | 6 +- .../distributed_training/_profiling.rst | 28 + .../distributed_training/_save_artifacts.rst | 25 + .../distributed_training/_test_and_submit.rst | 45 ++ .../configuration/configuration.rst | 1 + .../distributed_training/dask/creating.rst | 69 +- .../developer/developer.rst | 41 +- .../developer/figures/horovod.png | Bin 0 -> 53038 bytes .../distributed_training/horovod/creating.rst | 97 +-- .../distributed_training/horovod/horovod.rst | 2 - .../distributed_training/overview.rst | 7 +- .../distributed_training/pytorch/creating.rst | 106 ++-- .../distributed_training/pytorch/pytorch.rst | 2 - .../remote_source_code.rst | 4 +- .../tensorflow/creating.rst | 600 ++++++++++++++++++ .../tensorflow/tensorflow.rst | 18 + .../distributed_training/troubleshooting.rst | 9 + .../user_guide/model_training/index.rst | 7 +- docs/source/user_guide/secrets/index.rst | 6 +- .../text_extraction/text_dataset.rst | 18 +- 28 files changed, 907 insertions(+), 240 deletions(-) create mode 100644 docs/source/user_guide/model_training/distributed_training/_profiling.rst create mode 100644 docs/source/user_guide/model_training/distributed_training/_save_artifacts.rst create mode 100644 docs/source/user_guide/model_training/distributed_training/_test_and_submit.rst create mode 100644 docs/source/user_guide/model_training/distributed_training/developer/figures/horovod.png create mode 100644 docs/source/user_guide/model_training/distributed_training/tensorflow/creating.rst create mode 100644 docs/source/user_guide/model_training/distributed_training/tensorflow/tensorflow.rst create mode 100644 docs/source/user_guide/model_training/distributed_training/troubleshooting.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index e7e2ff7c6..292bae037 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,16 +12,16 @@ Oracle Accelerated Data Science SDK (ADS) .. toctree:: :hidden: :maxdepth: 5 - :caption: History: + :caption: Getting Started: release_notes + user_guide/quick_start/quick_start .. toctree:: :hidden: :maxdepth: 5 :caption: Installation and Configuration: - user_guide/quick_start/quick_start user_guide/cli/quickstart user_guide/cli/authentication user_guide/cli/opctl/configure @@ -33,19 +33,18 @@ Oracle Accelerated Data Science SDK (ADS) :caption: Tasks: user_guide/loading_data/connect - user_guide/apachespark/spark user_guide/data_labeling/index user_guide/data_transformation/data_transformation user_guide/data_visualization/visualization user_guide/model_training/index user_guide/model_registration/introduction - user_guide/ADSString/index .. toctree:: :hidden: :maxdepth: 5 :caption: Integrations: + user_guide/apachespark/spark user_guide/big_data_service/index user_guide/jobs/index user_guide/logs/logs diff --git a/docs/source/user_guide/ADSString/index.rst b/docs/source/user_guide/ADSString/index.rst index fb308c89a..17a85c9aa 100644 --- a/docs/source/user_guide/ADSString/index.rst +++ b/docs/source/user_guide/ADSString/index.rst @@ -1,10 +1,5 @@ .. _ADSString: -###################### -Manipulating Text Data -###################### - - TextStrings ----------- @@ -18,10 +13,4 @@ TextStrings regex_match still_a_string -Text Extraction ---------------- - -.. toctree:: - :maxdepth: 1 - ../text_extraction/text_dataset diff --git a/docs/source/user_guide/apachespark/spark.rst b/docs/source/user_guide/apachespark/spark.rst index 62dc9dcde..465054670 100644 --- a/docs/source/user_guide/apachespark/spark.rst +++ b/docs/source/user_guide/apachespark/spark.rst @@ -1,6 +1,6 @@ -========================= -Working with Apache Spark -========================= +============ +Apache Spark +============ .. admonition:: DataFlow diff --git a/docs/source/user_guide/cli/opctl/localdev/vscode.rst b/docs/source/user_guide/cli/opctl/localdev/vscode.rst index 29f36496f..62e1a0472 100644 --- a/docs/source/user_guide/cli/opctl/localdev/vscode.rst +++ b/docs/source/user_guide/cli/opctl/localdev/vscode.rst @@ -6,7 +6,7 @@ Setting up Visual Studio Code **Prerequisites** -1. ADS CLI is :doc:`configured` +1. ADS CLI is :doc:`configured<../configure>` 2. Install Visual Studio Code 3. :doc:`Build Development Container Image` 4. Install Visual Studio Code extension for `Remote Development `_ diff --git a/docs/source/user_guide/data_labeling/index.rst b/docs/source/user_guide/data_labeling/index.rst index 97df2ad7f..ed6777c85 100644 --- a/docs/source/user_guide/data_labeling/index.rst +++ b/docs/source/user_guide/data_labeling/index.rst @@ -1,8 +1,8 @@ .. _data-labeling-8: -############# -Labeling Data -############# +########## +Label Data +########## The Oracle Cloud Infrastructure (OCI) Data Labeling service allows you to create and browse datasets, view data records (text, images) and apply labels for the purposes of building AI/machine learning (ML) models. The service also provides interactive user interfaces that enable the labeling process. After you label records, you can export the dataset as line-delimited JSON Lines (JSONL) for use in model development. diff --git a/docs/source/user_guide/data_transformation/data_transformation.rst b/docs/source/user_guide/data_transformation/data_transformation.rst index 4ab5627da..0162ed9c5 100644 --- a/docs/source/user_guide/data_transformation/data_transformation.rst +++ b/docs/source/user_guide/data_transformation/data_transformation.rst @@ -1,7 +1,7 @@ .. _data-transformations-8: -Data Transformations -#################### +Transform Data +############## When datasets are loaded with DatasetFactory, they can be transformed and manipulated easily with the built-in functions. Underlying, an ``ADSDataset`` object is a Pandas dataframe. Any operation that can be performed to a `Pandas dataframe `_ can also be applied to an ADS Dataset. @@ -520,3 +520,11 @@ You can split the dataset right after the ``DatasetFactory.open()`` statement: ds = DatasetFactory.open("path/data.csv").set_target('target') train, test = ds.train_test_split(test_size=0.25) +Text Data +********* + +.. toctree:: + :maxdepth: 3 + + ../ADSString/index + ../text_extraction/text_dataset diff --git a/docs/source/user_guide/data_visualization/visualization.rst b/docs/source/user_guide/data_visualization/visualization.rst index 359620be8..47defde4c 100644 --- a/docs/source/user_guide/data_visualization/visualization.rst +++ b/docs/source/user_guide/data_visualization/visualization.rst @@ -1,8 +1,8 @@ .. _data-visualization-8: -################## -Data Visualization -################## +############## +Visualize Data +############## Data visualization is an important aspect of data exploration, analysis, and communication. Generally, visualization of the data is one of the first steps in any analysis. It allows the analysts to efficiently gain an understanding of the data and guides the exploratory data analysis (EDA) and the modeling process. diff --git a/docs/source/user_guide/loading_data/connect.rst b/docs/source/user_guide/loading_data/connect.rst index 6d7fe4703..03ec5006e 100644 --- a/docs/source/user_guide/loading_data/connect.rst +++ b/docs/source/user_guide/loading_data/connect.rst @@ -1,6 +1,6 @@ -############ -Loading Data -############ +######### +Load Data +######### Connecting to Data Sources diff --git a/docs/source/user_guide/model_registration/introduction.rst b/docs/source/user_guide/model_registration/introduction.rst index 9a80836b5..3f6dd37bc 100644 --- a/docs/source/user_guide/model_registration/introduction.rst +++ b/docs/source/user_guide/model_registration/introduction.rst @@ -1,8 +1,8 @@ .. _model-catalog-8: -################################# -Model Registration and Deployment -################################# +########################## +Register and Deploy Models +########################## You could register your model with OCI Data Science service through ADS. Alternatively, the Oracle Cloud Infrastructure (OCI) Console can be used by going to the Data Science projects page, selecting a project, then click **Models**. The models page shows the model artifacts that are in the model catalog for a given project. diff --git a/docs/source/user_guide/model_training/distributed_training/_profiling.rst b/docs/source/user_guide/model_training/distributed_training/_profiling.rst new file mode 100644 index 000000000..d202e0343 --- /dev/null +++ b/docs/source/user_guide/model_training/distributed_training/_profiling.rst @@ -0,0 +1,28 @@ + +**Profiling using Nvidia Nsights** + + +`Nvidia Nsights `__. is a system wide profiling tool from Nvidia that can be used to profile Deep Learning workloads. + +Nsights requires no change in your training code. This works on process level. You can enable this experimental feature(highlighted in bold) in your training setup via the following configuration in the runtime yaml file. + + +.. code-block:: bash + + - name: PROFILE + value: 1 + - name: PROFILE_CMD + value: ""nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s none -o /opt/ml/nsight_report -x true"" + + +Refer `this `__ for nsys profile command options. You can modify the command within the ``PROFILE_CMD`` but remember this is all experimental. The profiling reports are generated per node. You need to download the reports to your computer manually or via the oci command. + +.. code-block:: bash + + oci os object bulk-download \ + -ns \ + -bn \ + --download-dir /path/on/your/computer \ + --prefix path/on/bucket/ + +To view the reports, you would need to install Nsight Systems app from `here `_. Thereafter, open the downloaded reports in the Nsight Systems app. \ No newline at end of file diff --git a/docs/source/user_guide/model_training/distributed_training/_save_artifacts.rst b/docs/source/user_guide/model_training/distributed_training/_save_artifacts.rst new file mode 100644 index 000000000..165631200 --- /dev/null +++ b/docs/source/user_guide/model_training/distributed_training/_save_artifacts.rst @@ -0,0 +1,25 @@ + +**Saving Artifacts to Object Storage Buckets** + + +In case you want to save the artifacts generated by the training process (model checkpoints, TensorBoard logs, etc.) to an object bucket +you can use the 'sync' feature. The environment variable ``OCI__SYNC_DIR`` exposes the directory location that will be automatically synchronized +to the configured object storage bucket location. Use this directory in your training script to save the artifacts. + +To configure the destination object storage bucket location, use the following settings in the workload yaml file(train.yaml). + +.. code-block:: bash + + - name: SYNC_ARTIFACTS + value: 1 + - name: WORKSPACE + value: "" + - name: WORKSPACE_PREFIX + value: "" + +**Note**: Change ``SYNC_ARTIFACTS`` to ``0`` to disable this feature. +Use ``OCI__SYNC_DIR`` env variable in your code to save the artifacts. For Example : + + + + diff --git a/docs/source/user_guide/model_training/distributed_training/_test_and_submit.rst b/docs/source/user_guide/model_training/distributed_training/_test_and_submit.rst new file mode 100644 index 000000000..e8b8278cd --- /dev/null +++ b/docs/source/user_guide/model_training/distributed_training/_test_and_submit.rst @@ -0,0 +1,45 @@ +**Test Locally:** + +Before submitting the workload to jobs, you can run it locally to test your code, dependencies, configurations etc. +With ``-b local`` flag, it uses a local backend. Further when you need to run this workload on odsc jobs, simply use ``-b job`` flag instead. + +.. code-block:: bash + + ads opctl run -f train.yaml -b local + +If your code requires to use any oci services (like object bucket), you need to mount oci keys from your local host machine onto the container. This is already done for you assuming the typical location of oci keys ``~/.oci``. You can modify it though, in-case you have keys at a different location. You need to do this in the config.ini file. + +.. code-block:: bash + + oci_key_mnt = ~/.oci:/home/oci_dist_training/.oci + +**Submit the workload:** + + + +.. code-block:: bash + + ads opctl run -f train.yaml -b job + +**Note:**: This will automatically push the docker image to the +OCI `container registry repo `_ . + +Once running, you will see on the terminal an output similar to the below. Note that this yaml +can be used as input to ``ads opctl distributed-training show-config -f `` - to both +save and see the run info use ``tee`` - for example: + +.. code-block:: bash + + ads opctl run -f train.yaml | tee info.yaml + +.. code-block:: yaml + :caption: info.yaml + + jobId: oci.xxxx. + mainJobRunId: + mainJobRunIdName: oci.xxxx. + workDir: oci://my-bucket@my-namespace/daskcluster-testing/005 + otherJobRunIds: + - workerJobRunIdName_1: oci.xxxx. + - workerJobRunIdName_2: oci.xxxx. + - workerJobRunIdName_3: oci.xxxx. \ No newline at end of file diff --git a/docs/source/user_guide/model_training/distributed_training/configuration/configuration.rst b/docs/source/user_guide/model_training/distributed_training/configuration/configuration.rst index 1de07915e..6d275ecc2 100644 --- a/docs/source/user_guide/model_training/distributed_training/configuration/configuration.rst +++ b/docs/source/user_guide/model_training/distributed_training/configuration/configuration.rst @@ -17,6 +17,7 @@ You need to use a private subnet for distributed training and configure the secu * `PyTorch`: By default, ``PyTorch`` uses **29400**. * `Horovod`: allow TCP traffic on all ports within the subnet. +* `Tensorflow`: Worker Port: Allow traffic from all source ports to one worker port (default: 12345). If changed, provide this in train.yaml config. See also: `Security Lists `_ diff --git a/docs/source/user_guide/model_training/distributed_training/dask/creating.rst b/docs/source/user_guide/model_training/distributed_training/dask/creating.rst index 8ffb2247a..2f541ef4b 100644 --- a/docs/source/user_guide/model_training/distributed_training/dask/creating.rst +++ b/docs/source/user_guide/model_training/distributed_training/dask/creating.rst @@ -73,46 +73,37 @@ example. Now running the command below Before you can build the image, you must set the following environment variables: +Specify image name and tag + .. code-block:: bash export IMAGE_NAME= export TAG=latest -To build the image: - - -`without Proxy server:` +Build the container image. .. code-block:: bash - docker build -t $IMAGE_NAME:$TAG \ - -f oci_dist_training_artifacts/dask/v1/Dockerfile . + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/dask/v1/Dockerfile -`with Proxy server:` -.. code-block:: bash - - docker build --build-arg no_proxy=$no_proxy \ - --build-arg http_proxy=$http_proxy \ - --build-arg https_proxy=$http_proxy \ - -t $IMAGE_NAME:$TAG \ - -f oci_dist_training_artifacts/dask/v1/Dockerfile . -The code is assumed to be in the current working directory. To override the source code directory: +The code is assumed to be in the current working directory. To override the source code directory, use the ``-s`` flag and specify the code dir. This folder should be within the current working directory. .. code-block:: bash - docker build --build-arg CODE_DIR=`pwd` \ - -t $IMAGE_NAME:$TAG \ - -f oci_dist_training_artifacts/dask/v1/Dockerfile + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/dask/v1/Dockerfile + -s +If you are behind proxy, ads opctl will automatically use your proxy settings (defined via ``no_proxy``, ``http_proxy`` and ``https_proxy``). -Finally, push your image using: - -.. code-block:: bash - - docker push $IMAGE_NAME:$TAG **Define your workload yaml:** @@ -229,31 +220,7 @@ This will give an option similar to this - OCI__MODE:WORKER -----------------------------Ending dryrun mode---------------------------------- -Submit the workload - - -.. code-block:: bash - - ads opctl run -f train.yaml - -Once running you will see on the terminal an output similar to the contents of the ``info.yaml`` below. To both -save and see the run info use ``tee`` - for example: - -.. code-block:: bash - - ads opctl run -f train.yaml | tee info.yaml - -.. code-block:: yaml - :caption: info.yaml - - jobId: oci.xxxx. - mainJobRunId: oci.xxxx. - workDir: oci://my-bucket@my-namespace/daskcluster-testing/005 - workerJobRunIds: - - oci.xxxx. - - oci.xxxx. - - oci.xxxx. - -It is recommended that you save the output to a file. +.. include:: ../_test_and_submit.rst **Monitoring the workload logs** @@ -286,6 +253,12 @@ The alternate approach is to use either a Bastion host on the same subnet as the For more information about the dashboard, checkout https://docs.dask.org/en/stable/diagnostics-distributed.html +.. include:: ../_save_artifacts.rst +.. code-block:: python + + with open(os.path.join(os.environ.get("OCI__SYNC_DIR"),"results.txt"), "w") as rf: + rf.write(f"Best Params are: {grid.best_params_}, Score is {grid.best_score_}") + **Terminating In-Progress Cluster** To terminate a running cluster, you could run - diff --git a/docs/source/user_guide/model_training/distributed_training/developer/developer.rst b/docs/source/user_guide/model_training/distributed_training/developer/developer.rst index ad1a78b78..f2064bcce 100644 --- a/docs/source/user_guide/model_training/distributed_training/developer/developer.rst +++ b/docs/source/user_guide/model_training/distributed_training/developer/developer.rst @@ -18,7 +18,7 @@ Build Image **Args** -* -t: Tag of the docker image +* -t: Tag of the docker image * -reg: Docker Repository * -df: Dockerfile using which docker will be build * -push: push the image to oci registry @@ -27,10 +27,10 @@ Build Image .. code-block:: bash - ads opctl distributed-training build-image - -t $TAG - -reg $NAME_OF_REGISTRY - -df $PATH_TO_DOCKERFILE + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $NAME_OF_REGISTRY \ + -df $PATH_TO_DOCKERFILE \ -s $MOUNT_FOLDER_PATH @@ -92,10 +92,10 @@ Run the container Image on the OCI Data Science or local * -i: Auto increments the tag of the image * -nopush: Doesn't Push the latest image to OCIR * -nobuild: Doesn't build the image -* -t: Tag of the docker image -* -reg: Docker Repository -* -df: Dockerfile using which docker will be build -* -s: source code dir +* -t: Tag of the docker image +* -reg: Docker Repository +* -df: Dockerfile using which docker will be build +* -s: source code dir **Note** : The value "@image" for ``image`` attribute in ``train.yaml`` is replaced at runtime using combination of ``-t`` and ``-r`` params. @@ -128,7 +128,7 @@ If required OCI API keys can be mounted by specifying the location in the config -Developement Flow +Development Flow ----------------- **Step 1**: @@ -179,3 +179,24 @@ Finally, to run on a jobs platform ads opctl run -f train.yaml + +Diagnosing Infrastructure Setup +------------------------------- + +Before submitting your code to Data Science Jobs, check if the infra setup meets the framework requirement. Each framework has a specific set of requirements. + +``ads opctl check`` runs diagnosis by starting a single node ``jobrun`` using the container image specified in the ``train.yaml`` file. + +.. code-block:: bash + + ads opctl check -f train.yaml --output infra_report.html + +The `train.yaml` is the same yaml file that is defined for running distributed training code. The diagnostic report is saved in the file provided in ``--output`` option. + +Here is a sample report generated for Horovod cluster - + +.. figure:: figures/horovod.png + + + + diff --git a/docs/source/user_guide/model_training/distributed_training/developer/figures/horovod.png b/docs/source/user_guide/model_training/distributed_training/developer/figures/horovod.png new file mode 100644 index 0000000000000000000000000000000000000000..449e150009645f967d19e18ef9d42d181e3d1031 GIT binary patch literal 53038 zcmeFZby!tl`!#q3u?T|{1S~{Ex?4p-y1PL-B@P`1pi(Lz4bt7sp#&8rrAs;wE#1t$ z^?kpYx#sWhx@NBVoqt3)oW1w++_CPp*27zQS#iP(lowDa6rse^$BHP_>3S3jNATPk z_zrn?nh+ToFe@;P{UoO|{L{(sf+iMpebt%;+vfxR)x)W+7@n90G&-q_g2!OYfi z6}L_p-o%8wNz~rhz|q{+=DM=EwJ}QB$@n@O+jTi3$Ls8@?CjUsSot})`MEf*%gbDs zf2=Hb$Dl~+z&31k}mxym9IwEXP-Mak7s`htCCWF_w=I=4~G2SA0CfKc-#;OwUXg(T-q$H^?z4RO}$^&_1_2GdDNaBGn$tm;C#;wkTcqtgWpT<@xpRx-ZuA z@$sFm`}d>f7x2cJ>FF>huIlfWr&wd!W^+Yx|NgALG3Q##A5GNK?@EVJe$IdI7i?`V zyq1qwUmdAQ@b5b`CNt{6{W$vn{^!GT9fd43yrgdZ&sYDm2LGRm1N{4o7wVmJbTFzN zLOZcKdsMd7iG8_R5f6h#!|3VGYwy)>P@^aQW0$jC_cjLY)J zLR&pbLq*0dPmF^oxw`ymz0}3Ha8PrXQILaojf+h?`jB_SzeCnia&l^JZdL0W8!ewr zAiw$^qaX!8lfdmb@AuaotMv84cPmmZp*E7fygIu${^b?(G8G9)$_3JUcR4wg6%-V* zbZe>%4Gl@i$a=^aR5t9aPt~)To10Uz>idX?U{_c3fBlm8_4B)T|Gu=-VxP*? z_n_VkgpHujqFvuAg?xKsqt zOBH7B($azaP75iP`8A(B$WYBS)zyM=d=6r!?Xe$jFAo-rOMSZk;r5d!Pcqf=br*)q z^DW?JKJ}jd&22j>I5Rs-X|wn%@zqd?dGgwtow13@JplnsnB`d%N?+%6eRJmY^fNko z`o$RAT7_)YTu!rJ*UO1}d8`KHjlKsKGKK^MI4oB9ung6D9z6M(nvYrGC#2$TP4r|` z&d?uFUFgeQ%v_kjp_hyA;TiKt=5v@e8?S#+H5DvCarsPoMfa$X^+=VI-%O+bMYXY7 zPpL|7Sl9?6w@YLNt{W!SqtzPA!{z-2293%!ZcY)wN~w~hbhmH2RG@uqhe|CYS`PLP z4!TvF>j#TXxs1LOscDNzF?RZA=>EZKU3f%<+T`>8Je_Q0Z!^>572#_k4dZ%y zk5o}By6}4{T{97SD_ynRF)~q6QT*s7^$E_W92n=-zTyty?6<`9u%oT1JB_K|)P@c+ zmC^~)MatlE1Qrl!cIG!ws%OMPXmTaP@9`bF#k8fpI(yOLV0&4!+$Ih(z%*LqHY20d zR8vTQg7BrW>$a{G1N+m@$N($XxB_P6%8`t?d-p&I*_Mr*We z>yFBFT_*6#$1e8GF)j<;y46@vZPFH_5S!(%tU%x)4yK~2)Et)G@(6uO- zR>Wqf%DCkdc7HM7S>WVGPFuv#0qg-`i|C_Al9hLmO{6&~%0tDHU%i?suu>t)VbUsZ zXqbduG--`YOqGiGr2Eas-moKHh)9M}Y!HdSJ2KHJ5SmwQPS#skq$2@9fN> zA@%BqE2i!>H8eDAPKU}Z44NjW_+B6-(Bm-gx=k(YQ4wZ&W|vzpiU_Z%Cr3jGVu|N> znQ;qQYI=HveUNZF5 z&rMI8t&Z04+fTodjTdBqNH{$7z&m@Ek)2(U%c4gf_O~Y5@enWpzvKLSA{xOA4%7C# z92`o6gM$}{h?oy^Aic674KQ0egEj8uXDydCASB&)R(dBI{hbziG8-oKj8Ak^@mQBIbl2TGi^78MKl9NekX>A9%Jt}M5MRj%K zKZb@n&HuVVE#jrEp`p>-)}~@-R{&d2LQS1JFrXfx7KnZV#ThaxP1K+0r%7=pWD>G5 zlw20d7EX6%;WLoMg3FMX9lcz%*;?rBO_3mmeebkf)f*jFm+<&=*!(iTqz&mPJ1ZlS z5)ulqM~6p8q?DAJE(zq+)XTj)XMKtuqgrqBJFnzG9k7JuL=UUx>*mVDaEC|In=TKP zx~){sEj5q1Oc6%dFIkP%>PST}_pZ;hghxkfPA=l&;`UZNH`>B>i$jUq7-*YC@)?rE za;lc&Q#lGC$R4MYU`A*Rp2u-3bhNmO18lF3v_xa=#W7z$EJ@1+k%hlQgn-PZ&A-cZ042QXPh#!dn zGN!IHnV5G+*I+gOKK}Bs+6czEF2(+=)6zhv9j9Dp88#3ij!Cy#9a%5f$0eGZBr2JT zbl;kq7Nafm6w+nm-syDqx4u44T=MMa$AD%eYR?nj(N^9bwyncLiSGZzr1dUW?eXK+ zRZfd(TDCPR5jyCo?kzU^r2#f3^@0a25iI@3M+Z_Kp8EQq^C~7U=s5MNEDRQ>KN1y< z^Xh=2Y|)paK_2de?Rt_d9>RRM1EqIs)P418pLuf_gT+8WV$(22V5Me9*I&$_?s$LE z4G_t4Gptf}hwwqp*P{(60T#XYj-Ub}m7AS|L+Up@Bn-6xS{V{SN3(_tQk7@KF;r)- z?s$vk+qZ8L{A0DSIBBjMvkwWgE$G!Hw+J;HMR z`U6<|ex~x@NsRf`U8!h2Wmg-c;JZN^2dmyE*=l;8y?Gw>KBwFuV5qZ9C4>)_I6Sv{ z)zC8?A}7_bG^&%w#+}?j)3dV{e$&oT9%`H3$$kWc{Bn<1Wo705REWrC6Rr1rsGM*ov7F#UUq&!Y zG;NV=qGMhMxuSb|va!60`E|#dBFDSQ@h+3+bsXBbe1`*V0;Q_>IELb*qB0K;4;y0U zAd;}qxOk-3tZ(!2X~47AnsmhHLhO^F*<+#orqg;KDnLc)iQ+Jg9dbT7J{qd?uB+W& zV1y>LPf@Gn_{rD;bCDZDJi$VegD2t*hJ&7KV-rAZoPp&B{dZr#4~EQOX;@1X@0aj6%q}1 zgv88<*TFKg^Dho|_6xXV!Kc6CBRjA$+lF9QBm$tTV`pP*RUbU?X>4M>@XXZMnDqFh zIh>ITz(hjFoXQY{V>*LU8gJbE$?l&i-xu}V)S$nOSyuVV7VXd^=k)RTBL$6`n?D ztsa-=yfT@m`i;1{y2YMMrMCy4REFtT&0!iW7JMy} zF|Xp$N5S=-(>_IFM6CFKHa5C(FAC~cSOYX53Tm+L&r-?Wgq0AWKD@WEiS35QT0QE0 z;u*2s@$+Y=6iZDCM92`9s0JY?GCW~7_em8e??AR@PgbZfM+o92y^=H<#e8i^RSX zycOvmmv4!Y=rUe2=4_+CbxX8sZ@j--PGq@@i<5^(Ex~I)a}=rjwHJ|gD|%5Vh1P5L zOJ=a80#w&%ey1#CtpH5;?{Ms6NKVru*G7uw@6zYC%@BpK+R))g zH;2H6M~_}kut~ZtSd@(b?yi4<>$Y6lzhrYsR8*9dlr$9?qX1?!h0yZUTxB}fQ?oPb zP9$gsaboL7Jvt>>372O#9lQX$BHd4WJ~A+x!jx1^uu(W&roDuC8uSrk%#?PhpljJ3HZy`T-a? ztxx|nOz*r8oe{oS1;KbNE5uP)4%w}MxI4Pps!T8-iXx9}W0MDuya_QI9b7+LWE_CR zn?&dxKF+wziuOGSoomtjjts7gx#cBhoj1Fep{KtK2vCAa89Ja7WK*1=t0KfM*P%|~ zpcT^G!eLGl_T1-H_6MAXv9w2{%ZO-&bD%npRubRCd-(7nQUhv__boBju;r>V&EYUW z<--z^Qn?I5eHPxECKlSCPbR7>?;Tdxh|>dfPq<3|iCG6(^-47R$(Z-afnMXfZjBo$ zCV{0y27B1p6`|{vInYt+b+nHJ7~s6HPoL6Y`qASw=Ax$;N9G{RYONHCz=!SIr6_fPLf=DZZ7B{Rrn;8P$MUlCYlm11P2l?oQe&yzZCH~Sghk3@)~VtZ zJQQGH__-;e;pkZ6M{qSsHjaNV=*}L%i7jYt3ozj_w2E>WH3L!av>`_Pf=RB^+|qPdwU-=QZCPZ$41#dDWL*56x2(dPk;eK5ZQ^Q8^V?5By5wcoavs z)U{HNJ^Su)wHR*eu*k@a?d4%T&Tn@honm-w`<4caO92*Oj%GU&@`0H6of$K={yl^6 z>0CZrup2xf!7Dx3>8c4os-Ap3kw!tyx%Rkjz-XJWyp+N1(L(NJ2%3ap_sXLFW?XY~ zb60~eKC*ocMF2~)_3FG}N{t(F%ocLrHtS4!vOnQZOCTVv=Y3QP3(}J!5qf1Y6t);D zF%?=HK(fXFQ+o)tfJy)5 zpto-ofbxJyL3pC+A0HYzpVsz_0%&wo&>_0U`#Dk6f2hW0cWuIN?{zDG?3l+!koj2w zW@cHaBKa^bner;~!Q6{L#K5Z@1mfeh8|?y+91imsBa+rOXj3^F#mrT-=gyz+gh4+K zCKv`05k25j7<5QNYN*w1i)C@L@erDY-<8Eky7X7JLI-M%;@IkvVhL9ddCBlVESkqQ z@6S|I%o(#R<;*1a)fzPbm!-P zc?S&$I2oibQ3|=00IOuR6OD&Y#!G_If`A89W~4>~tT-SQEF=aq`UJQ-Y{JQ5y4&V_ zqG`PA_mvG{n1K;uK%B-kWniim<_(*#dXNjcfM%~aFsNk6KZJP;YV2}}+m=NTt;lGe z(ZSwa!Z6)8r`vBWA3S)FnUfa@Q3XUs|JqT4$GU`$`*Nw@fv>MG!;Kpc&!0cP57#Cj zHu8pH9~wd^ovkANmGe{p^eRQ05Fn|Vqi&$4n=D2MpI$p~-5sErW|>tc3>p?dI=TV# zw~X^eDNS0zpNLvgJGXa!=zUzAXQ!!V*G(#*yp$XL2zzVXorQLOJD{mMV3G%vD+64V zW}(xEcD>L*vMX*$b+(DSO)_4xCIj*iZ};q`g+#Ydl@@scvChPlMC zSEUVEaWiR_WUSeae13>aNX=g|K$O8!x|8I@rZvNS_wJ90{oC~PmH>7AhHWs*+(#P; zC!B=JPv^v?bs_K;hDx)M`U3Z$Qz8U_1)+w7cgJ5{@tzCB@k^IW=tF>dh3Bu&4;T9L z6V%i$p&tWTDDKrWXx{ccj<rnL`70s$RZKnJgc5vwXX1UCOPh ztPu|!T7uj5QYKI;>q+a#L=7_%Bt&@Q-19_On0bIY=$UUq(`%_^8U-j-g;S?aQ3{}Q z%=I4fpHw11d7(4eKi+d&1K1Qxs5W%Wyr)q!r;Ie1GV~>|NnPFDPSDx&tLK0SA#6zi zLiCEjD?C6Vn@hzVl;~0KgXQvoE2YsIr5526@3f!^cSo`rU5mbX2`C33j@+R=w$H z8Z(IFp&9`g2P)khW>HE`Gx?~qbtxO>xagJN4A{fwcAy;!%wbK~*x1tcRDy5_o@mqr z?KSM_RG4-828!A7HQ(KI*#9I zbHFGB$kJQi@O}Wby9v!uz;VOVATCgq8tFg?7qV0fJU835#Qs`OcOWquC-_M+^y+H! z>W&To6Ma8i$Fp!{8TlLSMKeGJ~KO)=#FiH}0xEb(l7TswwnfG_c-{s`wG<>*B zUN!lSF;@56I*gkf1oQ0)B4acAds#XdG!_P6w%_QfARc~_KwT(t6P#&4xM4U+7h&T$ zuW1zMe|c%>FdlZBn_Cs>0!jxZ*28^{)BYMnF*qPFaNn22DtOtk z*DkQ?Sdpv~jg)XG-2IuJRnDt6kqwCz+Ar~PM!$PZ9D)$hQ(~@!S%)dsaHA_45TD=j zePQ8In=GqSdKNJ-Irwp$0Vb{&dVcx>%$QC{49YI!qG|(Ckx!zW=BqDLfQb9+)kWb9 zC^#~}Lgyl^j93sQU&-T=%f%la&;>pF(bm=r>9yS4*w`o~t!^<5Yz|xX`Hk0FHkIEt z?_}{pNn^dwBPFj_f1h>Uihlf2`3O43U@pCaKF@?vI9dkZ{wt6^2-dJz`CYjW`@A(| zneV#n;LzWdDnC>m#Uw3PgfIl|gjZyl2*BX}G1dD@& zroVWi&RsUP&Qy0*)Y_A2RPo*7l$=rZC)vOZ63O*Hf2^$?Yv3x;Jp?Ih1lo<4p03#b^9%a;>j`tkMm z2R^zh@!Hec`+y+t+Dmu+WtnAmwDJ(HYtRIz)U8S(9VPBz>2_@zO0P?HM?1fNvBWha!U~axO4|#$h+d&PJ(h8eth4P)SDj+x*5ejrB_ce-5 zGPTNWK6y7(0KtwaB4g655X{Z?JlIBl1Jk=685Tx@;G9%X2M34n4<8bHdXyUly`CU^ zD^bWOGUyvr7RMze0j1WHrz3zd_Zu#=cEGwYX_Yy;a;88%2AE~cgNL+Dggu+syq?_q zl5QuT5HLm$shYL{d9|S1W(K0=@P_B`LYS0JHxRW$J!L=c;fm9%ms?TN#jGKE02g7*Ql9HP?+=RdGzRgI6 zc?+g6j_!2xJUi2#aE;F?6{wNX8h64LDsHQJd!Y-FL46!S}GJS&Sk?wNld7Ic)xHuMzwV|U%KL${D<|55fPewQN@PTm9I5HqBZTi!p z%c4&e0D3{-O4_aGZ51)UN0u;QA3qvWpg7#k#@{l0eg6EpU-Ax27~_s}hwm8k>@dQr zb}+&>dWgmESGSIKlL80kkjy>W4YYs64%fjE%o1!PO$*ko{q3tz3JdXl5Mg$)Ki`@6 zZcJF%GZ#4(?dt;(Lngpp8t9H9QV!=pGw^?3o<*Le38K4cn)VM?{I z>263X&WHGO4chl8xver_5bD4(^oz+g#w}luW{V+HQPdT_g@?~y(Y&N-iniv=G6KzI z_!imija4T}$#>mN_H#G>&e4mD8Ih5KP-5SH%hrKDBv!*l*wouQ zx97XZ@?l=}x@4oq>2|?Lw&-K^eBBzuRXop3>$J2qN-z468uC1aWIwGv%~SPitT^zH ze$x@NGFX5N$fOH6UkV_07fd4vO)+T2-DsRNVjOm>b!=>G+1x3fNInd$OH%web7ho3 zVs@{*_2Y=tT*4dSiGQfwkqYBQplt5vQF>UHxL$xq-bu6|O3 z|Abj8)+%QyOF$;~I`!+!*&km?wrqqcYo=?GBA8v%sr2@W7kTSTH5fjlF`)OD`MCdd;a1#%RNxjnp!8bpFK3kzBFv)$rn@r?~i z08BZ(k3HJr1j@gvPzmLlt(!{FNSs;8{BRRbTjs~=z2f$KdNJH+>f{s;YlLHjU)cQ9 zL`zOZAOpe{Y#-Tu_3PL}rG?#H7X--XYL?oYb0rrUmdU@!-m7EAyCoL*i>qIIsw?Xa zIeQhii%gb$bYIz5m*4`RcMBwT$vmYpqf|<)HPZdlXPW zo12d5>!N$>PaT}oeE3=*-qkwtO*`TtqX1au>R`&gR(}Akr<6*F6!=ZcyvnqJBPh7> zd=B3yj9ijvQ1YW;38S}S4Y{uk$0$hv@XQ3cr-IYCfjAI{A^se5*bc2nvhPKxb%OS~ z$H&JYSQWLrnEa7+3RJ#+C}xSkeVGHtNK~U|Z(jrwBA03VwI`1sBUDKhHZA6NJijVs zD~|rm(ScJU^gnFlMG;50Iv7*UfGkGOh#YN(PIPUz|M>ay&-_iYV0+^)mz@Qq+NaNP zwrEXft|l|?eQZ_F460~c5|~-;XdPTFVmVtc8H((1yk4*f=hfl~!I_C6Z&pj{g1`oYkgFGDv7WgM70to$0IprAG|MasZz z7#bN7o;dg_pjwGX=As&LIMz zS$AG_3O3}UZK;fHpRXkwic$Y`B%bw9$AZ>K9iwNr8vacDe9JH0^vgz!>jsoDJ4ktb}jeS3g-taH5`#%q}Ex`CE|<}4}# z6dhm#2~e`p*a4G=Yj8V@KuV#pk->I!I{}!=%WZioVsT`)-n4BCit%Yo2b9GhRO0tr zTVLXOnB*0@QEkj@EP{3j48?N=4d6u)axqT_L%E{l$fYOoE-W@mXb0Hz#W1XKDNub{w^^O<+k~q0pm=}Y}xx` zmP-dP0PIfNkC-Ij@`TMH8dce_O*haBt_tv3-5`#p!oc2=y7*?$nGF<0sDgCjM0bc!%bmaK9=?dZK;k6t?uon5$wJha zR3k74QF+mN{#jkInQO>-WmLBNR-wp3?ntNF)4GBr3Qt^UE9z>{B+4lF1rz$o#_j>8Whw0lLI`X;W2D!v<$%qt!EPAf z>B72m8Gt;rG15)i!WC7437qXaM*er72jAx^9U9@)#E+Rx5>$W|84=!2h5o*h>1!IO z8j=UdJTq{->|S;`o1SnT!kpWjE`Wf6LC0iAGZ`BK?hrLdqkj)>vO~Etik|=#JC#)I zRDC~i@$W(Wmr-&-T7L22$jb#S-RSj zsht1c!xa>;LIwDl=KTk9YoKSK!83>NIEz8gB}iwiIN$@6FnDCj?UYjOGN0&AaH0w{eOI;X zMyznW#-#yb>-x~`)v?vSW0PdUR{lVf{xiwpzhX;4gvGFfQ4LH4m7YsdDlm5tr1Q8= zZm?rMfvs(9*+{P$nK+LQ1JfSn>D3)4VP|0+D5jz&rJ$Ja@hoO3Z3A~-qe2}(D|^dl zqk!nON6Aqh=YfxjWH)|SW;JM7VZAo~g_)j?6Utc>Qht(B4Wf&TSo^^ zbsrcc5$BcP0jKp)Pr>gD9f2v38HWmu{B>L*I%PJxKR@&G@&e8Gt4Y#AH zg!Fx;{;kO5)YP&f`!txX;O)LA+|4n=&{Vq3gTy>?Vdi_<(%K3G6h=F*33-}-i&9|E z%)%nKTD=kMb@)fTaTyt^Ecp#30#(T!=s#=#H=YXOZp5~APsWD{#U3gi8)azg-Az(b zQr&<7b+Q6j&Jz$QgUqS32Nb0AtF&NVyKx_bt^weaqwlI zq+l`W(x>|xlk03SW|kr(=Q<#h1TBWrYNYRNZQ!3fcUN0w{}JAv<;KPap@UzQ{n(O} z^#;s8toNO_Zb(^JWR1LwhGqw*$|gl6r7d7(7oa$?|E2-!ABgBKr9@&sxx2f&WkT7= zO@UgM$koZu%kzsBh&q;~o+KA+kiP-<2eB3kiA;>%X8?bGZTkE?-;A8UOI=&@gti}kP4U*iz>)c!03(Z zkVTb_9a~WTwZM2IyLuK?`v#@X=Lqu<;#(2*BVb!L!#i`vqBW8YG1paVk{6r%w+sJ) zvKjL|@MS1n=j+aYz^EhV)!K}N0wSP*TM;yQ%W?2 zYMwS$QMV9LO5z1IK%Zi|-|~+?Bvcb~P*zm@*e!?MSsl}Jsi~3&qaZe;5yx|XlZ}AA zJ({Z}HTD_o7ms#Z5kT%VCAON}oFNPk%yU22oSDI|0RSMeyGa20AGR?NfeYBFFYyRk zPtb!hu)OYv=voImt30iI!^z?o8#frBJzk2=MO4TAW;%4y6Zg@K94}M;uD&=)npu3w34#!T8%SBx;g4-q?cpmew zZ!Y_G`_p)=J#3tg2g#l34+EmA65758YZ2lC0oCTwrj8RrC8h_XxgffS`PbILjXmH%E!B$ zoibf2_OK1Coe|Af#!hydL%cvslD*=@o6aIu=&l~ zO^OcS_d2Z5b;Hs%ZihxjsvDx}AkaZlc(3CM21`UvY&35dLnLyD!mD|YwG$uVjeurH zMBPNyTvX%U2u=@6if202E``EJn@Zrs;&Y5em@rDNX1h)E1YuA0f_hP?>&=aOSA%f0 zYTQZNTk+1@oRR& z1BFJ^l2>et2g86`GinOv6ZGH*+b!6l8B4D{fK6709A18@19JxuCXt9;0HEM-&{GNU zmVl3B6ZFfur0^C*(uVK>>1r6yxIJDd8$4TPAU&ta&EeqSY}VsmG=m&P4^xTM7%MGTNu)mTF6Zq2%-MTZviX|BwMRvW2}Ydu02Ul@Bx0w0lqwXG6OY$lEWnM zKW0&iGqg~NOMz1JJUQA)+slG`Ah2a;+3fH>RsvDGJjQDiWb1q2#^f1wNd}hQwkj#z z(W`k1f+P)!Bvmj9vE#h`G8P$0f%z>5eolDq?&J_*O^W&-&CQB1L4y{c0>Nw;g3Y+6 zk~I&mw{9Tjm&%xQs*sYcROfZ%*sG$3EoxzAtX2h3V~!MdsJYS(&k3%we`TeWawUTU z5r9=6K_M*}E(2tYcqP+ri3C*b&Bl}hzQ%kg2F&2LTC)Q-&nqL(o2Ws92{O381G@uC zdM4tPXbhGBxr_$*>$4q8hS2DdKt(F9v9U2?gX#gtPd)(dtHGnKV9EGMC$$ORr8r{` z3?i4^_yhmZ1Bl~(D6q>{Zqq`}BWe~BIK*GMp~e}{`Gmi~S4;Slf{4i>LfesA-L|na zaGb<_rBWQJO(SJc8o|pk+6Cx58#Wm*RI^Dj1DqIvcq$)RG%_?qG&Cvxx}q_1Jr9GO z=qCYW=CB&?z*_;TLYivgNVfxGa>Hw5V93;eKUlw2O z?D4@WVzea`Z$J#ah>HrKt}-~4p#NZDW#ME>7|<~RZHs-NKsBPCC;-dR0}5IuqF>ky zBd!Ksna?nc{B?=LunTlBkbZCwWdUqYX}4%oH!AD=F4F263W3j7C-U^pBLp(PzadI; z?b>}2k-COz2MFH{Gye*xtw;kj7)^tW#)7A(5A3Qr;AMXzCI-(RPH`FxCHaW#64Kil z0UjwaC}CZYdCO&<;GjiMYoTSsKmx@b0PRWPoJCUur+#YwzIT)I{xU2KFdK$OD#QwmmKz#(A=`pGYbJJmh~q z`5HJb4J+NR09g5B@mW||IP!zIg8P63 z4-XGFc3Xr~BmOzT-FPHHTgrNHk6GR0RXte~e!6AdU+?3{YW81}^g( zIWPv;s-Ao_0$F5nPOa2JrPO+OJIP%n`7n`A=&QqPu*Cxc6avGM+ia96!7F7()tp~Y z@kqc82ges+ERq1LO&2(*m&8c?{rnbSxZ`PEoRwZUgIdD_t>iOMaaCZs1y!Jl-Sq@U zGjrIVw70Wh2h*BtHUT8sy_jF8%d3(KQ3r<;J|IrHogs)>a@mXQQp#>n*5EkKU=Z3~ z8T#rbXzi6@XB#5mG{GViwmul|WYjeHJ$BcgB(F<;eER-dtV9rgBJG0B(;n zZaXq6%ADf!U668|I1L(3H`2F0flFr(n12cD(RcCg05%3APs z^L+4nc-df;lO@y+Y4E{v$4w^0#px`qN&%aX91MaCZ($9&bBYx;L&yhbyWn7p)9R=W z$R0h&1x>!>)~q$) z|K<3Yv)A(gM7Uv7BRD;F?HW}CsiA$S1W~X(f@5L|z-uWTjM_C4B8P{tiX-jkh=}^Y z^||%sEG@+yPp-*!WiC0yN(=|-wm>=K)m}}4p)U)7w+3R7OMRu+wbxCpm?A=I!|A%} z)-3BBAC?3?aGn%iT!2C_-M9Ksp|*>gn3#AAW-tsVKafVzi05h|>!a8niIip{HOeEX z)URAU5TFTCB%pLK+`NgnOOPCe%OE8`fii9ys`m|1zYM^#`{4R2fzL6Fhzfc0<`)o) znNVM2@@{M7>k1(V8gXGFR(EKtcflYDQGyssAhOdzL?qG5hY7-SADcc@<&+KP8Xjw) zec||>GB{G9G{FIvEEsdMpj{&_D5$8O3e2Y^(&i@G+jKMlS1188=mkXc_gq8Kv$tN5 zj8_t-@$m36VbvE#YcLRiv#Re4;1k@yd4sqK5l{frf`dz#6nM6xxvjGRWoAK}V8)XK zaX1sI#{!^>duZQI*t7h+(?h_AA~+X0h63dm2LCie^3!WOJ8DoV@I^$jb9U$7fERTh z0-zJf>*X@U0c=g{G}ib?tR6 zASO2$HcDWw1^Gk`rX1!nUEnI1S|{dg&N2fLi}Y;d_yY`SdJ+;6M7f`-K=&r1=Fb5A z0x>dbIjx=1GhZG3Rqg5^6UVP&*_Q)-KcEDH%P{?&)SNdiAmSXbmm(^DGPO5J2NEWr z2|#VBa>T-0$*$Ve0e>iT91pPq?HLi|WsoBhWX#&?lQpWK@Pp+u9ITq}-@k9LPrk1C z1xFR;>D?KDs-^JIKzqT7E`#F15{EfiIP0bk7#ulo_U#=QrAbqPc?8Cn@(yb238Pvb zXRI!gQW?NV8Oz^Vg)@A8a4yqvcg+Bbpd`?&>#)QS)d;$SRRA{0m22(q*#&?%P*4j1 z_JgSsaYjQAKuUdLV&c1?AcSucR1JZ-&Kw4pAt3i;6NGcYFMyCU_A|{{V6=fkKJj7a zc^EK=(4s(Rx{;ci$^Zl&c)>5=<0AwFatbS5{P!0&@Dn1fRZ&q9@o)@Ul{K8b3|9hk zO&N$EBSi08WBI!6To1rT`IGf?)Q1lWyK%e@v(g4JZ@@eb3kI7%*L&Fq4$~qx6MT&? z2{b4;TYncCDUuN2@P!Xj!Jti8&%OVz7hu_*?LuciOo_2@WZ>t|pT3n;#l^+QaT9o0 zL@R=VQ%iOo_P##S3||7AwX^5uWnti&*wJ#<ZxMpEGdi3 z>3W|ltq?p=Zp#=G*o^$Z`lfP*3r9Ylj|w?ijFJph!PTEe;Yxzc(hCx15jh@w7lq34 zLs8skp{H-MJ8vcqhhTg2?lUvD+i_j{`_plE5BdQ9@l%Kr3J;6F@$?Anj&Rg860xs! zNk$1cuW(9_F1+om%qJQFcHqY&4?3a|bCMqr(4|Y5d{odVM~S{M`P0Bj?9d*|-DS2I-UGi%-t)^c$G8MVL@Fl)3(`1%ttb+xHnjP=kS|3_0NQVQK`t&Bz%8 z5)zW>4^w~N=F`=9j#XV1)NI5l3C>(gSPGEf!L9|bY$(N5%Voo0SSDuaz4CW4T&Pa5 zs_TM588JK{M-2d#hXZ)Y1rZWCKLgeYMc{*A@*t(5$pg&B1;?a8s8MVA!~{~gPG2va zs)m=%dP@f!K>7Xx&eho3UL*VaO_VNR;P9fnmul~E|MT}AK8Yd6&XL#ueD(jEHJB|$ zMFLtVCxSEo$m?RU{rx7(Qw4CG)|Q&`-}nCehd`jT{y+X)$UWtZ&oJ`*pMQ}=5jYkw zP$%YOPjKnVyaYXQp{?;he|H2ICKBa7m9=|yptX9~)=5D4f_(rvGOy?&@b6+IjUcB2 zxAW{|DUf=|L<6X1Sjp{lgoe!UqpL- z&RP7|1?0WLT~V!N>wH^Yc4fg*T`#bd1MV;TA~dXt%=YVDzBQ5WS=}B~h zL{_~MZX+aGec_7lEy@;Cz8lP)ggY2U^qtg7bHtCPawwT%E74z4<^AoBM7CF0$~qbA zR!`jfxUnwZbetJVZj^ty6yswNbK_E&q=FCwf5?ll-}uiJB5_HQhKbJJP0@Yx$}mIs zjo8hs7eA?<%MQO#Hn-#7+zOscE^I3@>Opo{nJ*T%m z7OK_ZOLBY^JDE-3W%SmqN8L&A9&?Cyp!uMe;4qK>Z7uFgxNb?`(7jUTihb%g|6Z}) zK{!gp=2<%Q5zHZ1>_jo>gx&+G>bgM#JH8iVqls>HlH@1Y4lO;SWrLlWZ%n3H9k=TF zt$HY3s%cOQrTnt_{$GyI;PJBAKKt>^?WCyEm~7PPluTc9jKG1q70E~G*V2IoxE!1h zvy{CBAGKzE4De+m#N{BlrPWT%;dhnDbP$ze7&tLu@F;31Ia@XKp`SnDq_K~ z5{f;0zYOzBgHcrGDjT=Txcmn+b!^$AyoW$r?BbUUrhqs0K)y zynJtYHrgvTjd|$Z=DV@@C&;B!6klZW!4_Y$R$!fO`4AUmK7z1Nu+wMAD*(( z!&(uoRH>Now+4Z_29N)}bl=J5O6TI^CUe|>cIE5TdBw`%)EE_=vzOWDl_`~oNwPCm<`cW|l9-Q=Gm6sHurMa4tmd()u>OA0tD!iCwDm2aKQ;R6gWJ!wGO|u}Rvshri`am+urlF5J0P zFj#PExtAb-i)A9?ZT7+Y(ijPQDmymESI#VhD5~Sat0p%i;xkLY>x{h84QL%FQDs?p zs#%0?l?*(I&-|p>-?7aFcRd*U44y$E2@*UNkv}q5Pc?63VPG z9(0LzxlPU?Slify-!W^@B2LmTGbWo;$Y7F#<0~!^w{W!aJtd(=y6|aM&lns&ipV>w z8iw~&P4AO8*=DnOFB9ce@_iRv@$xNAmYFe${a(xPUWQ_g`259Fgexo~ruZzKR8yw~ zebU>9+@0@|Wqqw7`&P|khF+}9{y2dgOGr}@F$MR{>Upn+B=pinaf0NB{aT5ZfJHoY4yr+%c+LM>!L(wMbqB)Kliuj|MtV| zQX^5^%DK*~EJXQl`KCXMMl~GlXyo_Zn5Q3i4%K!2;luOb$2A;Tnw)N%K66~%e2MGR zjhET~_*0Ir3r^`%xGLh+HsvL-zE)g zo1IeQ?x3r#`8cs%@j{0G)2vv)-}3cC!+}_uj5Ue0Ih-r2m$G-kQ#fmME?FSkW6i*C z{KnfenC=&)Wj_q>$Ry#9th751trUBgJB7SnyO^U?;}GQhDOP)Pj}6BlZ$IpKpoexpc~ahL5d)j1jg&S1fgmD+=ja_}H? z4rVxy9m`%Uz?`N;8|VMq+v)I$HHz1QXA;l-GMAUuHrhV+T*>J%zcQ+le6?8 zWakexZk?K5Z)I<3aZM9Hv!<(~&*vRrrWdMrmLQ0YZ{cO`L2HK^diDy2!v0hVcA#e0 zL)e2x+%}Ol1{c%q+j54}z5Bu`{>9!nygzPM7)x_5^c;Qv;P2mV&ErcIg3O--e2dBb zJ68a8b!3za(9f0z3g!_$@OVM>=@n!I-@ z+}gFqP~~IF^AwtfFefYXd3R#AxW^4|Td`07d4TzrNJDsTSSJSa(9^W!ZE0ogD#j~a zC4KwX_oti>O%fkuItD0^moUyKNq-KD$h1|uzDyKgGW?2d^1G0_X{mV=)t`x#opn9A z0wd{Q4#jw5YdTSzRynI5Z8L=}&&KLHBrV9CdqPU92DEs+OrorQ$kmI~AC+>=rn9R@ z805+idaEYTvg~N+#n4AtPg&+ET3!6LsFB`rjnKD2TeIq?8?!_g*>^RT_eW;b2lGsy zLOWQadPKg@eFIv5Z5@t;Lul3V&G?6(mD2mx#zorX?rk>ezj5ga2=c$5 zir@3%i~dJntU~q3g@9u73r^%Ou=g{6U1xo2cjss1*jp8*0(G--+K?h!{TPAP&7c#?K=eY{sH*k2@Es=m2-_yDmn) z^8+KzveeE_b(5qfu%+zVelxY|;_&Io?$gXeW_iydS%xU2nCzyC@P}tF1Te}hiM=kA zlC^5%vfty19$>S%?Yi?ZbY=g7v+7V}cc}JIlvnR5zwaC#Yk48HDR{0wYA+5c$4p|`r_?sdZrt_M!R_U9_i(OFTPtxRHy#4qc?Bbp9w<| z|K}$k9AuQP|EGFCp4RR@$NHZiT;}WlzrE}H|DGS{dTY=hgN~-xm-)Z$f^K(or?!9f zlJ13||I2+YH*9Zjqw8K>`tJ?8YH;F6|J#GdkK4KE>i_%ow6@a!o*$IDix=+T$3LsBIC`++#b<4yEuLG z46$z?7Hu2$^tFs%(A~X#{EM@Df-*XPJ5y}o^0Ia9fi;WmU~OB7cOO6Z-j?O7y0?4T zCt>}}o8M16)g2CM_%e9*>|EAYWs+lO>+m#<4RJ6yb3lE`r}W|T=O0q-!`&S4&uF_G z-Z7e*dhK`ZD0qdf>0CFyt`!X`U|sP=8Czo$$86}u>`&>fCa)8Gu?{LV^Y)zyB@QvU*NQ@*|Vg814zZjj^)899pXqv7DIJern zg$5~S=$c1R_|8+>Zg@0)2McCn8)C?gPxm@ZG~m&Am9?u4>(NwpIM*GOyR5DrQ=x

Db)v5cawBIB(QB^X+kTUl5e->6IwV!N4NyG3V1X3@c;Xmhu@VFYtu z?m#b2xU%C5y0-jctiB$tSz%eEaq0g-+gnD(wRL;mxC9FlT!Kq*Na2tG!GZ({?(Xh| zgy3$$DKtO`?pnCJyBF^6(v@@0bMMn{_kBl?(YNbE4F;|JZGx#s-M|D4Nbu9l#G zWGyB-TbZt^xE7I|V&$Vi0prA=OIB1ye(7IN`~4cdUUih5kN4y?;AQ9q7ADipcL>Y( zGT7`*YBSt@Dom9%UEw1f2tz5M6qp z8fYyjI!0(Z$BruZli@u%h1y^*aEtJG+GXQcV?GZR2R5n!Ce}@9JVQCp&irrV&_KCA zp&2Q)mIcUG>%F8PP8WN`%F~D{dGG2Rm;ij+iAyvPscgMiluAXYl@0cZf7j%)I-eC2 zact!uh>Kf~iL`7B?TzrT;_YyRxI4v2M}^?FPo+4wywHaPS{l}KY5^8sN+#gRmj*fZnnw?3u+4#jlttz$6EGAZ0vrG$ewxA6FV^8R>PV~N;~Z!&4q_Y*k<~njJ$9dy5+ra-NYkGqLo0)^i1R! zNVT;@SbVu~{-wC>sED8FyiclH_Fk)ddSs!0_9DTeS=Q$ERzOm{!<~p=EW(52if?;a^OM`$%F6Z% z#S1WX=4DU$#{`=uf^V68yab6L`6#37v7u8iVe~Cw^&%0sDB%d;N&=bR3Qu+Y+!$~5 zZQI+HilXEhrf#M$?@@qNvKmA+kaV|+>!jiK~)7!4s}1VxmwP6p4^Wo z&!>%S5>nw{=<8`2*zhQiU9t0%cR^s97HNpDCyGn2B<#tQ2yP{aE*czou?y+aY*OuV_!uXWJy*vlo zB+v)N4w@CR{+L=?I@Xx<*DDf3;(m^FA&@y6`dt;q9|D9B$gSH#mapJI$XC<6`yWK* z1Pi*vDryJyaB8TH(Ers%h$SL#44QJ2H}0gsx>Pj4 ziw}YLV#!5XByH}#;htn;>~4@n8rG9tHtG_j(SMjbfjLCK1EV{G5jw=cwQl{6#hc^O~h7cE&kqjPc39Z(pwMV zWqQhQvbYW6yy113!-ErMOR#`JtR**7hH)|OC-!)Rui6y4<-G3mqq`i!@g0m<&+?7+ z;6j=CtB~v}#)i-F`Jvf+&)J*9;hcKG`d><`!+7ZKr-u#YlUo6vM~E6({l+8MwiSlOAm5y4Yg;t8AXiDN=d)=?N9;`nVmrhmQF zWGq`lg9cbIaR+Dfy*}?${tSg)B1)89}JmjZa+?waEC-PT<+Q9+>OZB8< zE1?hdEl!-s@;i{qquO{#>deOirjyCqRdAHW;6u7ImN$(2(lLCb$I0Ch;1GZuwXAdK zdgx3cS0G$_LJG;7js3HP`nYcUN1xMMA2(jkuGyY2e4dhCD1GlCFT;8ir?2m?d=cBA zi8p%YC&kLvk20$au(e6cfxXjt+C}~zyRWk+t8_N^k2)oZH9keXP|evWSpkX8iYw~X z+hqy8^_ri5B97ZAfDP}_G!4PZv_PTZ`z=1(Q6lsOKSwyOh9>{uFZm%FWlDuVG<2k$ zY%1EEg+K4=n>rZ9-dd+*ryw390j2Xb>AJf7*2Q}BB%>Ehn*@8oL)epIPaKYU9DteQ zDotGzj#??!bXtNN2(RrT>eq{<_m8CX?xj`~lWuEx(&bzl&?)etT2oQu>y{5bb~Ie| z&@I~@W%%pv?^Iukd+fY!?ARZ9BpF|PDb7cOC%qsYR?e-D-vGt zh^X|nH`fQK^5|R4>KN!n1C*z6B_-FKA+)YVmygL@U&ap>dq^p|u@_F@t?ONQlA<>M>X zj!}y6)6$~x3p$Qm#K0qv%#Y?@`ga&-6{<>VefhJdb804I_!B?%3irhj(Isl@^O;Pr z;{O7bAF0dp*lPI?gOv}8`6FvFk`DCui^bj(P$vStRIRMg4&moK#*ex~2T@DzGw$uC zJ?gcF=`TT5_P@_F0t*_0N7uSqoT}CGqyCCOFmF3w9w=blpK5sOpRs&0?G{#$)D`9AWB@Ti2M^6vyMrvW<_qe|JNkNe1K7YyGZNEUQvv ziH+2wR&>CP(&qO#PxcP5Z);yLOmxgutBn+myD6n%)cD3G-u9K2M=!@#quP>hqF+Fz z33=o&G%np)DMQZ5-urQ91MQ|Xv*a?4T{UvakZ9->(*0sXf`>oI&HtfDw)9Kydrrr` z%U&^bPkX-9Vl}ro(ny@oTib&vPn-7ZIUj3gVEDP2G%CLm?$%mpjoGp8a0AO9i2-0CVcm&GMM#?r&DF`(Jb_{P-H=wSiD_O3RK6QhY`&tc?8eYk4UZHJ zn(uz7MwXw$M}Z;>_RQ5bIR402DWUJQ^_ITL2VN$;p1r661*nG)YxZL6V3kbTHw=>X@b@g~@@fg{wA%t(e{IHjKMjA?H4L%>whhXd0y;@}T#G{#h5+I~`7g(rJtTU33u1kkiq_MH{W{ z(lGYkECXC{ruZ?SrQ0KM8^Zj)PQNX17A0mN+Ne?CqI)t*B*TP1o(|U|xR8}sh)%6) z5`hdxn2w9fdE!Fc|76m|oroyYXu;Y3bmvJhmZsuCaiSk5iSr|po)l#;;pm9|aDKzN zd|K&n=xhiFkIe=QzjGUrzO{KmAqCAZEr2$)gA$|wguie0DDkZ@4f{JEJ!`ZyH)n_C zdLTk;W+x549&JX$_gIGwj+~*PZjF?X*p@isW81KlZRYtt0a+WD3O;Z|M1C@+@Eyyt zZG;mwd-rgk05iW}M17j8oj_||B-R9Hf%z#U6mSAvJ3Z0`xY{XpmEAC`nf=a|>6A@RyU+>kUh89^Zs1l{ zvka(0+?cEjDVU@wC9G}&q_@CK&hrLu$uF%FDP3uB_TQ-Q^Pfe@a`J!$uCBXl2}bA8 z&osYZCafCHtap>P%KYO03}Rj5%|2;nNJ5G73>np1mvKqO5(Cze>8(^5_3|E*3)@ND z1++k#1;DLQaX>7eulrJeMUB!|t-sD@SN5Ly8B{nMswlNPSiL9;i% zFGKO$Uo^|-e$5)Rjeckq`v+Ngv|mRZT6jOxKQ4;Y4~(`vrnC9_TvB~rTwL_J%fs(} zY5W3rXqA=P*wOVWqN_7l9`p~Ux%+fIO<;EhVS9cYKN#ipghKlx8~2MySkl%@P!dsU zGZiSx7JgA~ z)bn=Oa7dh9w`+PKhqr6-MpBd!v3Rw~wa^#9ZuqH~z}v8>F94ga>Fn}BGXWiYQlIxn zwyXWBBOG;dj%O-ihUqHT>BPRKE?5Bh{D@A%q5=l%M9H2|it>F@WidFazwhDVH}DL=`Nuo^QdyjliXLIm6dF zj~lkArJbhLXRYQNw-<~g-z%X%uGTt{1pU)YD}s7K&tn&}iB^E@vypI%&~V;1)tPf) zE46nqQ{nFWi8lg%{pfW`Ne!SBnh$}OHK>&&1;lBN5=dvaLH3KV{LU=)p239G6mi1# z?x%N4yd(VLd;$lbmzV`V!&%3)HBmb=Ar2^zT34ttOcHq{wY^oC6+yD*cC>gE>AxeG z+IiHu!>9AEW&z*b%N_ZsKqeWi%`$^uRXpRScXE{W*R$)0?{oesysWlW(D8xC#_nSD z5n9&dant*FD(UO4n%(~{%NLcs2&08#qo3Vp^sT=bT?L28@)c8>H`*)9a7N*Mq~yW8 z5Tqw`4?*-he{n9fp$L8pJHXAMjpA@ON*Yq9M2vaq3%-Rg)Gc`z>zZeSEGHxm=ZHOy(`@-W6-w1$M+~zw%q)ES3>5 zQcwfey{#UM&VDydY*u@@0Z!CTAsMkgNW1H8B3m4K8hFpD1P)gs0POFj8HgGo6~+I7 z){jU=ktshpBmgy)E<5zl??6OxF884EDrQpyPBZSXc>OycrAO7gY;|+McG(TUK$H8hpuZl1P}+%Zv@omANJ04BB7w1Zo}+^f zGo#YCUMzyd1)qynUZC9R1g}q%E+B?B>9U8E8;jgnM25C* zk`@9~S2kxX-B zciQm!j2;$kt+(JpLs8bVLW!a%Jj(vEubcKFmBL$*vnyWQH4bHV+U}9Q=OnHEPgixf z`#XB)23KykmJen3uk$WHzu@Tm)19fHJB{i`yEb94VwJWa*0ftTgz}JfyBoe9c-t^Wsy_`F5#1xF7}e=vGOkyK-~BS&Rext6Fy$C1W0LY?o`x$Sdja+r2Zch_ zF^Ep@KA_17b|GB(Cl+Sw-c+A_USS&Jc1MPhL_>Wj^ODJZ9L9-ph79HNCSD)>4k&t5 z2~0@sTPB+49zJdxHr$o}6SU!u`d^@pqF7*5kyW@O|3?F>A6j|Hg7ry8 z6CRoK^P*+UBz$q^MxM!&oqOXf@|?jT<$!aL#)%l zv;&WP9FueipG@YG2_@XSP>rw>+}iiAnkm>e2wR82GJ(LlvJG{)Q=e>D^&gxTGuj-9 zCqirOV8M#;H*qOAGZl-Nr}Xtg9;j(INP`K2iqW5Fy+PK?3XTOBbWbjvvy&>AgGF-~ z9kBG%9p2X~fiwHBzV4XxJi2C-`oi#X(9xf%+HvZ6+&fqRtgXK${<<~mdlUA<$*3G! z>LXSGS68On1>lq`OcyZm_)i>jxv=i7DxjvpfX8UUI$4iPuWFAo`i22N9{p<-bEyk& z3#b8=ZHR-P{eym~D4$CfR{GqkbsOEgKuk5eEyf>VG{H0dLuHoB0)G*|g42aSp4-`N z93Iw}VEQ=ym%azK+l0CoOsFTY^Z@Zyn7kM94pX|{O3*-Pt%{`w8*>n&tsiT>mLs+~ zbr&p{YX~1EGZfq~8nZPULowi8oFXCHEml+y9E?}=@`v%i?I|_XEW#`LP|{~PIXdkC zU)MR3Wfx(*kCgks>5Cx~icmCSvAyw3AmHKvc6%}(I+PoCM8^(ADlaQ3uYHJYIxuh+ z4C#e+t&`?H%+t_0h+O0U$$A>ex$Ok8Pj?+jinezy&~~Mu76TBc1!q3ZHTL+WFex>^ zG3bMx6H0~^nc{IrDL1JI#0bt+^Y?E-nOE|g8#9KoO^uPjXx!|+(YMYu5HVgwl_j)krmOu9N3QmQY5IfH;>>E1qR@?v# z>0bTd`bgSqEAX*ECVQ!!kfOJB66pz|C{fJ$Zn{$v|E>d8J1tOX@|L(5->M$Yz{?eC zIu7lFlcPKs3aUsQ=8c^H!ZqzHpQ9;Ob02)#LbqyQb((V((d+zsho=Ui)fR4g%|3#F zQ$#qhSqucnMyH+LG`i9QcYdMnvH_Ui(#O7gFLuI0ML!Z(da7Z_h> zz1nkCq6fTuwyXC|YxO;&fIsamLH1I5VS9R|c)#24;ODLXwWQ*oC>u-Mh z)USc*^6AmKmlC4mN1{z1jf(p3YYc?>?L%6ZJ1luB(pi_H!z_D^5YAx0loUEMI|b^$ zJ(F8Vyw*`a7L7MOx8x;ilWVIKp7=nZfOFo-pkNU7h~H+nyCM320y%6V4%NB)!~&{d zSy!LXnqk)yJhE_Bzm*?~z5&v{5tnb`(?{FtN%gsnuj2?6cE^gD^0#^oY>yXIyj;<|{s zfRU=h;|C7^GOtq)f*yQ=A6#pvkX-H-D(c{GZjG{!_$p1eqtJi8L-{&-ITdB~t!BN1 zDaX*!oO129(eZ;YeZnmATGx@VJ;X+yw8@U|aL#_sSy(vOW^^yg#;ljOqV=lM=Ao=f zc?8lJ@RwJ4coU!b#OHh0{Q^1CyC-?PXL}KQzU+NTYZYFjygYb$^$l82sGxqIYwYX) zqh;>_rCfsXBX>oR-8LZQ%W99$)trvIwhqAZYDXmLh)PYwtIZ$z;e{o@RicXYX1I3o z=Zq`IMqy>Wx(99k$9ee$p9xF_?;hoSm402Qfll4LUdG8&3RRjgUK__G7fqqc^9ni} zlqY^?Q5tAS%D?FN2kj6&Od}pcdyS@|iSSY0)Ccq`E?Ot;+-`o*a{nT;22c7-*f7z% z6cv2xJgXj|Lm5|S2n7{13VNNka0Vhr({S&zRMqqLuyE&&5=WOrf(X+vCmDY-PU+2o zd^-w9fc6EG+h_9DW^8SPzX|^`I%Toa>si&5Co zQRWSw=w1@koz&c-sYt_L*QyI8@qa=$>3t0gp~6zbXB4WibPIQ+66gIYa!n2kCS#dOO+AfUl&?7mm!wUntXyizkMs>R z$VkaW+NIKEef%8U67s}UZx!wi?a3fDpz0bV>&@RoD)WA$N zKtB-fmPs(2hpIfz>&=9LFXazCp=mbwTt(G-5;;3WgFs9A+Nxa8@7BKpsXnx`;FlW_ zxYK(pJNqY!N8#0KFh-m2eeNox!1pTNCy35_rdQ98*@+7rWX37qW-)GAG7f}KD|AadT0&>28h0n1B8&#RlM80Q}D`Npr(+s!PYO{TNS@kkK zTE6*ato41HWs%B)O|5B}z3(cL$Rq{Z>*LvP?Ef{+FQxq{&6D|7P8;tHhvd?DzhRQuBN38ln(QN^?k8f*N~KTTX20vD=or0? zgi+?9MsSkXCXY8a4w&vhL-i2}mS%2|5p+VNhIAQ;r=)~eabi<@)Albd<8cjZRK~g` z|Jf%!iHPJl$h>eq%J z~?cep#h+(y{PCacapG%VV6YM7LX2ufE=ICfvS}@|b4^?7I-2 zw$6QO`seTH!U83)ptY2X|G2l-~WwAiA6`b}AgSi1F{&j^>Nc zf6|*x^#7EPQ0=7R=>O|uj-`zMp)2{n9soaA%0EKev$hl~|3kP!`aYk6PrS=kDVy;K zYU}Fi9?1UQOaGvv!5qez|0mb+U)~aUYFPyeAxx(mF{eAziU+a|$mF?qpUtv}^28GZS2tfrUl{%5wKy*_%|((E!n{+Lim zg}aLYa8gF7V;_9VoMYlN2jp73<tZ^M=|u)Q%S;qnqRIv0O)->Z>awa#}yxHUcYFRe}@{?aeJxyt~KWi~)DGz%sE zURIc5c@kh{EIP|@Z>uOnH0QQz`zHCeovO5?ggNi$OFzWbkl5biEVwr%idf-#U9#YJjCARG z+K7?h3B-JrYx7PJNK)|&{~bEQcl3FMl?T}dv7v~v^OZv?>N+Dc3cZpKrThAioq|SU zn2pbPXgXMBn(u}B|5h8BJs3S#X7HHh>*kNOh|jH_U~!#xnZY-?+zb1zId!%?oO6mO zA6nf2m`|$eXtA;H?i=ja;jl!sc8ZSJqW_I>!DK#Fx6_N^#~su{EPZ{X*$&^WAz?0*=)5b+jEy z9a)yxEL?$G$j1jty=YQ$FX0<5T9Hjsb-fMK?CXf!UCm+^Q;m|=xfmF)d!~JbI-R{9(F_W` z*}x;|#q$|;fVRlypjaJ_2VG7K$#Eo>!s$&OdcDm^vYj|&IL1aBh}xWr@R3) z(Vr|}`DaT@2q12&J)9;LT8-CuoTBrU!(-NAW$qBHGuoQj_w|l(Z87VvTx=Ix<5g`A z4EqFEK-QoOX7}Pjn1{5UjqQS~yT$cI&_-HZyd?_C<__jIXa6fOS3dIAlv$e7mUENb zc+7~C{94RA-p9wfm?i)A7l}S=5*6_!*X9}fCKk>xGdtJ*PjCYd|G&wqwtD-U zcPJ9Kn~X8wgJ5WRNhkUrNPCQy^%uLfnZ?0)=&Wc|8z_ z<(+#OX(zZ*Uhww;q?QclBZ~qRP6sLZad!+wfCo0q-?&kwW#k+eS7;yBr*KX{cN{ zP<_PhWM}Wh{V9W-L7e`BPeu)Y6= zJdlJOfO358Cm-s*>wRoj5{sGI1*KI3PJI_a@%N@>nZa_<>EwuIRF*;Fh)kSIKVE>u$ix^F0_$B`XVTDtpQxBDvvw7S}ggBBT;61E%yRd>CsVvfo~$> zm=pVhFbrwY=60Xl97cy1`)~ks44U8va;_ZXaH5?2kKC*8<}03DSFfmOUZwTYOx?)S zTgOBbcb%I!o-wR{dCf&gsaQKBqlsZ|!QCDB^)H%qu@^9!KM&AWAikx5cP-bFWU*9cTC85#K zQ3d$Znesm0+O_A7VgxTG72L_5ar<6mn%}$}Bo`LaU+F+iApd>r6{B7A^>%JyB+wfw z9(asjy(K#=YwcDOd5Oa+YVA~8^qmf7$kV$|f+4xYeU%p7qVB5s)$!f?bZTn%2mGnO z1#Ooxg`fNpa5ds6M@OtkmNospF7l$~0lfs>h>Bv?oHxELULnPy)2ah!`j3?Qay@ z0$jXWiH5J9ZT>($I?zoDh~;viGM2ibzI(R>2a8;A?Rbe{21L^kiCZg_E#qpL(J0b? zPLFlFd5pdiiC*>jYGvCqqfne^$31F--c2K90os6Hub$)#1;_C-JVV1&@j+AKfvX%7XA^{RzbI88{FjYQ^LJ5hiejI9V5*2_wP$cElwS*`ufFY35* zI=GX}lekjKc}_}NHgxzS)8gk~s+#Q-zNV?NPYAEY&i`yuncWUuwsRNT0WBI@G#q-p znT>z|t$EGRbi*d@M-qlb&WJxin~$u|qFx743mf zEG;vym<|uS%>-_oFKYSmR_DT=8hQlaId_X=qUc4Rs&@!~Q4yf$g0|A&CKP&*^jyQr zxe$|^9fu~Hl5OU>l5-(z$wma*7luUj3}>Z&=8UI{Ads2fBu&jD>P@2Ascn8rYMGGE z#7|`dA6G;zXFnhoxxF1#5mw9pkyC zp(}FmWj&N;PUQR<&1dVVwG--b-aPwWAB!{|owvL^bG81AQKu=YRt&8j#{eHC^rY81 z^40L;o8;oBFAiA5Drk|^@;=y}-_t@5bgEkrq8;Hsbg3ciK8~&@;duCtq0{ABK8oSG z_=yoYMG*lU`}rLfXCQtkW+;v@Whfqf%S%i=Zzv0&NJ8o?gmfg-dGDfOW%laM@ABG3 zCo}0Ku=-5TBur@_?=gIh{=vd=_Ow6EU%dFE=fd)ZRF*bP19u@aFKEQWi#NJ|oq9|q za@e|O`+B|BR65@=hJ;%1)d7G3oQztK z`E53|+d6k(Rl+Qd!mO}zoc-+gHtxZf6f#}7NcjyhY^6U6bT=IOm3p znyR9#61;JpWGHK?!Xi3;rk0T?ofUV_=*}Cfhq(iWJ%^`X)NH8uydT5r_MGvxkN4vz z$(%T@yU2=N_#7IFq@ucCiFCzr7;4bhu#&{kf}llee3DuqUQmo2J_)LR4C%g!eMi_J zB>l`MQR-LV^|)Zh49xV7!+HuRfmbR%!!0h0J+9D>JA4w7;XPpD`2A;p=!LjJkWeQF zvkPqMD*_lyg>U)$+_|0fxZe^NB;Z&{OsfL-dnA^+LLxF}1kTuV8F~DMZLKG75Ph>S z2j}b!DxN%+>$c6+W?AkO=X%LZyW8b2RYUrdjvrelET`Q3f`q^+XG{#qj}w3q##(+&!4j8)52$-QJ&eK&*4_KF&}Z-+9C%H{3wZ9Xx%fDNu1z&}>bNd1Uc-fIT{aR#F0 zuA_ewgn+e(BKRzT&3y}EJAG1b_C-t>iJEW)IO2L+uT+N~V=+{l-c(9gvJX15Vq_V*Lm8Ow)d@u2>Vfl!2S_(#%N7_M4N^0XhgK-tX*p+rq{At7b1Z=WLLrp zlp3nVhvMh^ zh{Qy%Y&%FaKHHQaN7qI1C&4?7SK?GMPIw@@a1r=%EIO!P$o|CGb~#Qfo>S2NxxEK* zsASu=#WhPGtzB?Y3F;8X`2vc@sVmMKM+I0qTt;9I-l>z2P+OGc&1mpf$8(I_-uM~R zHcBeAjO?^)N-x*iexK%A0P05wJpd~2W;i?rY9aIUp*!hz5)x;Mw0GS8X-feLEg#9> zmEPMp^9YzN)0h!%vT)5VOUKLlL4Gj*z-U?)2P9BQzR~Z{<%8UFFIH&2%@A^l2#%GF z7iRC7QHXN+oJ ztNIYpm|mYwCsA;aCbgBaYm=3KGbJUP$juJIs8cUbMvu_I0d(JY6Q=fpudPo4165;3 zz8h`JEhv7&NRHfmXyaqigynR-Rh^5G`UI!t9H z%&Z4IsW%dQNaC@fyP&#A<0JG(>*8MCL4L=B)qPme?J4jjM(^kau~luo z>t>w|?zBgs)%di4YzsqaApa4SotZZspH!ZHdIs-V?kOT)aOgUCDtyGc*J2bml)l|5 zLoY+tYOdbtdR)Bcb$9FS)?`;z!W4h+?m_s5g%C{|)jXQvzh?+Q(oqwr2zlGffUg(yzdqBR3eq7QX81UOhzL_=0Rm^uRFi%}0)K=hS zg{>tx$A%x&892D|nK_XvmaO*|CSLwW0iEKSfKYkD%^Xzb(zZ(lVgY%9pY(L>zf$u` zq~oN^8(&Gsk$&jrU%2WRx<74yP+rs zV3DmaBWz$(}0gl^27k^mcVqf=Zo8_L64`H)(YC3lS}O5i1+q{L|~4sOhq?2(StI$PNcd^i*TJVMt8Uw z&b|Hb1XXwA(!Uer#-Z6Cg@yEe|KY_oB3arJmU=Ud>#=3=J98;6AoLW?-AeJKef6be1U#@k5$ z0Z3~z)=%lt5H`@r_v!vZ-%d69^nf3^vPo zb==dw2Ozn=w(la0_$l)PG?1M9*5+z#S4g;nsk5uAzXB^t2AL+Fi+nAH3Yz?<;mfSN zEi}T4Ku_6=XNKB?A;51zA58(*@}fyGR|eT;aScUPWK89-(5VQnd8xR8)bhnqk!j7} za%R6fAhmH3k=P^(iaH0W#}JvNX^$T2+=$4u!mDD?b2Cv=8NRZ0%(I4S86f8=Al`L< z-|U&VTHl4=o|lxDlof7~xB}g6$k8T+Jq4PY<`*a&ysoHu=XkAEs*OVh7p17n!_Qndr>)H-~tCgSim}_I&Fj_d}uKb%C$F@DbnF zLaRy3nwZgiGty3)v&}P=Adn0_4dg+7&xtyo?}!o(Nn(XlDDXn1``$$NX2Hp@+cwbJ zwdb}-<$L@vz=%UkSOn7kZvHUhVIw6>uP*SOv!CduM>fvej01dQ$XE0bIqf(3!;p$E zR}Y8!5^MPU6c=~XDP0iy#rH&D5e324GA*^3E&!2s&%|g1U3+|Fbky<^b{}w7c~}F( z-Tq^TO~|-Ek%UoiG$zt|UV~LlRfRv@RcP&Lr?yMt;%2O>q-|ri(&1GyXcWc9DMU{( z{_(tGFd>gs>KO5ys`|!!XC{oJ*6!5=7Y=o-Y+Oj>819ldaLTXM!e+0PGtKI3R)g^1 zfHH1C(OpRqx$eC)yCKZ9@ctpkJy24L7%9bFM28FuXE3wY%#4dHtwg-m&Efw3b|-H^ z?^i3m=54Hg*WO9^w_abzGhgNJ_Id#-n0?~ASo9UMGb2`wxBzMsgH3H={EgK&%^&_nG*j=X_=sy`TtE|fv!`EGChU^kT%MOiSc z_db7Ml~MDLW`RH3qi`ak+h9Cb6__zJsdA3*w?;DhZrt?}aNNhRp=N3%`M3EQ z0_u#ZM|oc0i2V<7&QDB4^qgbzpC1_TeFcU;1>vU@!C6~zb1b|u9C@tlmMa(d_I{#W z4awBw)d6~gb(kzP^_)OMjygF$(UG;jUfLvhK>@aK7++8g=9rV$^NLzXPvE`ikDm-H zR|HzI^q5nu2f%JhMP5#btaph}!I}U47Pe9s2BJoyJ`zq{2$mv*;p4=0P~x@Mgsnp43Zyt~SbALGQZBTrs+=Ga7Ac89x?*fe#=NBO&Zl~mW=r7fdZj%tKT0U^vB!zu zchIm7^ks*;w{kT7x=9Ybyuy(qA}nM8FV^8kW%3F4>zV|IG6d$GqvNdi5{l?1?`F@> z$}sSQtU2OjX)u^ok?CILzKc`|i-@IG+izql6kkocZ*geH<6|W_5@a*Yn1u$A&F{b- zr|*57%1@D}-{il8y&V@90j_#wryKN}WJM_WU5?LJ5t#{!f9@P!TB-(7?(ZH7*D-Ps zM09@wzQ`{Kdh~euTU(mzE0-rFH$;v2L1ViSv$#Mawen!VtgxuuL7@Y19tTiWQ2<|t zn3X!5ZBsu1<_6AI85}UY@v~hpP2pd+-p`sQ%Cp+WB_7p!lsC=eqdfqY;u6abBsaO_0HcJX&Y#)jd1+CbJwcl;9_#=y9<9lO#M4 z+osGcE(xGN$;~}dBi+oz3~stn^@L{gl{^j@gS$F9Q5Xx_0Tk0T30aL*ZegFF7S`5b zdPBCLUll0-K)KaM;F(MYOaFjWTUBTPPm-#M8o6y>b7U;pZ;@sC8;8;l9fQh*lEbLp6ZS|#(3;UStbK`@> zeC)w!N#IlQA&B(~rYArX#qMmwwaiRe4`0FTCITS~HK{e~5G?1uqzl0KUX~llrEtNd z3FJQ`eBv%E%kEhe)vJvf0X|MDbA~VfSx)%PHlQhvK72cj_wNrZQ1Tq?7K2t{d94b| zMmXm-a%t1kC4ur^_ZPUA(&o48*oiMUuDK_ZZYB>oL4EFX&19F={2?yfYo!_rVv-2J z^0c*xaLPB~@ukJx?+>KE7VCwrT8&D6bwY+UOOL{^g_gw2PT3eQhaT1;D!lDm*aY7; z3qP_5ZF&U;PF^@|A<1ahjr7V?}g(gCXoA;r}!G(`eJ6SVyW=5`q zu8QsE>jVD zCFjJfl$Usk_-Z`ZR3hs^{9c;Fm&W~_Y|RMyj<1h4gx;KhRkCGLe%K)-Yl|X<8sjhx zsr(SY!{%d?e=!v-^tm4a+O(1OwD2;@1Nj$4<9FLzTSB9x!btg)#)XlLRJF->ngOHe zR~!+U!!Ai`=kMOwnmz@+6_bW{B--%D!g=3xs*)(c2~_~Zo+tBF(qD|QHJtR4+^ulA z;mi5R$;ol>6@EHfWnuL%_%P6oe_)+uLTKslBSfgZSqFk{3MGl#UqZ}XdilyK;=`*U z%ltt#wj>#xrA_YRUlDiF^Y`{G*W$G;K+?MBQStljWux!z0aqrO;Ww|2Z*m ziBcOc9D0Ie{=mQG_PhTrw_lm{ZwYy{e-j`4w+^tdxWW8OO|SzogL)BfUuNcQxDmsn zS#}_^Q#$Eco0U2LQMK}@`tW?AXLQGjrk1+ex$)_9fAht=b4jd(r4e>VuRaI(H-l<= z-x>1Kxe?DvX+?#XO2h~WJbQayv)GCVg`$BWpJ}D*r-Pe%2POV#lFQ5r%o=n$C(9=?Bxo^f|>pjqzjj3xIry=Ryc~Muxz}4*&p?rfaDYcLn9B zOnDUYFOqe2UiLC0kdtSfeZTzr19Mb0qINl8z{l|KQkcFi=<$A^4uMYTxytcQKHK|W zTSsju9<*Rq1d})R>N4w9)}>Q)@KMXPo1;e*;guWUVc89x0d<@a!EM{GYrd-X(GimD z=2|I7Y>t4cGh9@64LKDX9FT8#7GV2!s$?UsCu(XCU5}~My)i3@6_G?c-351IUb@9COXNs{3BmU2iIYlgweG^{oxDxQXq^weQRbW_<1Vtl^iR4QafHBg1=5D?~@Df zAla+eCkLAqUS!KpAa0MFr<)V`sz0N`&>OZDm7Lis){_oatrR(JvvU(P`Vh9syY=?+*FcC6Z>v z4!$5MQ9IY(p4D6ZasZyz1%&)GMWtErt8+;{^iy0|o?OH)N~3v_v(-cn+C2r@BTH^u zLqZ|A=j#>}R5+G-^Cz8iRSysbL*Gx*3J?{_w>k zR?_f{-H4|*77tzz|LBHT8)qZI%vzO?T~j?N_-vXTJHqE*hd!&>U(V5B^|lO*7gC#q zK(Y2AYWZdx7+sSG_mXdb>{X0DcU=C&YQ#gtJP^=)B;{K|s>L71&Qs$0`T-fE-Wg+2 zk;NR-69Am-UNcjgsqSQOf=*K=q+uDpx-@hGo0yNwVCPu>X92m@C$i=7j@s@6fB|Q_ zxI4O8FAOzVMw|_|!Nv#Ym|s>TPwSLaPm%0i&8uAKE-vhr7REbLKYE*Hu}^?82>O6w zMLnh=weMo;a02sWk(Iz53Fh1*FWPjt2r3&k)qffmB`w5#8oZ4&i)oq>a+bh3R8H_6 zc~H~qotnaH(t9SD=ts)C;`zauD|-n)5*`UW`OFF? z-dwi9h=W8#zetfWH#szllZU^}`xOYu(rmJjDfc&`JxRF;MNVTfo2d$i3(`rF$&qWV z=S_aPI`iQ%6gM;#9F|kjfTkI6F4f{j(-h`?OuhU60sndb3I0DV`j9{UV?zm+WG*Zd zSu{poO)?#-`J8q=P&h1g(a(4he@o&TJ##+S8+?=VLy>3}X7GQs_tjx-Eo;A2C=R6* zD8)9VKyi0Ti#sjuf#O~yIHWBF+T!k^K!M_J#i3|$cXtR70>Q&g+r7`(*S_c6`<(B4 z?jLvkHCbzAX3aY@@4UZxjqAZqhxOcx7!rB30}$`JiNtDAp>`pY@k3eis6iWQT6fqJKfn|Smi{l zu}+k7OsNc61ZjSDTX>OXChODZPBkG$Vsj_|R-q7;YqrDxe1JPCMm9AV|BVvk=P7cy zV@knd7@JLDt>x1P?Bg7JIb{@->H0w1Co&S%X(@Kq>DqD()0qx{2>(CXH{Slge5PoR z!$t4H*|_cfT{LYScUjp|RMiI}FNOojN6aSJ5Kwvq*`!$rcHWQVEO(Gl5i?LIat&>2 zdIoX#hVEiRcPzmUA`?NVeVWt0vM5hZBLXH_3A?Za_jVT2oMov zgUFk8%YgVt4i4(CZsX*Q?(!DxT6$QAZzpuj{B699kNSmZ>n>Pq7j)Ft;)2gdlR3~# zLel@}{`H`vh3$@Rom4Cj^XS8FsfZV!>PggY6K8=+Tv38`{EIV^{ zQ-X6{@*BtH_)+e(r~%!cz?m|UH;LuwT1q=SRj^Q*f+3Vf*YE>Ur)A^1)a<$l z-9g^ngn^$DW)5$U`DvqP%vXg0Pt|u(A7j{}?cLC)Mic-LMrITpFSScqb-HWwpWB%X zd<+b@-7>-eojSrIDjlhO{8kQp!B0rzRiB!nkQB#!%QB~W)fwXJOJ46zilW3m*E7p! z5IW>?fE=P-pZjV=p;v)lC*Kb|-P7~Z z+HXaCQ@mW#ELe4#A0VypO!~f5kbnMY`!V5DuD6CPir=!|vHRh4liK){mE$e`6kO3) zQm`OVQ>POlnXF79mk~}&1mnf@M8l1{L#oR@jB&ldz)}*d|Xw^Dfy13NeOZYjYyB54nF_^-w<9oyy&b6w$@z? z%q|b_?+**Di6EW+F(@wnnN5H2FI#|nMj7m$J{2bVQdK2k`_D-e>4bD<`#0*K9I~Uo zU65Ob zuF=%s+#gkroHa)i_0EkKd0iIP=1)Oz4mR82!?NI$CmHv1?Z`)kw3W?|V=j9{Auf$frt8)?lj{pVwC|Z53tzyYx*Zj55N3w%PsxR;PIRI(1zMEVi&EI|S zH=q))e19FhAOycq*--VR7Uai-CTN+|t%#HJ^3^F@|I-DV&H8WFzW-@b_1`7;nURpH z)kdFFcetO;AQK+SIOT)NMa4O|ACdF7Ht>7C_6Tw<+38e zN4LS6ImyFc2DOzDJxd`QP9*I3<@NcF(*5k%n4IU3Ea~pxdJy)irvo^Xs1k|T%LTz)(C zIo};K(;(&2ZttT^zLi9@%7>a?U}#P`e;4Dscl#`#Z9U9^QHk9_l!aUP7XFg})AD@> zvAX^M*zw@^+k0CA%c+y$1h1uDvFtQxCZQ8(A!3xcc#^wd1sgm&Fuc{m@b>Iw`+ak{ z1qETks$YDzJFwh(_zAL6cwjiMM;PFGlE1iF9dXT$;Y6bhKb}=H%s>3v*PDpJ%XOr^ zvp?Nq#4*}a({G{b^@v*}#ekwy%Ea~K3=69{Nhrbba22zx%ciP6G@3LyjKJyyuBj8= za7$ZoR>&prT zlS<0$DT&J&zxuRGgR#(auxV8S0Od)>K%bxIJl{BaB#Y^udncve{Rp7s6{7@iQl)`T z4$m(tR7}dmjVi=Jm0Yc{W_OCrnJfwQct4)hten=OPX%oVWrVnwSrg2^R$!)gh0VfR zC+AvUuB@4t8M=HJAfu)6AaHadX?=ILH_FUQUwPS3`60$im0y$_lc(c3^xpSQgz~Y# zGBP3YwhjL9j$;nfgPg`}oaV`ZF50K5-7g&fdjP>D|6$e!>Gh_5-t(&HgRJ@}{9(@b zDnk6><V;@GnRLWSi z!-XG9TPasT8L&P0RuN+l!7%3P{!%OfsyzQ^M5U-A52c{GNuBdf2cBmF&Z^ItPNpGCWA$=Z%P%fIO;D^=5Mx z|Go33mi%k@!xx9EeF?*hkp_Qc4Y2~hu-V3rsa*=OO$)o56~z~avYfNR7cXCSILK(< zO!aqoLQBMJi(6`DE~w^S(vBVM$we4RPiDKm;8*3g|M@}Ke&%QlVd8`k|?!Kn^U z3(7n6Y6wpc>glE;HyKqsUk=s5365xp>i6r8oNvO`GjJgXI~7ewnm-;B#y=W1yUqBB zaC6G@24gvjP9(n*=Op}fV&eyQd-Jfo`06NZaS6(#D8lOF%|j0&;;?;7ogsz7g19DW zQy?RW@`tWB?)Y3I%dLj?HyC+bKpa9-oIP_7zTW*b%CgXEMBE!(L)hM21DVFU(cWjm zp*=C)3uRAHSQaVYX~f~2EB)S~RPD7f?!47~*o>qa+W4V3ce>GzGLFQ^9u^kEs;|A0 z$Z)Z}CHnmOnqCZ@T2ySoC<1PRb^YmqMMrACdym9S{cE9sJA3~7KfxEn?5wI9RbR1) zZ9liH8Pgywy@+C;VaT8b$1qUlVIHE^*FN&qrV&0G)xjf!_e?|^het^avWu~Pi40MQ z8uMYErZtMSVh~XpI;5NCdO89%KFQ6`-?9HUvnXGjQeT=F(okHY)_v+@4*}D#VC(Hi zNq%>GvLlsZdf$(3r*;J+5~(C^tBg10Pa1n$iVv02)Whq<{e$Ta$Gpfo}&Yj|xPvP8GOV{6<}sl7sNGl!EIR>egs zMvJVvQv79QbJO>imCc!F4kOX{37@uTY3id^!BbRBmg$OCzdcfj>yBk`!t<<{vzO-a z7lY;n2ODssw6j#Q7D+85YOV9T@Q0Bh+*G#)J7x17EZGPJi)KZp4egj;p-E58CG-|2 zB>T-%`+pctY4=UrZ{iQ#oMAZ7U|IG}<@LFFkTlO-GCzS2^1jIi>{}6cG-Z0Rm z@A?VFlB1i}rMHWHR#2MQKV?V>*P#rR=~SCn%WL`_CASXaU$eGBM;>oi8;k^ZQmR&0 zFKzYvf41qONpnw+ao{| zAr9``98vyygl1JQM}2HR^+exedy7+mI3~7rJphPCJ^9l#W_iK}}~)EE2(CAFW4noo(&$5FsTA5v9c zTGSH^rv`-5J@Cw{g+H?Gduy>#Nqu(tr4Q)ivc;M|^}DAc#iBePh-C*I#G2^0ccxWD zY;Xse#4C>~ob$o$8L9@jFD^1YXXeSRxIHztBdBh^6u_L_`;z0>@acK_t4p3-3jPFK z-Mn_mWoIo<1ERvUFUtFm_vT(bdS#Ag_b>qbQ>KxD{Un+iQ-^^;hfmF!-%Z za~!^4rMK&SEz}J}EOL;da1F3j?jF##04*{~k@_T2+b$Ia;ri9QB9QhtN(Rfk>{(32 zc1c}qe#MX#LM-evvgpsKEs})a6 z=uV4`B6n0Oyye{w>}x6KP&7*mr+D;8mM~;Au!dRQ+63#CTxhF&zO)`Unl=_22*hlX`>dHIg+w#03fJW-s$SsS%cl^}Z}C6Lx-3rbC%VwOH4g}PJOeb2G-zvJc4yfsD9+|IP zyMe^y*}yZ9w0<}H>Ic&cTHdWkcs=&5%5(|QBB5~%y;NXvpp`|+nl*b*;)Pj)bA8|rqt&^|42~eGEFQsV3&b@7EXZ` zy0bog{ez%I{yt%#fgz#ccZ0_iAW(YZGX_1 zb2+xI7m0!7{;_l5t@Q_#z*7PO0iz}VBidXyxVyN#33*p0J);lzGQL{a8TiojqJyN3X1XeV?8#7s-LRG8f)0yPPVh=21S zr6vHPja3$YZw{^5+xloihV*ik41mj5{5j6JK;bKWz>M`65bc-j#9?hPyZ z9R(Bb$7}DLJ6XlX(mnC5^H9m6P3oYH9*t$2Oo4?L0bOB&k6IsZrLH$T1N|v)84C3* zre0<6qAqCIy*V-1lF=8-Kf_&Z?))EU(+sskmi}_dOf5>H_u)CXioW!3h-!~H;PSHH z&jzCPH;JiZnP&%N{|M&Q?NpT&tX+CDM9E~_+%uQe6rPg&ejUnEY6ypfP$U~N{vF^| z9@vEXt3=5G35f=&Okv`T-+zyI&F!Onj2j$RI7PL#vtqQ@Aq&MBk(57H+Y ze4qNzaYa+eT5LJ}K%=gm+4d(iA2SS;3*VSNO-bjj!j5MXsPKeGD^aBU7VSTNG|UO7 z7VRB26DCt>O@v;8hiENmk^28nv#%l8n>ZuCF)mx>}v(a ziKb?LW^4(aOJUQqZ_+ILtHy4pB!&+e|;uTZbwLhA#%5K z_n$t%|5kP2zj5G<1aL}aH0oS}(ZWbVfLZX{9v3;fUmd~gW>9;pLJh5OaoV&wpfMYd zqBY!kDJ}iAk8JZM*y1u>biI-F$XT1pCr_F9cVq`iw;mD`rEqgy2!-&vxa+=^Zn!u& zPbw+F4{*iz#0{#CuJcfj59N2HBSxNYc?dI@=6ES3G1zcZWy1KZm=@KmwYtxYgE~z!0U~>Y; zl1K2bN6Bk9r?1JY;t>hUJt)p0;{I6CNY|qQ#3|fLv~4eCbF)9`RXijf!vJ!_Su7;t zyd3a5=7C3%QZfdw#ZAEC`He)^%GJ%A^|SPfttnFs78kV&>!W@zl>pz}j!SEt`L2jX zkw$X>z~3B?JwmLXLgv65>--!sSnlL48`UWAyxxjEM?FP_@sXE5C+(Rg`RT>P0JxsV zdo@n{hrcM|p4FMd)&U6*=V6x407IEu2m> z;~m_HB>#k~gE`Nmusj8_m;m}NU1zq|*$}|1*F*xSvkd~Vlfel1wa&zYT6at;n$2a} zhPSZJ`t#NCmbHQ_>n*yVQ&jYI3{)31E|=H5?v6Ukw|HYhbn0tIGM@!g$v zdNUbfcU})@#_TEt(X5cyAFK{_DCL39{M=*w5CSbn@VVTx#~OmHLA)-5>8 zN)dq9+3A!WU;!;J?P?BBHb7VV$B?6vy|rSa%R1Nqr+PhK{}c%k{Q6TP)5=vJdkrF3 zr#~7y(Kn8kNc=}L?3y8os<2$!dDTt)%?H0hz>k44a!=3=o$k4RKEPk})+v8ss&70P zo&3TR_3TG;U7(f33BThth&=6JuGfRov4wBazeM#gVsWMiA;+|w!Fi}@@s^Vve7xq~ zVsHsD*(CCp7xTFvKbzNt(iSKJiq3P|_=_EjHvytHmRv-OXFv~?(5PXCXdAo|+;9fa zBjl)ywqq46?WqapM@UHVUVXbMIL>9HyTUZSdifS&fD!)!X>$Sb47EaS{sAqtJKxK_ z6X=?uNvRrTxs{HJPX~I47OX}b_2ti=?{7M_98chh9j}>*RdBYqX0w)-)CH=X2*O*} zF|fRZzsEF{IrGF6oV9=?P}WD>G?pZy5*6|3M3{9>NAB*v8(E{Er&H(smqMNhkJ_w! zF3clOaP!5Fgt~)6w3pl#iP)7sDX_tnWKxYx7)%K02l%=Y**Ts+%d^UACmO{0@>@|KI&tW@~g_jV&k7;JB4c=$iP?r2!=WCcRQiPImX`R+L$%*Qa z_}dwdln;=}ZwVW@i#yqhng>IX=?w$KO|9uL5F%UjiY;W{d{eE{>m*Gf9&N-^JTq(4 zwES2b;@$_@yI~O#0y==)zgwosuYQ;>Nf-o7uM&? zq`t_k5`>dlV}DpbaMe{O&8z6wjAcTl;On?JtQXlI`5jfdZfnuZ4|g<01fT6js4!80 zsW(b4Pu0O)31{t}+sZ^M2y~A6wl}A(B=oPu64^I;YXw2;bVR^f8z9pzrj3uCGa#>GXT%$Idhu|q*1{TonnFs;`| z^1dm?xA{SLw+6FxtSIvs1ZG z)It1{es^a}($_ogbe)1e=omlV?0zgb$q8s1Oh02JIsihcN1YeS7~9(;#M%a}J|V(3 zX)^&n1Rj;Xxi8gVxDX2e!l4`>V4%i7vc-9H-nZs@c5e326bTsbIz7FmcD=AXrcCSF+AAbONXl4cNdmmtJ%A6gr|8Cbe`v}b zu*8iG12{GXQ16P zfOccBpm;j!8Kd#~FnX|EcQ?=KmpnX|!bQ)sdYH~S506nmpHwID6A{{r<3;lDgDMhy zwTAw3SBmY*hlEYvgLR6}7w2YOL?<4CFTK#E7FJEYdqlBOah z?Y$9BnCXq&!P>%- zpiQ$s^Wc@qg?IWK^ptkKXHaKe|J%WUXdV2JWKK{_oW(usvbDtZWSFL} z3rRK~xEeGI=a`pTY(5=6UEyhHO2N&lVI;Y-UcsK6bbJ;XWI{@-55^l01}^3446Fx_ z5yxxQp7#+|qxwGrt$#NH$)SE<%$*`Kk)@>nx=PG+GBe+eYqL9Y#Fion3MVk#v z_uc9R#*ODDV=)lduy&L*;@=msS1( zc`LBj3$nM()ks9GJu4S_>;2SgXL7S^u81C7A*R;lPM@lIpP%WTc^y&sS@Au}R#rrNeFq-0lrL2euiPZDx zq8~~h0;%5NVySa8NZw+4lbQOGL3?>;rKqs2uBn(4k4a zp0d}gl7YKV(-sP@-#uVY)F1=Y=$$9)yyEZ~HR<97eJg$ujuXOHvU#-;%QiA+Sc077 z_*u1y!99)oX5_oorl&TsY=jkQVAzp#t0kWpowJXM-^+2_tiJJZn0wke)cYx$pH zu6Lr0D7dvfe&&FF0}FD%HMjiwf;%Nce#Ux!2=&PjcEkTBcb{W>#S2uE8cDwjCT(2b z+$r~YKyD44T)~VJIo@YdpGN5W+RT`=FqMcU@9*>Wc5Y~gvt}~#J|$SD^TjZWBPN zGqJ_OVE1s z)6qwB3=Bemg7gc`GFh)I`12KB18aU3W` zUuAc!=c7~e-cC?Ix@c6dt_L0n&N*%$GJB1?s`7|G=GE1^l|CRxpu*2F*^^|DLs1v) zMoN!cl1Y$v@lJi*>po4En!attDXRCyXzt5+1G9@DD_Mw`NJhL;l1KCE`Pgek2#(*V zj+AhaQO>qjZVp4E8bf{hy$a>P^10q^4RW<&1Egj48sWs&dxS9vWYvr9pkw*fbV+rX zu!G}Df#RLFu%UzzRRtrc^=BsO$d*G)6KaBT3-c{#Wj$!vtA<15M%~5;_PyhBbb*;; z`Q|cU#{V#L2JhgNzW0vDVpuo(s~4S)d8MU`@{!91ilSd<9JFSr5=(Mv{Y4}mFg55j z>(ys@M7$sA;jxv_F!IcC3HK|0*{ZJ68~rdomRle1L6K?f#{y+m+G{3uN0Ce^CW%p0 z1aPw2FXR|Oidg?)0F$;FsUTfkh97Mhj@i&hB1LFr2ZCf2{r* zwH~KqSB~PHavOabxF~Qi>22CO9Oup zOViz1PoELd$w^+*{hgV}LfUR+qs@}V`w^2BYUP(>f*=4jF)C}1evmmHb>}=(ZjG`^ zQja=)*Ybuk`m3&0y~sf7SfIj9hP827Y&@J`{O8hbIc+Ufey1E!)r0N#Eh-9de9osb zPeFLioU5pw{A5G8s{TM*=(HsGp+79M9W-6j%i|1+^NR0LJK7JSIzK8hsq7`){#nTw1#nRi7_@du zbi~+aE#LahNEmaJ&uKvMdcGa1wKiKMUZk!x%FNtc&V=^(KgDzXhzDU{7Nk%smOJ7{ z(rS=!tw$)1Jo8G8-rlW<*Ip_dxJ!~iYA?>9xqVb3cr5UB2MZ{eyBtr9+*V#Z^2=R=2}j)N0Gw7M z{R%Sk5SG$G!@QZ?+kUo(mwBT$-87D)=Mo?r8}SsL50!F{Eg~TC#IruUgPg3q<&yfR^JHV|t+RJw`5~&BGEt2+kc!XL zBHyjD2+@(HByB|ui^v!Av|(?tHY07pH>t*R+3ZJ_dXP%YJ^|~VOvTr^A&RMJ`wMa7Gq)W zjyato9QUSIuayaQhiN1J7%X+qrIV|D)n!1hJ~jsTqpWOhr^B3a192xBxf@9Pj~1Bl zolvMO1kpY6sn4-WAE{3B0Wq>L&t=XOHIGyr987pAcy)B}=zG{A6g1L?Ba;c8#xq*h zO@$$bHDmfEF|QPG^YYwlR0*r{bjHEz+?pa5nNPX|Y+YlRnV*11-L)F8SsN$$Dp}1w zsNg0UpSKgI3BSzlS}Yo-bGoWeThOp2oLTR2|N2bkG4Hnf`lb7Bz;`l`B5w8mM2=a` z=Nx5KE(ZQv2A^1PVIq}QSvyyBnAay4CO%aX4vj8PW9TC2s`vzc1e95aAgOG&2s#Iu z9NI~**<=9n&t8_Oxx>PM6*>i9)JH2QD;;$F#0m-cYaVgDNs5^*zg4f-Y zcEVYpr)8ao&fi_0bmhU^~6?D4fqMKZ9|dc7pj%YfQPsl9B?{Lo}Cv|z4i&SUGS3BqK({L>2jwzy)Vejs2NI6$S6!}GAP-||)GO(#eA$h@t4BGc9QUT|l8!uEC!H(S zQes&(RcZCp7j{U?uMx`)<vALw+6R$;=!AmO5gg@B2C}e*&c(WLN3PUNUyPC!2Pl zSXw-R!F(~Ob&qwLgl1&)-Q#h4F9iUuthr#P%}!dx_Ph5P%5N(QtVj4L4DwIbX4CD) zvsh2efK>twmml&)GRsA~Y}B_E#rfz!7>tAVMS8Do=LZCj1!pCrdDUWib!^6C>B4TJ z*FFMoX6-8XVdFgfiZUJ0DofQ2*<;Gx&7m>T1e=?90&f=>BSoiBf3kL7X3WWF`QD`H zLIR!R3Jczt(*Wr{ZwJw#Z*>+uotU`3!1)Z={L_aZ1Kj+hwJtf6_f@QH9d# zX3fQQ`gUXKS~th=u@B8?o5ZLUR=pZSWBN?*@z8t4#;R42xUpRVQ?oi0s^oUOv_v;m zN0}Z~aNQi|2fu?k=xsqJp1sw*vH&Y{lNt_98Mg>gwE1)+M`A1fO}xwho1RDfw!YQ) zPgZ;8udutnzfWVo3U{q~oa8k+>GCCIW_I5gXyx~DU#^m^;1Ky+TL5(hzihK61DK|ORyUs}+PR7B>x6lpLlEJ8f zwpabU#UA|k`0b%Q!Fw?eT_cnl*zNM|1qaBM$uWEZZaER!V^KobgtIrw+^a#R#l%0? zA1h@w23sostH(*6&X@a@%9uy~e1=-LFbu6E%(B1p8sgLh$|=7*Rf!3zll}RAq5V|O z^PinUiv!6`FpWQ&;WdB$#+W3z_2}0V47M-~{NJT`S&VzXOG@+@cYc-LW)r{t`PJwe zMzC9c6);R}?wb5AmGa%@`d!j}{4Z9=K+s~pP5&?iM4)5z`>%)8Bq5-lc!?En9s0bl zx26b^1oK!YxD7#nw(Fs+Q>angM_J;*AFcTKZ5_2)_I~1%VRza0zhC&G!#0sEUum?n zAKm2)JtcAh(lv z{_HG96Vwy$>F?M6YpL>YCi4HU&gAzf{$EVc{}t_Si2Sl+T>1+m?0#0OAfqZ>ENSxo FzXADIsiFV? literal 0 HcmV?d00001 diff --git a/docs/source/user_guide/model_training/distributed_training/horovod/creating.rst b/docs/source/user_guide/model_training/distributed_training/horovod/creating.rst index 497fe00ba..71b418ac3 100644 --- a/docs/source/user_guide/model_training/distributed_training/horovod/creating.rst +++ b/docs/source/user_guide/model_training/distributed_training/horovod/creating.rst @@ -205,50 +205,36 @@ To build the image: Horovod frameworks for TensorFlow and PyTorch contains two separate docker files, for cpu and gpu. Choose the docker file based on whether you are going to use cpu or gpu based shapes. -`with Proxy server:` +Before you can build the image, you must set the following environment variables: -.. code-block:: bash - - docker build --build-arg no_proxy=$(no_proxy) \ - --build-arg http_proxy=$(http_proxy) \ - --build-arg https_proxy=$(http_proxy) \ - -t $(IMAGE_NAME):$(TAG) \ - -f oci_dist_training_artifacts/horovod/v1/docker/ . - -`without Proxy server:` +Specify image name and tag .. code-block:: bash - docker build -t $(IMAGE_NAME):$(TAG) \ - -f oci_dist_training_artifacts/horovod/v1/docker/ . + export IMAGE_NAME= + export TAG=latest -Source code directory can be provided using the 'CODE_DIR' directory. In the following example, the code related -files are assumed to be in the 'code' directory (within the current directory). +Build the container image. .. code-block:: bash - docker build --build-arg CODE_DIR= \ - -t $(IMAGE_NAME):$(TAG) \ - -f oci_dist_training_artifacts/horovod/v1/ - -**Publish image to OCI Container Registry:** + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/horovod/v1/..Dockerfile -You are now required to push the docker image in a OCI `container registry repo `_ . First, tag the image using the following full name format -``.ocir.io///:``. Skip this, if you have already used this tag in the previous build command. - -Tag +The code is assumed to be in the current working directory. To override the source code directory, use the ``-s`` flag and specify the code dir. This folder should be within the current working directory. .. code-block:: bash - docker tag : .ocir.io///: - -Push the image to OCI container registry. - -.. code-block:: bash + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/horovod/v1/..Dockerfile + -s - docker push .ocir.io///: +If you are behind proxy, ads opctl will automatically use your proxy settings (defined via ``no_proxy``, ``http_proxy`` and ``https_proxy``). -**Note:** You would need to login to OCI container registry. Refer `publishing images using the docker cli `_. **SSH Setup:** @@ -398,62 +384,16 @@ This will give output similar to this. OCI__MODE:WORKER -----------------------------Ending dryrun mode---------------------------------- -Submit the workload - - -.. code-block:: bash - - ads opctl run -f train.yaml - -Once running, you will see on the terminal an output similar to the below. Note that this yaml -can be used as input to ``ads opctl distributed-training show-config -f `` - to both -save and see the run info use ``tee`` - for example: - -.. code-block:: bash - - ads opctl run -f train.yaml | tee info.yaml - -.. code-block:: yaml - :caption: info.yaml - - jobId: oci.xxxx. - mainJobRunId: oci.xxxx. - workDir: oci://my-bucket@my-namespace/daskcluster-testing/005 - workerJobRunIds: - - oci.xxxx. - - oci.xxxx. - - oci.xxxx. +.. include:: ../_test_and_submit.rst .. _hvd_saving_artifacts: -**Saving Artifacts to Object Storage Buckets** - - - - -In case you want to save the artifacts generated by the training process (model checkpoints, TensorBoard logs, etc.) to an object bucket -you can use the 'sync' feature. The environment variable ``OCI__SYNC_DIR`` exposes the directory location that will be automatically synchronized -to the configured object storage bucket location. Use this directory in your training script to save the artifacts. - -To configure the destination object storage bucket location, use the following settings in the workload yaml file(train.yaml). - -.. code-block:: bash - - - name: SYNC_ARTIFACTS - value: 1 - - name: WORKSPACE - value: "" - - name: WORKSPACE_PREFIX - value: "" - -**Note**: Change ``SYNC_ARTIFACTS`` to ``0`` to disable this feature. -Use ``OCI__SYNC_DIR`` env variable in your code to save the artifacts. Example: +.. include:: ../_save_artifacts.rst .. code-block:: python - tf.keras.callbacks.ModelCheckpoint(os.path.join(os.environ.get("OCI__SYNC_DIR"),"ckpts",'checkpoint-{epoch}.h5')) - **Monitoring the workload logs** To view the logs from a job run, you could run - @@ -463,3 +403,4 @@ To view the logs from a job run, you could run - ads jobs watch oci.xxxx. For more monitoring options, please refer to :doc:`Monitoring Horovod Training` + diff --git a/docs/source/user_guide/model_training/distributed_training/horovod/horovod.rst b/docs/source/user_guide/model_training/distributed_training/horovod/horovod.rst index 082db0647..64673d325 100644 --- a/docs/source/user_guide/model_training/distributed_training/horovod/horovod.rst +++ b/docs/source/user_guide/model_training/distributed_training/horovod/horovod.rst @@ -11,8 +11,6 @@ the speed, scale, and resource allocation when training a machine learning model OCI Data Science currently support `Elastic Horovod `_ workloads with `gloo `_ backend. -Check for latest information `here `__ - .. toctree:: :maxdepth: 3 diff --git a/docs/source/user_guide/model_training/distributed_training/overview.rst b/docs/source/user_guide/model_training/distributed_training/overview.rst index 927fdac2a..834443c15 100644 --- a/docs/source/user_guide/model_training/distributed_training/overview.rst +++ b/docs/source/user_guide/model_training/distributed_training/overview.rst @@ -64,8 +64,11 @@ of service provided frameworks for distributed training include: - `PyTorch Distributed `_ for ``PyTorch`` native using ``DistributedDataParallel`` - no training code changes to run PyTorch model training on a cluster. You can use ``Horovod`` to do the same, which has some - advanced features like ``Adasum`` for better convergence at high scale, auto-tuning to improve + advanced features like auto-tuning to improve ``allreduce`` performance, and ``fp16`` gradient compression. +- `Tensorflow Distributed `_ for ``Tensorflow`` + distributed training strategies like ``MirroredStrategy``, ``MultiWorkerMirroredStrategy`` and + ``ParameterServerStrategy`` .. toctree:: @@ -78,5 +81,7 @@ of service provided frameworks for distributed training include: dask/dask horovod/horovod pytorch/pytorch + tensorflow/tensorflow remote_source_code yaml_schema + troubleshooting diff --git a/docs/source/user_guide/model_training/distributed_training/pytorch/creating.rst b/docs/source/user_guide/model_training/distributed_training/pytorch/creating.rst index 70c037fd4..0f904e971 100644 --- a/docs/source/user_guide/model_training/distributed_training/pytorch/creating.rst +++ b/docs/source/user_guide/model_training/distributed_training/pytorch/creating.rst @@ -291,52 +291,37 @@ The artifacts will be saved into the ``oci_dist_training_artifacts/pytorch/v1`` **Containerize your code and build container:** + Before you can build the image, you must set the following environment variables: +Specify image name and tag + .. code-block:: bash export IMAGE_NAME= export TAG=latest - export ARTIFACT_DIR=oci_dist_training_artifacts/pytorch/v1 - - -To build the image: -`with Proxy server:` -.. code-block:: bash - - docker build --build-arg no_proxy=$no_proxy \ - --build-arg http_proxy=$http_proxy \ - --build-arg https_proxy=$http_proxy \ - --build-arg ARTIFACT_DIR=$ARTIFACT_DIR \ - -t $IMAGE_NAME:$TAG \ - -f $ARTIFACT_DIR/Dockerfile . - -`without Proxy server:` +Build the container image. .. code-block:: bash - docker build -t $IMAGE_NAME:$TAG \ - --build-arg ARTIFACT_DIR=$ARTIFACT_DIR \ - -f $ARTIFACT_DIR/Dockerfile . + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/pytorch/v1/Dockerfile -The code (``train.py``) is assumed to be in the current working directory. -To override the source code directory, use the ``CODE_DIR`` build argument: +The code is assumed to be in the current working directory. To override the source code directory, use the ``-s`` flag and specify the code dir. This folder should be within the current working directory. .. code-block:: bash - docker build --build-arg CODE_DIR=`path/to/code/dir` \ - -t $IMAGE_NAME:$TAG \ - --build-arg ARTIFACT_DIR=$ARTIFACT_DIR \ - -f $ARTIFACT_DIR/Dockerfile . + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/horovod/v1/oci_dist_training_artifacts/pytorch/v1/Dockerfile + -s - -Finally, push your image using: - -.. code-block:: bash - - docker push $IMAGE_NAME:$TAG +If you are behind proxy, ads opctl will automatically use your proxy settings (defined via ``no_proxy``, ``http_proxy`` and ``https_proxy``). **Define your workload yaml:** @@ -410,26 +395,53 @@ the output from the dry run will show all the actions and infrastructure configu **Use ads opctl to create the cluster infrastructure and run the workload:** -.. code-block:: bash +.. include:: ../_test_and_submit.rst - ads opctl run -f train.yaml +.. _hvd_saving_artifacts: + +.. include:: ../_save_artifacts.rst + +.. code-block:: python -Once running you will see on the terminal an output similar to the below. Note that this yaml -can be used as input to ``ads opctl distributed-training show-config -f `` - to both -save and see the run info use ``tee`` - for example: + model_path = os.path.join(os.environ.get("OCI__SYNC_DIR"),"model.pt") + torch.save(model, model_path) + +**Profiling** + +You may want to profile your training setup for optimization/performance tuning. Profiling typically provides a detailed analysis of cpu utilization, gpu utilization, +top cuda kernels, top operators etc. You can choose to profile your training setup using the native Pytorch profiler or using a third party profiler such as `Nvidia Nsights `__. + +**Profiling using Pytorch Profiler** + +Pytorch Profiler is a native offering from Pytorch for Pytorch performance profiling. Profiling is invoked using code instrumentation using the api ``torch.profiler.profile``. + +Refer `this link `__ for changes that you need to do in your training script for instrumentation. +You should choose the ``OCI__SYNC_DIR`` directory to save the profiling logs. For example: + +.. code-block:: python + + prof = torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU,torch.profiler.ProfilerActivity.CUDA], + schedule=torch.profiler.schedule( + wait=1, + warmup=1, + active=3, + repeat=1), + on_trace_ready=torch.profiler.tensorboard_trace_handler(os.environ.get("OCI__SYNC_DIR") + "/logs"), + with_stack=False) + prof.start() + + # training code + prof.end() + +Also, the sync feature ``SYNC_ARTIFACTS`` should be enabled ``'1'`` to sync the profiling logs to the configured object storage. + +You would also need to install the Pytorch Tensorboard Plugin. .. code-block:: bash - ads opctl run -f train.yaml | tee info.yaml + pip install torch-tb-profiler -.. code-block:: yaml - :caption: info.yaml - :name: info.yaml - - jobId: oci.xxxx. - mainJobRunId: oci.xxxx. - workDir: oci://my-bucket@my-namespace/pytorch/distributed - workerJobRunIds: - - oci.xxxx. - - oci.xxxx. - - oci.xxxx. +Thereafter, use Tensorboard to view logs. Refer the :doc:`Tensorboard setup <../../tensorboard/tensorboard>` for set-up on your computer. + + +.. include:: ../_profiling.rst \ No newline at end of file diff --git a/docs/source/user_guide/model_training/distributed_training/pytorch/pytorch.rst b/docs/source/user_guide/model_training/distributed_training/pytorch/pytorch.rst index 4a4e5fa99..cf0810f65 100644 --- a/docs/source/user_guide/model_training/distributed_training/pytorch/pytorch.rst +++ b/docs/source/user_guide/model_training/distributed_training/pytorch/pytorch.rst @@ -6,8 +6,6 @@ PyTorch is an open source machine learning framework used for applications such PyTorch distributed training requires initialization using the ``torch.distributed.init_process_group()`` function. By default this function collects uses environment variables to initialize the communications for the training cluster. When using ADS to run PyTorch distributed training on OCI data science Jobs, the environment variables, including ``MASTER_ADDR``, ``MASTER_PORT``, ``WORLD_SIZE`` ``RANK``, and ``LOCAL_RANK`` will automatically be set in the job runs. By default ``MASTER_PORT`` will be set to ``29400``. -Check for latest information `here `__ - .. toctree:: :maxdepth: 3 diff --git a/docs/source/user_guide/model_training/distributed_training/remote_source_code.rst b/docs/source/user_guide/model_training/distributed_training/remote_source_code.rst index 4abcfb6f8..d58328510 100644 --- a/docs/source/user_guide/model_training/distributed_training/remote_source_code.rst +++ b/docs/source/user_guide/model_training/distributed_training/remote_source_code.rst @@ -52,9 +52,9 @@ To fetch code from Object Storage, you can update the ``runtime`` section of the runtime: apiVersion: v1 kind: python - type: git + type: remote spec: uri: oci://bucket@namespace/prefix/to/source_code_dir - entryPoint: "source_code_dir/train.py" + entryPoint: "/code/source_code_dir/train.py" The ``uri`` can be a single file or a prefix (directory). The ``entryPoint`` is the the file path to start the training code. When using relative path, if ``uri`` is a single file, ``entryPoint`` should be the filename. If ``uri`` is a directory, the ``entryPoint`` should contain the name of the directory like the example above. The source code is cloned to the ``/code`` directory. You may also use the absolute path. \ No newline at end of file diff --git a/docs/source/user_guide/model_training/distributed_training/tensorflow/creating.rst b/docs/source/user_guide/model_training/distributed_training/tensorflow/creating.rst new file mode 100644 index 000000000..4cd5a1bb4 --- /dev/null +++ b/docs/source/user_guide/model_training/distributed_training/tensorflow/creating.rst @@ -0,0 +1,600 @@ +Creating Tensorflow Workloads +----------------------------- + +.. include:: ../_prerequisite.rst + + +**Write your training code:** + +Your model training script needs to use one of Distributed Strategies in tensorflow. + +For example, you can have the following training Tensorflow script for MultiWorkerMirroredStrategy saved as `mnist.py`: + +.. code-block:: python + + # Script adapted from tensorflow tutorial: https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras + import tensorflow as tf + import tensorflow_datasets as tfds + import os + import sys + import time + import ads + from ocifs import OCIFileSystem + from tensorflow.data.experimental import AutoShardPolicy + + BUFFER_SIZE = 10000 + BATCH_SIZE_PER_REPLICA = 64 + + if '.' not in sys.path: + sys.path.insert(0, '.') + + + def create_dir(dir): + if not os.path.exists(dir): + os.makedirs(dir) + + + def create_dirs(task_type="worker", task_id=0): + artifacts_dir = os.environ.get("OCI__SYNC_DIR", "/opt/ml") + model_dir = artifacts_dir + "/model" + print("creating dirs for Model: ", model_dir) + create_dir(model_dir) + checkpoint_dir = write_filepath(artifacts_dir, task_type, task_id) + return artifacts_dir, checkpoint_dir, model_dir + + def write_filepath(artifacts_dir, task_type, task_id): + if task_type == None: + task_type = "worker" + checkpoint_dir = artifacts_dir + "/checkpoints/" + task_type + "/" + str(task_id) + print("creating dirs for Checkpoints: ", checkpoint_dir) + create_dir(checkpoint_dir) + return checkpoint_dir + + + def scale(image, label): + image = tf.cast(image, tf.float32) + image /= 255 + return image, label + + + def get_data(data_bckt=None, data_dir="/code/data", num_replicas=1, num_workers=1): + if data_bckt is not None and not os.path.exists(data_dir + '/mnist'): + print(f"downloading data from {data_bckt}") + ads.set_auth(os.environ.get("OCI_IAM_TYPE", "resource_principal")) + authinfo = ads.common.auth.default_signer() + oci_filesystem = OCIFileSystem(**authinfo) + lck_file = os.path.join(data_dir, '.lck') + if not os.path.exists(lck_file): + os.makedirs(os.path.dirname(lck_file), exist_ok=True) + open(lck_file, 'w').close() + oci_filesystem.download(data_bckt, data_dir, recursive=True) + else: + print(f"data downloaded by a different process. waiting") + time.sleep(30) + + BATCH_SIZE = BATCH_SIZE_PER_REPLICA * num_replicas * num_workers + print("Now printing data_dir:", data_dir) + datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True, data_dir=data_dir) + mnist_train, mnist_test = datasets['train'], datasets['test'] + print("num_train_examples :", info.splits['train'].num_examples, " num_test_examples: ", + info.splits['test'].num_examples) + + train_dataset = mnist_train.map(scale).cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE) + test_dataset = mnist_test.map(scale).batch(BATCH_SIZE) + train = shard(train_dataset) + test = shard(test_dataset) + return train, test, info + + + def shard(dataset): + options = tf.data.Options() + options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA + return dataset.with_options(options) + + + def decay(epoch): + if epoch < 3: + return 1e-3 + elif epoch >= 3 and epoch < 7: + return 1e-4 + else: + return 1e-5 + + + def get_callbacks(model, checkpoint_dir="/opt/ml/checkpoints"): + checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}") + + class PrintLR(tf.keras.callbacks.Callback): + def on_epoch_end(self, epoch, logs=None): + print('\nLearning rate for epoch {} is {}'.format(epoch + 1, model.optimizer.lr.numpy()), flush=True) + + callbacks = [ + tf.keras.callbacks.TensorBoard(log_dir='./logs'), + tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_prefix, + # save_weights_only=True + ), + tf.keras.callbacks.LearningRateScheduler(decay), + PrintLR() + ] + return callbacks + + + def build_and_compile_cnn_model(): + print("TF_CONFIG in model:", os.environ.get("TF_CONFIG")) + model = tf.keras.Sequential([ + tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)), + tf.keras.layers.MaxPooling2D(), + tf.keras.layers.Flatten(), + tf.keras.layers.Dense(64, activation='relu'), + tf.keras.layers.Dense(10) + ]) + + model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), + optimizer=tf.keras.optimizers.Adam(), + metrics=['accuracy']) + return model + +And, save the following script as `train.py` + + +.. code-block:: python + + import tensorflow as tf + import argparse + import mnist + + print(tf.__version__) + + parser = argparse.ArgumentParser(description='Tensorflow Native MNIST Example') + parser.add_argument('--data-dir', + help='location of the training dataset in the local filesystem (will be downloaded if needed)', + default='/code/data') + parser.add_argument('--data-bckt', + help='location of the training dataset in an object storage bucket', + default=None) + + args = parser.parse_args() + + artifacts_dir, checkpoint_dir, model_dir = mnist.create_dirs() + + strategy = tf.distribute.MirroredStrategy() + print('Number of devices: {}'.format(strategy.num_replicas_in_sync)) + + train_dataset, test_dataset, info = mnist.get_data(data_bckt=args.data_bckt, data_dir=args.data_dir, + num_replicas=strategy.num_replicas_in_sync) + with strategy.scope(): + model = mnist.build_and_compile_cnn_model() + + model.fit(train_dataset, epochs=2, callbacks=mnist.get_callbacks(model, checkpoint_dir)) + + model.save(model_dir, save_format='tf') + + +**Initialize a distributed-training folder:** + +At this point you have created a training file (or files) - ``train.py`` from the above +example. Now, run the command below. + +.. code-block:: bash + + ads opctl distributed-training init --framework tensorflow --version v1 + + +This will download the ``tensorflow`` framework and place it inside ``'oci_dist_training_artifacts'`` folder. + +**Note**: Whenever you change the code, you have to build, tag and push the image to repo. This is automatically done in ```ads opctl run``` cli command. + +**Containerize your code and build container:** + +The required python dependencies are provided inside the conda environment file `oci_dist_training_artifacts/tensorflow/v1/environments.yaml`. If your code requires additional dependency, update this file. + +Also, while updating `environments.yaml` do not remove the existing libraries. You can append to the list. + +Update the TAG and the IMAGE_NAME as per your needs - + +.. code-block:: bash + + export IMAGE_NAME= + export TAG=latest + export MOUNT_FOLDER_PATH=. + +Build the container image. + +.. code-block:: bash + + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/tensorflow/v1/Dockerfile \ + +The code is assumed to be in the current working directory. To override the source code directory, use the ``-s`` flag and specify the code dir. This folder should be within the current working directory. + +.. code-block:: bash + + ads opctl distributed-training build-image \ + -t $TAG \ + -reg $IMAGE_NAME \ + -df oci_dist_training_artifacts/tensorflow/v1/Dockerfile \ + -s $MOUNT_FOLDER_PATH + +If you are behind proxy, ads opctl will automatically use your proxy settings (defined via ``no_proxy``, ``http_proxy`` and ``https_proxy``). + + +**Define your workload yaml:** + +The ``yaml`` file is a declarative way to express the workload. +In this example, we bring up 1 worker node and 1 chief-worker node. +The training code to run is ``train.py``. +All your training code is assumed to be present inside ``/code`` directory within the container. +Additionally, you can also put any data files inside the same directory +(and pass on the location ex ``/code/data/**`` as an argument to your training script using runtime->spec->args). + + +.. code-block:: yaml + + kind: distributed + apiVersion: v1.0 + spec: + infrastructure: + kind: infrastructure + type: dataScienceJob + apiVersion: v1.0 + spec: + projectId: oci.xxxx. + compartmentId: oci.xxxx. + displayName: Tensorflow + logGroupId: oci.xxxx. + subnetId: oci.xxxx. + shapeName: VM.GPU2.1 + blockStorageSize: 50 + cluster: + kind: TENSORFLOW + apiVersion: v1.0 + spec: + image: "@image" + workDir: "oci://@/" + name: "tf_multiworker" + config: + env: + - name: WORKER_PORT #Optional. Defaults to 12345 + value: 12345 + - name: SYNC_ARTIFACTS #Mandatory: Switched on by Default. + value: 1 + - name: WORKSPACE #Mandatory if SYNC_ARTIFACTS==1: Destination object bucket to sync generated artifacts to. + value: "" + - name: WORKSPACE_PREFIX #Mandatory if SYNC_ARTIFACTS==1: Destination object bucket folder to sync generated artifacts to. + value: "" + main: + name: "chief" + replicas: 1 #this will be always 1. + worker: + name: "worker" + replicas: 1 #number of workers. This is in addition to the 'chief' worker. Could be more than 1 + runtime: + kind: python + apiVersion: v1.0 + spec: + entryPoint: "/code/train.py" #location of user's training script in the container image. + args: #any arguments that the training script requires. + - --data-dir # assuming data folder has been bundled in the container image. + - /code/data/ + env: + +**Use ads opctl to create the cluster infrastructure and run the workload:** + +Do a dry run to inspect how the yaml translates to Job and Job Runs + +.. code-block:: bash + + ads opctl run -f train.yaml --dry-run + +This will give output similar to this. + +.. code-block:: bash + + -----------------------------Entering dryrun mode---------------------------------- + Creating Job with payload: + kind: job + spec: + infrastructure: + kind: infrastructure + spec: + projectId: oci.xxxx. + compartmentId: oci.xxxx. + displayName: Tensorflow + logGroupId: oci.xxxx. + logId: oci.xxx. + subnetId: oci.xxxx. + shapeName: VM.GPU2.1 + blockStorageSize: 50 + type: dataScienceJob + name: tf_multiworker + runtime: + kind: runtime + spec: + entrypoint: null + env: + - name: WORKER_PORT + value: 12345 + - name: SYNC_ARTIFACTS + value: 1 + - name: WORKSPACE + value: "" + - name: WORKSPACE_PREFIX + value: "" + - name: OCI__WORK_DIR + value: oci://@/ + - name: OCI__EPHEMERAL + value: None + - name: OCI__CLUSTER_TYPE + value: TENSORFLOW + - name: OCI__WORKER_COUNT + value: '1' + - name: OCI__START_ARGS + value: '' + - name: OCI__ENTRY_SCRIPT + value: /code/train.py + image: ".ocir.io///:" + type: container + + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Creating Main Job Run with following details: + Name: chief + Environment Variables: + OCI__MODE:MAIN + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Creating Job Runs with following details: + Name: worker_0 + Environment Variables: + OCI__MODE:WORKER + -----------------------------Ending dryrun mode---------------------------------- + +.. include:: ../_test_and_submit.rst + +.. _hvd_saving_artifacts: + +.. include:: ../_save_artifacts.rst + +.. code-block:: python + + tf.keras.callbacks.ModelCheckpoint(os.path.join(os.environ.get("OCI__SYNC_DIR"),"ckpts",'checkpoint-{epoch}.h5')) + +**Monitoring the workload logs** + +To view the logs from a job run, you could run - + +.. code-block:: bash + + ads jobs watch oci.xxxx. + +**Profiling** + +You may want to profile your training setup for optimization/performance tuning. Profiling typically provides a detailed analysis of cpu utilization, gpu utilization, +top cuda kernels, top operators etc. You can choose to profile your training setup using the native Pytorch profiler or using a third party profiler such as `Nvidia Nsights `__. + +**Profiling using Tensorflow Profiler** + +`Tensorflow Profiler `_ is a native offering from Tensforflow for Tensorflow performance profiling. + +Profiling is invoked using code instrumentation using one of the following apis. + + `tf.keras.callbacks.TensorBoard `_ + + `tf.profiler.experimental.Profile `_ + +Refer above links for changes that you need to do in your training script for instrumentation. + +You should choose the ``OCI__SYNC_DIR`` directory to save the profiling logs. For example: + +.. code-block:: python + + options = tf.profiler.experimental.ProfilerOptions( + host_tracer_level=2, + python_tracer_level=1, + device_tracer_level=1, + delay_ms=None) + with tf.profiler.experimental.Profile(os.environ.get("OCI__SYNC_DIR") + "/logs",options=options): + # training code + +In case of keras callback: + +.. code-block:: python + + tboard_callback = tf.keras.callbacks.TensorBoard(log_dir = os.environ.get("OCI__SYNC_DIR") + "/logs", + histogram_freq = 1, + profile_batch = '500,520') + model.fit(...,callbacks = [tboard_callback]) + +Also, the sync feature ``SYNC_ARTIFACTS`` should be enabled ``'1'`` to sync the profiling logs to the configured object storage. + +Thereafter, use Tensorboard to view logs. Refer the :doc:`Tensorboard setup <../../tensorboard/tensorboard>` for set-up on your computer. + +.. include:: ../_profiling.rst + +**Other Tensorflow Strategies supported** + +Tensorflow has two multi-worker strategies: ``MultiWorkerMirroredStrategy`` and ``ParameterServerStrategy``. +Let's see changes that you would need to do to run ``ParameterServerStrategy`` workload. + +You can have the following training Tensorflow script for ``ParameterServerStrategy`` saved as ``train.py`` +(just like ``mnist.py`` and ``train.py`` in case of ``MultiWorkerMirroredStrategy``): + + +.. code-block:: python + + # Script adapted from tensorflow tutorial: https://www.tensorflow.org/tutorials/distribute/parameter_server_training + + import os + import tensorflow as tf + import json + import multiprocessing + + NUM_PS = len(json.loads(os.environ['TF_CONFIG'])['cluster']['ps']) + global_batch_size = 64 + + + def worker(num_workers, cluster_resolver): + # Workers need some inter_ops threads to work properly. + worker_config = tf.compat.v1.ConfigProto() + if multiprocessing.cpu_count() < num_workers + 1: + worker_config.inter_op_parallelism_threads = num_workers + 1 + + for i in range(num_workers): + print("cluster_resolver.task_id: ", cluster_resolver.task_id, flush=True) + + s = tf.distribute.Server( + cluster_resolver.cluster_spec(), + job_name=cluster_resolver.task_type, + task_index=cluster_resolver.task_id, + config=worker_config, + protocol="grpc") + s.join() + + + def ps(num_ps, cluster_resolver): + print("cluster_resolver.task_id: ", cluster_resolver.task_id, flush=True) + for i in range(num_ps): + s = tf.distribute.Server( + cluster_resolver.cluster_spec(), + job_name=cluster_resolver.task_type, + task_index=cluster_resolver.task_id, + protocol="grpc") + s.join() + + + def create_cluster(cluster_resolver, num_workers=1, num_ps=1, mode="worker"): + os.environ["GRPC_FAIL_FAST"] = "use_caller" + + if mode.lower() == 'worker': + print("Starting worker server...", flush=True) + worker(num_workers, cluster_resolver) + else: + print("Starting ps server...", flush=True) + ps(num_ps, cluster_resolver) + + return cluster_resolver, cluster_resolver.cluster_spec() + + + def decay(epoch): + if epoch < 3: + return 1e-3 + elif epoch >= 3 and epoch < 7: + return 1e-4 + else: + return 1e-5 + + def get_callbacks(model): + class PrintLR(tf.keras.callbacks.Callback): + def on_epoch_end(self, epoch, logs=None): + print('\nLearning rate for epoch {} is {}'.format(epoch + 1, model.optimizer.lr.numpy()), flush=True) + + callbacks = [ + tf.keras.callbacks.TensorBoard(log_dir='./logs'), + tf.keras.callbacks.LearningRateScheduler(decay), + PrintLR() + ] + return callbacks + + def create_dir(dir): + if not os.path.exists(dir): + os.makedirs(dir) + + def get_artificial_data(): + x = tf.random.uniform((10, 10)) + y = tf.random.uniform((10,)) + + dataset = tf.data.Dataset.from_tensor_slices((x, y)).shuffle(10).repeat() + dataset = dataset.batch(global_batch_size) + dataset = dataset.prefetch(2) + return dataset + + + + cluster_resolver = tf.distribute.cluster_resolver.TFConfigClusterResolver() + if not os.environ["OCI__MODE"] == "MAIN": + create_cluster(cluster_resolver, num_workers=1, num_ps=1, mode=os.environ["OCI__MODE"]) + pass + + variable_partitioner = ( + tf.distribute.experimental.partitioners.MinSizePartitioner( + min_shard_bytes=(256 << 10), + max_shards=NUM_PS)) + + strategy = tf.distribute.ParameterServerStrategy( + cluster_resolver, + variable_partitioner=variable_partitioner) + + dataset = get_artificial_data() + + with strategy.scope(): + model = tf.keras.models.Sequential([tf.keras.layers.Dense(10)]) + model.compile(tf.keras.optimizers.SGD(), loss="mse", steps_per_execution=10) + + callbacks = get_callbacks(model) + model.fit(dataset, epochs=5, steps_per_epoch=20, callbacks=callbacks) + + + + + +``Train.yaml``: The only difference here is that the parameter server train.yaml also needs to have ``ps`` worker-pool. +This will create dedicated instance(s) for Tensorflow Parameter Servers. + +Use the following train.yaml: + +.. code-block:: yaml + + kind: distributed + apiVersion: v1.0 + spec: + infrastructure: + kind: infrastructure + type: dataScienceJob + apiVersion: v1.0 + spec: + projectId: oci.xxxx. + compartmentId: oci.xxxx. + displayName: Distributed-TF + logGroupId: oci.xxxx. + subnetId: oci.xxxx. + shapeName: VM.Standard2.4 + blockStorageSize: 50 + cluster: + kind: TENSORFLOW + apiVersion: v1.0 + spec: + image: "@image" + workDir: "oci://@/" + name: "tf_ps" + config: + env: + - name: WORKER_PORT #Optional. Defaults to 12345 + value: 12345 + - name: SYNC_ARTIFACTS #Mandatory: Switched on by Default. + value: 1 + - name: WORKSPACE #Mandatory if SYNC_ARTIFACTS==1: Destination object bucket to sync generated artifacts to. + value: "" + - name: WORKSPACE_PREFIX #Mandatory if SYNC_ARTIFACTS==1: Destination object bucket folder to sync generated artifacts to. + value: "" + main: + name: "coordinator" + replicas: 1 #this will be always 1. + worker: + name: "worker" + replicas: 1 #number of workers; any number > 0 + ps: + name: "ps" # number of parameter servers; any number > 0 + replicas: 1 + runtime: + kind: python + apiVersion: v1.0 + spec: + spec: + entryPoint: "/code/train.py" #location of user's training script in the container image. + args: #any arguments that the training script requires. + env: + +The rest of the steps remain the same and should be followed as it is. + + + diff --git a/docs/source/user_guide/model_training/distributed_training/tensorflow/tensorflow.rst b/docs/source/user_guide/model_training/distributed_training/tensorflow/tensorflow.rst new file mode 100644 index 000000000..de3796a1d --- /dev/null +++ b/docs/source/user_guide/model_training/distributed_training/tensorflow/tensorflow.rst @@ -0,0 +1,18 @@ +========== +Tensorflow +========== + + +**Distributed training with Native TensorFlow** + +TensorFlow is an open-source software framework for distributed deep learning +training. Tensorflow has multiple strategies. The following are supported: + +1. MirroredStrategy +2. MultiWorkerMirroredStrategy +3. ParameterServerStrategy + +.. toctree:: + :maxdepth: 3 + + creating diff --git a/docs/source/user_guide/model_training/distributed_training/troubleshooting.rst b/docs/source/user_guide/model_training/distributed_training/troubleshooting.rst new file mode 100644 index 000000000..67d3be88b --- /dev/null +++ b/docs/source/user_guide/model_training/distributed_training/troubleshooting.rst @@ -0,0 +1,9 @@ +=============== +Troubleshooting +=============== + +Check |location_link| guide for troubleshooting and known issues. + +.. |location_link| raw:: html + + Troubleshooting diff --git a/docs/source/user_guide/model_training/index.rst b/docs/source/user_guide/model_training/index.rst index 877a1d357..5c99f6e75 100644 --- a/docs/source/user_guide/model_training/index.rst +++ b/docs/source/user_guide/model_training/index.rst @@ -1,6 +1,6 @@ -============== -Model Training -============== +============ +Train Models +============ In this section you will learn about model training on the Data Science cloud service using a variety of popular frameworks. This section covers the popular ``sklearn`` framework, along with gradient boosted tree estimators like LightGBM and XGBoost, Oracle AutoML and @@ -19,7 +19,6 @@ TensorBoard provides the visualization and the tooling that is needed to watch a ads_tuner Distributed Training [beta] - ../template/monitoring tensorboard/tensorboard model_evaluation/index model_explainability/index diff --git a/docs/source/user_guide/secrets/index.rst b/docs/source/user_guide/secrets/index.rst index 70596ce27..539f2327d 100644 --- a/docs/source/user_guide/secrets/index.rst +++ b/docs/source/user_guide/secrets/index.rst @@ -1,8 +1,8 @@ .. _secrets-1: -####### -Secrets -####### +################# +Store Credentials +################# Services such as OCI Database and Streaming require users to provide credentials. These credentials must be safely accessed at runtime. `OCI Vault `_ provides a mechanism for safe storage and access of secrets. ``SecretKeeper`` uses Vault as a backend to store and retrieve the credentials. The data structure of the credentials varies from service to service. There is a ``SecretKeeper`` specific to each data structure. diff --git a/docs/source/user_guide/text_extraction/text_dataset.rst b/docs/source/user_guide/text_extraction/text_dataset.rst index c42032697..68dc080bf 100644 --- a/docs/source/user_guide/text_extraction/text_dataset.rst +++ b/docs/source/user_guide/text_extraction/text_dataset.rst @@ -1,3 +1,6 @@ +Text Extraction +--------------- + Convert files such as PDF, and Microsoft Word files into plain text. The data is stored in Pandas dataframes and therefore it can easily be manipulated and saved. The text extraction module allows you to read files of various file formats, and convert them into different formats that can be used for text manipulation. The most common ``DataLoader`` commands are demonstrated, and some advanced features, such as defining custom backend and file processor. .. code-block:: python3 @@ -94,8 +97,7 @@ Both the ``.read_line()`` and ``.read_text()`` methods parse the corpus, convert Each document can have a custom set of metadata that describes the document. The ``.metadata_all()`` and ``.metadata_schema()`` methods allow you to access this metadata. Metadata is represented as a key-value pair. The ``.metadata_all()`` returns a set of key-value pairs for each document. The ``.metadata_schema()`` returns what keys are used in defining the metadata. This can vary from document to document and this method creates a list of all observed keys. You use this to understand what metadata is available in the corpus. -``.read_line()`` ----------------- +**``.read_line()``** The ``.read_line()`` method allows you to read a corpus line-by-line. In other words, each line in a file corresponds to one record. The only required argument to this method is ``path``. It sets the path to the corpus, and it can contain a glob pattern. For example, ``oci://{bucket}@{namespace}/pdf_sample/**.pdf``, ``'oci://{bucket}@{namespace}/20news-small/**/[1-9]*'``, or ``/home/datascience//[A-Za-z]*.docx`` are all valid paths that contain a glob pattern for selecting multiple files. The ``path`` parameter can also be a list of paths. This allows for reading files from different file paths. @@ -164,8 +166,7 @@ This example uses the default engine, which returns an iterator. The ``next()`` 'ok', '/etc/httpd/conf/workers2.properties'] -``.read_text()`` ----------------- +**``.read_text()``** It you want to treat each document in a corpus as a record, use the ``.read_text()`` method. The ``path`` parameter is the only required parameter as it defines the location of the corpus. @@ -315,8 +316,7 @@ there is a difference in the metadata. 'xmp:CreatorTool': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36', 'xmpTPg:NPages': '1'} -``.metadata_schema()`` ----------------------- +**``.metadata_schema()``** As briefly discussed in the ``.metadata_all()`` method section, there is no standard set of metadata across all documents. The ``.metadata_schema()`` method is a convenience method that returns what metadata is available in the corpus. It returns a list of all observed metadata fields in the corpus. Since each document can have a different set of metadata, all the values returned may not exist in all documents. It should also be noted that the engine used can return different metadata for the same document. @@ -355,8 +355,7 @@ The ``text_dataset`` module has the ability to augment the returned records with Examples ======== -``Options.FILE_NAME`` ---------------------- +**``Options.FILE_NAME``** The following example uses ``.option(Options.FILE_NAME)`` to augment to add the filename of each record that is returned. The example uses the ``txt`` for the ``FileProcessor``, and Tika for the backend. The engine is Pandas so a dataframe is returned. The ``df_args`` option is used to rename the columns of the dataframe. Notice that the returned dataframe has a column named ``path``. This is the information that was added to the record from the ``.option(Options.FILE_NAME)`` method. @@ -372,8 +371,7 @@ The following example uses ``.option(Options.FILE_NAME)`` to augment to add the .. image:: figures/sec_filename.png -``Options.FILE_METADATA`` -------------------------- +**``Options.FILE_METADATA``** You can add metadata about a document to a record using ``.option(Options.FILE_METADATA, {'extract': [', '']})``. When using ``Options.FILE_METADATA``, there is a required second parameter. It takes a dictionary where the key is the action to be taken. In the next example, the ``extract`` key provides a list of metadata that can be extracted. When a list is used, the returned value is also a list of the metadata values. The example uses repeated calls to ``.option()`` where different metadata values are extracted. In this case, a list is not returned, but each value is in a separate Pandas column.