update examples for CLI (v2) public preview refresh (Azure#890)

* update setup to use preview CLI

* Create setup.sh

* update schemas to preview

* Update setup.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* add environment examples (Azure#736)

* Create docker image example

* Rename docker image example to docker_image.yml

* Create Dockerfile

* Create docker_context.yml

* refactor and updates

* update creds, readme, workflows

* fix

* remove workflow

* fix mistake

* fix

Co-authored-by: Cody <[email protected]>

* Update docker-context.yml (Azure#739)

* Update setup.sh

* Update setup.sh

* Update setup.sh

* Update setup.sh

* major jobs refactor; add pipeline job (Azure#745)

* add basic and nyc taxi pipelines samples under jobs

* major refactor

* minor edit

* minor edit

* save

* remove sweep dir

* update inputs outputs and compute for cli 0.7

* update component yamls nyc taxi example

Co-authored-by: Sharmeelee Bijlani <[email protected]>

* Create simple-compute-cluster.yml (Azure#748)

* update compute cluster examples (Azure#752)

* update compute examples

* shorten name

* low_priorty -> lowpriority

* add locaiton example

* add compute-instance

* remove instances from testing

* refactor jobs; add simple pipeline examples (Azure#753)

* refactor jobs

* save

* pipeline edits

* switch to canary

* switch to eastus

* fully test

* switch back to master

* reduce python-sdk testing freq; cli and setup cleanup  (Azure#759)

* repo health

* rerun readme

* environment example updates (Azure#760)

* remove version pinning; add version pinning

* edits

* Update setup.sh

* add workflow_dispatch: to cli workflows (Azure#761)

* touch setup.sh

* add workflow_dispatch: to cli jobs

* Update setup.sh

* Update setup.sh

* fix low priority compute example (Azure#762)

* Update setup.sh

* public preview refresh changes (Azure#750)

* fixes

* changes

* revert setup.sh changes

* latest cli

* cli changes

* update dir structure

* update name

* update cli version

* cli version

* cli version

* cleanup

* update comments

Co-authored-by: Cody <[email protected]>

* Update setup.sh

* drop _job (Azure#772)

* Add files via upload

* fix validation scripts for user specified identity and system assigned identity (Azure#747)

* fix sai + uai scripts

* remove set x

* fix workspace name

* fix arg

* fix rest

* add sku

* change credentials to mir

* touch

* add check

* add extra line ending

* add object info to role assignment

* forgotten vars

* fix role assignment

* fix other role assignments

* use latest version of sdk

* fix arm calls

* remove hardcoded vals

* fix hardcoded values

* replace endpoint name with var

* add euap location

* remove pined version of cli

* fix sai role assignment

* update credential logic in readme for uai, sai

* format code

* fixed comments

* cleaned up

* cleanup

* revert

* fix tags

* fix uai

Co-authored-by: max <[email protected]>
Co-authored-by: Cody <[email protected]>

* clean up nyc taxi pipeline (Azure#773)

* update sweep timeout (Azure#778)

* pin badges to main branch (Azure#779)

* pin badges to main branch

* save

* switch testing to master

* manually update schemas

* Update setup.sh

* Update PyTorch example (Azure#791)

* Update setup.sh

* add cifar-10 pipeline example (Azure#794)

* save

* save - training updates

* remove cifar from untested jobs

* save

* save

* switch setup to eastus

* set CR to false

* fix some jobs

* update readme

* save

* save

* fix setup

* run readme

* add data copy script

* edit

* edit

* save

* save

* fix

* save

* edit cifar loc

* fix

* fix

* Update setup.sh

* batch GA samples (Azure#722)

* batch score GA samples

* update batch

* update batch

* update batch sample

* update anon env

* move non mlflow up

* update test flow

* clean up batch sample

* update batch sample

* update schema ref

* rename yml files

* update yml file name

* minor update

* Update setup.sh

* Update workspace CLI examples (Azure#767)

* Update workspace CLI examples

* updating examples per PR feedback

* property ordering and update app insights attr fix

* include example on adding private link endpoint

* include private link example

* simplify ple example

* disabled in capital and consistent resource suffix

* consisten resource names

* save

* fix

* add ws delete

* fixes

* fix

* fix

* remove workspace script

Co-authored-by: Cody <[email protected]>

* add diabetes data

* fix

* fix

* save

* Fixed typo in snippet (Azure#796)

* pipelines-with-components basic examples (Azure#777)

* pipelines-with-components

* data samples

* tf hello world

* create components and dataset in repo setup

* pytorch hello world

* update readme; format code

* update create-datasets.sh

* fix readme; remove unused job; run readme

* add missing files

* setup updates

Co-authored-by: Cody <[email protected]>

* create variables (Azure#799)

* local endpoint test script (Azure#804)

* script for local endpoint

* local endpoint

* Added Dataset and Datastore YAML samples. (Azure#800)

* Added Dataset and Datastore YAML samples.

* Updated README files.

* renames and minor updates

* Updated Dataset samples README.

Co-authored-by: Cody <[email protected]>

* standardization and fixes  (Azure#805)

* update schemas

* update assets

* update resources

* update setup-repo

* save

* udpate assets

* update run logic  (Azure#806)

* update schemas

* update assets

* update resources

* update setup-repo

* save

* udpate assets

* update run logic

* omit run-job.sh

* print job uri

* remove workflow

* edit run-job.sh

* update run-job.sh

* update run-job.sh

* fix cifar pipeline (Azure#809)

* Update setup.sh

* ; => : (Azure#812)

* model-dir => model_dir (Azure#813)

* update run-job.sh

* mount => ro_mount (Azure#814)

* fix nyc-taxi pipeline example (Azure#815)

* ever feel stupid? I feel stupid

* fix 4d_dataset_input (Azure#816)

* clean up batch endpoints (Azure#817)

* switch RG (Azure#818)

* Update setup.sh

* format code

* update jobs and things (Azure#822)

* save

* save

* save

* save

* save

* save

* save

* save

* format code

* remove cli/jobs/basics/data

* save

* save

* save

* update

* fix

* minor edit

* edit

* switch to eastus

* format code

* Add functional Batch Endpoint REST sample (Azure#820)

* add batch score rest sample

* add git workflow

* update REST sample

* update sample

* Functional October batch REST sample -- score from Dataset failing on request

* Clean up script

* run readme

* Fix some quoting issues

* reset location

* Fix modelUri

* Fail early if job submit failed

* Reset readme.py

* Fix location to be based on workspace

* Add other terminal state

Co-authored-by: Tracy Chen <[email protected]>
Co-authored-by: Cody <[email protected]>

* minor edits

* save

* update

* job updates (Azure#824)

* save

* updates

* minor updates

* minor update

* fix cifar dataset

* Update hello-iris-file.yml

* fixes  (Azure#827)

* fixes

* fix

* fix

* add docker context job example for sklearn (Azure#828)

* add docker context job example for sklearn

* run readme

* change image

* Add missing section opener (Azure#829)

Missing opening comment for create_environment

* minor edit

* remove arc examples (Azure#830)

* save

* add hello-code

* autoscale (Azure#832)

* fix unique output (Azure#831)

* fix unique output

* fix unique output

* beautify json

* update REST invoke

* fix typo

* remove delete prompt

* unique output folder

* add deployment delete

* consistency edits (Azure#834)

* --set experiment_name in train.sh (Azure#835)

* fix instance (Azure#836)

* minor consistency edits (Azure#837)

* removing aks sample from cli-preview (Azure#844)

* Update torchserve BYOC sample (Azure#845)

* add notebooks via papermill job examples (Azure#847)

* add notebooks via papermill job examples

* output notebook to ./outputs/

* fix tag (Azure#848)

* byoc tfserving sample (Azure#853)

* byoc tfserving sample

* deployment name fix

* r-byoc-update (Azure#854)

* Update setup.sh

* update

* update

* update vm attach example (Azure#855)

* Revert "update vm attach example" (Azure#856)

This reverts commit f3a7efe.

* add env version for component examples (Azure#858)

* fix typo in directory; rw-mount to rw_mount (Azure#859)

* fix spark example (Azure#857)

* fix spark example

* save

* try pin pip

* ?

* try older pip

* pin image

* format code

* update rest sample (Azure#861)

* update rest sample

* fix format

* fix typo

* fix output path

* update outputdataset to output

* update output

* move dask to named output

* add tag for wait (Azure#866)

* Fixing closing region tag (Azure#867)

read_condafile section tags were not correctly closed.

* swap install steps in setup.sh

* edit comment

* run black

* remove CIs from refresh testing (Azure#873)

* add defaults list to misc script (Azure#874)

* https => wasbs for dask folder example (Azure#875)

* https => wasbs for dask folder example

* test moving container

* Adding MLflow sample for online endpoints (Azure#872)

* Adding MLflow sample for online endpoints

* run readme.py

* run black

* Revert "run black"

This reverts commit b649bb5.

* updating sh file

* Adding MLflow model deployment sample

* Updating files to be consistent

* updating .sh file

* updating deployment YAML and .sh file

* adding debug flag for model delete

* updating YAML and .sh script

* updating .sh script

Co-authored-by: Cody <[email protected]>

* Update triton samples to ncd - single model (Azure#876)

* Update triton samples to ncd - single model

* minor spelling error

* addressing build failures -1

* addressing build failures -2

* addressing missing dependencies

* adding missing dependency 2

* update pip install tritonclient[http]

* addressing build failures

* minor update to tags (Azure#877)

* remove extra triton examples (Azure#881)

* remove flaml notebook action

* add ssh settings to vm attach (Azure#883)

* remove .schemas; consistency edit to scripts (Azure#887)

* remove .schemas

* consistency edit

* remove unused workflow

* switch cli install to public

* remove broken job (Azure#892)

* fix merge issues

* fix job logic

Co-authored-by: Saachi Gopal <[email protected]>
Co-authored-by: Sharmeelee Bijlani <[email protected]>
Co-authored-by: vijetajo <[email protected]>
Co-authored-by: Sethu Raman <[email protected]>
Co-authored-by: Bala P V <[email protected]>
Co-authored-by: max markin <[email protected]>
Co-authored-by: max <[email protected]>
Co-authored-by: Paul Shealy <[email protected]>
Co-authored-by: Tracy Chen <[email protected]>
Co-authored-by: Dennis Eikelenboom <[email protected]>
Co-authored-by: Larry O'Brien <[email protected]>
Co-authored-by: Manoj Bableshwar <[email protected]>
Co-authored-by: Yogi Pandey <[email protected]>
Co-authored-by: Christopher Pirillo <[email protected]>
Co-authored-by: Larry Franks <[email protected]>
Co-authored-by: Shivani Santosh Sambare <[email protected]>
Co-authored-by: Daniel Schneider <[email protected]>

Author: 18 people

.github/workflows/cli-assets-environment-python-ml-basic-cpu.yml .github/workflows/cli-assets-dataset-cloud-file-https.yml

@@ -1,5 +1,6 @@
-name: cli-assets-environment-python-ml-basic-cpu
+name: cli-assets-dataset-cloud-file-https
 on:
+  workflow_dispatch:
   schedule:
     - cron: "0 0/4 * * *"
   pull_request:
@@ -8,8 +9,8 @@ on:
       - cli-preview
       - releases/current
     paths:
-      - cli/assets/environment/python-ml-basic-cpu.yml
-      - .github/workflows/cli-assets-environment-python-ml-basic-cpu.yml
+      - cli/assets/dataset/cloud-file-https.yml
+      - .github/workflows/cli-assets-dataset-cloud-file-https.yml
       - cli/setup.sh
 jobs:
   build:
@@ -20,11 +21,11 @@ jobs:
     - name: azure login
       uses: azure/login@v1
       with:
-        creds: ${{secrets.AZ_AE_CREDS}}
+        creds: ${{secrets.AZ_CREDS}}
     - name: setup
       run: bash setup.sh
       working-directory: cli
       continue-on-error: true
     - name: create asset
-      run: az ml environment create -f assets/environment/python-ml-basic-cpu.yml
+      run: az ml dataset create -f assets/dataset/cloud-file-https.yml
       working-directory: cli

.github/workflows/cli-assets-dataset-cloud-file-wasbs.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-cloud-file-wasbs
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/cloud-file-wasbs.yml
+      - .github/workflows/cli-assets-dataset-cloud-file-wasbs.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/cloud-file-wasbs.yml
+      working-directory: cli

.github/workflows/cli-assets-model-lightgbm-iris.yml .github/workflows/cli-assets-dataset-cloud-file.yml

@@ -1,5 +1,6 @@
-name: cli-assets-model-lightgbm-iris
+name: cli-assets-dataset-cloud-file
 on:
+  workflow_dispatch:
   schedule:
     - cron: "0 0/4 * * *"
   pull_request:
@@ -8,8 +9,8 @@ on:
       - cli-preview
       - releases/current
     paths:
-      - cli/assets/model/lightgbm-iris.yml
-      - .github/workflows/cli-assets-model-lightgbm-iris.yml
+      - cli/assets/dataset/cloud-file.yml
+      - .github/workflows/cli-assets-dataset-cloud-file.yml
       - cli/setup.sh
 jobs:
   build:
@@ -20,11 +21,11 @@ jobs:
     - name: azure login
       uses: azure/login@v1
       with:
-        creds: ${{secrets.AZ_AE_CREDS}}
+        creds: ${{secrets.AZ_CREDS}}
     - name: setup
       run: bash setup.sh
       working-directory: cli
       continue-on-error: true
     - name: create asset
-      run: az ml model create -f assets/model/lightgbm-iris.yml
+      run: az ml dataset create -f assets/dataset/cloud-file.yml
       working-directory: cli

.github/workflows/cli-assets-dataset-cloud-folder-https.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-cloud-folder-https
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/cloud-folder-https.yml
+      - .github/workflows/cli-assets-dataset-cloud-folder-https.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/cloud-folder-https.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-cloud-folder-wasbs.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-cloud-folder-wasbs
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/cloud-folder-wasbs.yml
+      - .github/workflows/cli-assets-dataset-cloud-folder-wasbs.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/cloud-folder-wasbs.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-cloud-folder.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-cloud-folder
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/cloud-folder.yml
+      - .github/workflows/cli-assets-dataset-cloud-folder.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/cloud-folder.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-iris-csv-example.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-iris-csv-example
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/iris-csv-example.yml
+      - .github/workflows/cli-assets-dataset-iris-csv-example.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/iris-csv-example.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-local-file.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-local-file
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/local-file.yml
+      - .github/workflows/cli-assets-dataset-local-file.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/local-file.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-local-folder.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-local-folder
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/local-folder.yml
+      - .github/workflows/cli-assets-dataset-local-folder.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/local-folder.yml
+      working-directory: cli

.github/workflows/cli-assets-dataset-public-file-https.yml

@@ -0,0 +1,31 @@
+name: cli-assets-dataset-public-file-https
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/dataset/public-file-https.yml
+      - .github/workflows/cli-assets-dataset-public-file-https.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml dataset create -f assets/dataset/public-file-https.yml
+      working-directory: cli

.github/workflows/cli-assets-environment-docker-context.yml

@@ -0,0 +1,31 @@
+name: cli-assets-environment-docker-context
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/environment/docker-context.yml
+      - .github/workflows/cli-assets-environment-docker-context.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml environment create -f assets/environment/docker-context.yml
+      working-directory: cli

.github/workflows/cli-assets-environment-docker-image-plus-conda.yml

@@ -0,0 +1,31 @@
+name: cli-assets-environment-docker-image-plus-conda
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0/4 * * *"
+  pull_request:
+    branches:
+      - main
+      - cli-preview
+      - releases/current
+    paths:
+      - cli/assets/environment/docker-image-plus-conda.yml
+      - .github/workflows/cli-assets-environment-docker-image-plus-conda.yml
+      - cli/setup.sh
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v2
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        creds: ${{secrets.AZ_CREDS}}
+    - name: setup
+      run: bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: create asset
+      run: az ml environment create -f assets/environment/docker-image-plus-conda.yml
+      working-directory: cli