Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stress/perf automation #1077

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Stress/perf automation #1077

wants to merge 11 commits into from

Conversation

CIPop
Copy link
Member

@CIPop CIPop commented Oct 1, 2019

DevOps perf/stress automation.

Checklist

  • I have read the contribution guidelines.
  • I added or modified the existing tests to cover the change (we do not allow our test coverage to go down).
  • This pull-request is submitted against the master branch.

Description of the changes

  • Adding devops yaml file
  • Fault injection using Windows Firewall
  • KPI measurement against set parameters
  • Exception handling to simplify pass/fail decision
  • Nuget package testing
  • Handling import/export function internally
  • Better logging

@CIPop
Copy link
Member Author

CIPop commented Oct 1, 2019

Note: includes #1066

@CIPop
Copy link
Member Author

CIPop commented Oct 1, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@CIPop
Copy link
Member Author

CIPop commented Oct 1, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@CIPop
Copy link
Member Author

CIPop commented Oct 2, 2019

File upload tests likely fixed by #1065

@CIPop
Copy link
Member Author

CIPop commented Oct 2, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@CIPop
Copy link
Member Author

CIPop commented Oct 2, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@CIPop
Copy link
Member Author

CIPop commented Oct 4, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@CIPop
Copy link
Member Author

CIPop commented Oct 7, 2019

Unrelated multi-connection test failures in Canary.
Known horton failure likely due to pytest (under investigation).

@CIPop CIPop mentioned this pull request Oct 16, 2019
@CIPop
Copy link
Member Author

CIPop commented Mar 24, 2020

/azp run

@azure-pipelines
Copy link

Commenter does not have sufficient privileges for PR 1077 in repo Azure/azure-iot-sdk-csharp

@CIPop
Copy link
Member Author

CIPop commented Mar 24, 2020

@drwill-ms @prmathur-microsoft @abhipsaMisra what's the status on this one?
Please let me know by EOW if you would still want to take it or if I should close and keep it in my CS branch. I'll lock the branch on my side either way to ensure work isn't deleted by mistake.

@abhipsaMisra
Copy link
Member

@CIPop , I am not able to update your branch to pull changes from master. Could you update your branch; I'll then run the E2E tests and that'll give us an idea if we need to update this PR to incorporate the changes we've made to the SDK since then.

@CIPop
Copy link
Member Author

CIPop commented May 13, 2020

I am not able to update your branch

Ah, sorry about that - I've protected my branch to avoid deleting it by mistake. I have updated it locally, will unlock then push new changes.

@abhipsaMisra
Copy link
Member

/azp run csharp-westcentralus

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@abhipsaMisra
Copy link
Member

@CIPop Hey Chris, do you see vsts/vstsPerf.yaml being used to set up a different pipeline; one that will run these perf tests as a part of our CI/ nightly run?

@CIPop
Copy link
Member Author

CIPop commented May 21, 2020

@CIPop Hey Chris, do you see vsts/vstsPerf.yaml being used to set up a different pipeline; one that will run these perf tests as a part of our CI/ nightly run?

My recommendation would be to run them nightly and monitor the results or tune the KPI parameters to avoid perf regressions (i.e. with each perf enhancement).
At a minimum you could run the tests prior to every release and compare results with the prior runs. This though won't give you proper failure analysis for intermittent issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants