Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add post processors for simple crash test and chaining inputs #289

Merged
merged 1 commit into from
Dec 13, 2023

Conversation

tylergu
Copy link
Member

@tylergu tylergu commented Dec 13, 2023

Backporting post processors for simple crash test and input chaining from the Anvil evaluation.
The simple crash test is still a very incomplete implementation, where it relies on the operator's support to parse the ConfigMap and voluntarily crash.

The input chaining takes an existing testrun and chain the inputs into a sequence of inputs.
Follow up the previous PR: #288

Copy link
Contributor

Coverage

Coverage Report
FileStmtsMissCoverMissing
acto
   __main__.py88880%1–175
   common.py3549972%97–98, 102–116, 119–126, 129, 132, 136, 138, 140, 142, 144, 148, 153–162, 203, 236, 249, 291, 315–316, 325, 327, 330, 334–343, 363–366, 415, 418, 432, 460–461, 498–503, 505–507, 514–517, 521, 537–538, 544–555, 614–623, 634–639
   deploy.py887119%15–41, 47–55, 59, 66–116, 124–130, 133–138
   engine.py5374919%40–57, 64–80, 94–197, 209–235, 238–296, 312–435, 439–540, 550–575, 579–587, 590–603, 628–746, 749–838, 842–933
   exception.py220%1–2
   oracle_handle.py241346%15–18, 26–27, 39–46, 54
   reproduce.py1228134%30–40, 48–52, 58–59, 72–93, 96, 99, 102, 106–108, 111, 114, 122–123, 126, 130, 134, 138, 142–176, 179–189, 192–225
   serialization.py301647%19–27, 33–39
   snapshot.py622658%32, 48, 58–79, 82–86, 89
acto/checker
   checker.py13285%12, 19
   checker_set.py571279%41–42, 67–78
   test_checker.py1621526%13–246
acto/checker/impl
   crash.py30293%14, 16
   health.py52590%37, 58–61, 76, 92
   kubectl_cli.py29486%32–33, 44–46
   operator_log.py24292%24, 32
   state.py2255177%55, 63–64, 69–93, 104–105, 133, 140, 147–149, 166, 265, 288, 317, 319–323, 329–336
   state_compare.py76396%62, 73, 80
   state_condition.py391367%18, 24, 29–37, 47, 53
acto/input
   get_matched_schemas.py542259%12, 47–51, 55–74
   input.py60948920%30–37, 74, 92, 112, 123–126, 129, 159–171, 174–185, 189, 193, 200, 206, 210–389, 410–424, 432–437, 441–454, 463, 479, 482–489, 501, 503, 505–507, 510–523, 528, 536–547, 550–556, 564–591, 595–899, 922–935, 939–986
   testcase.py552064%40–50, 53, 56, 59, 62–66, 95–96, 100, 103, 109, 116, 119
   testplan.py18313725%14–24, 27–29, 32, 35–42, 45, 48–67, 70–74, 78, 82, 85–91, 99–107, 110–121, 124, 127, 130, 133, 136–138, 141–151, 154–164, 167, 174–185, 188–194, 200, 203–219, 222, 225, 231, 239–244, 247, 250, 253, 259–260, 263–269, 272, 275, 278
   value_with_schema.py33721835%16, 20, 24, 28, 32, 36, 40, 54, 58, 61–67, 70–76, 86–114, 117–124, 128–137, 141–149, 152–156, 159, 162, 166, 180, 183, 186–192, 195–201, 211–230, 233–240, 243, 247–256, 260–272, 275–279, 282, 285, 289, 301, 307, 312, 314, 316, 320, 324, 329–333, 337–340, 349–359, 362–369, 373–376, 380–385, 388–391, 405, 408–412, 416, 421–428, 431–434, 437–439, 442–445, 448–451, 462, 465, 468, 485–498
   valuegenerator.py62038937%20, 24, 28, 32, 43, 47, 51, 55, 76–86, 90–104, 107, 110, 114, 117, 121–130, 133–139, 142, 145, 148, 151, 154, 157–167, 194–199, 202–213, 216, 219, 223, 226–231, 234–237, 240, 243–248, 251–254, 257, 260, 263, 266, 269, 273–282, 285, 288, 291, 294–304, 331–343, 346–347, 350, 353, 357, 360–365, 368–371, 374, 377–382, 385–388, 391, 394, 397, 400, 403, 406, 409–419, 450–482, 485–496, 499–502, 505–508, 512–520, 523, 526, 529, 532, 535, 538–548, 573–589, 592–609, 612, 615, 619–621, 624–628, 631–632, 635–644, 647–653, 656–657, 660–670, 673, 676, 679, 682, 685, 688–698, 710–711, 714–724, 727–730, 733–736, 740, 748, 751–752, 755–765, 768–771, 774–777, 781, 794–797, 800–814, 817, 820, 824, 827–832, 835, 838, 841–846, 849, 852, 855, 858, 861–871, 881, 884, 887, 890, 894, 913, 919, 931, 933, 936–940, 943–946, 951, 955, 957, 961–962
acto/input/known_schemas
   base.py531375%17–18, 28, 37, 46–47, 56–57, 66, 75, 84–85, 93
   cronjob_schemas.py763357%13, 16–19, 22, 25, 36–39, 42–47, 50, 53, 59, 62–65, 68, 71, 82, 85–90, 93, 96, 113, 117–119, 131, 137, 140
   deployment_schemas.py592558%16, 22–27, 30–32, 35, 38, 54–57, 65–67, 70, 78–81, 91, 94
   known_schema.py753948%28, 31–34, 37, 43, 46–48, 51, 54, 81–84, 102–113, 117–135
   pod_disruption_budget_schemas.py562261%14–17, 21, 25–27, 30, 41–44, 48, 54, 57, 68–71, 81, 84
   pod_schemas.py79727166%16–19, 23, 28, 32, 40–43, 47, 51–53, 61–64, 68, 73, 83, 92, 151, 156, 160, 167, 171, 178, 182, 238, 242, 247, 251, 294, 298, 303, 307, 335, 338, 341, 344, 347, 350, 353, 356, 359, 362, 365, 368, 393, 397, 400–405, 408, 414, 417, 420, 428, 431, 434, 437, 445, 453, 481, 488–494, 503, 507, 539, 543, 572, 575, 578, 581, 584, 587, 590, 619, 623, 626–631, 634, 642, 645, 648, 664–667, 670–672, 675, 684, 690, 693–696, 699, 702, 713–714, 717–719, 722, 725, 736, 739, 742, 745–748, 752, 756–758, 761–765, 768, 774, 777, 780, 783, 786, 789, 792, 795, 798, 801, 804, 807, 810, 813, 816, 840, 845, 849–857, 860, 866, 869, 872, 875, 878, 881, 884, 887, 890, 893, 896, 899, 902, 905, 908, 929, 933–935, 938–943, 946, 959, 964, 974, 980, 986, 989, 992, 1032, 1036–1038, 1041, 1054, 1059, 1065, 1068–1071, 1074, 1077, 1088–1091, 1094–1096, 1102, 1108, 1111–1114, 1117, 1120, 1133–1136, 1139–1141, 1147, 1153, 1156–1159, 1162, 1165, 1176–1179, 1182–1184, 1190, 1196, 1199–1202, 1205, 1212, 1215–1217, 1223, 1238, 1243, 1247, 1274, 1278, 1291, 1296, 1302, 1305, 1308, 1314, 1317, 1320–1322, 1325, 1342, 1345, 1348, 1374, 1378–1381, 1384, 1402, 1408, 1411, 1414, 1421, 1427, 1469, 1473
   resource_schemas.py1495066%15–19, 22, 25, 28–32, 35, 38, 45, 49, 54, 57–59, 62, 76, 78, 82, 96, 102–105, 108, 111, 121, 134, 147, 154, 159–162, 165, 173–176, 179, 187, 194, 199, 213, 231, 236
   service_schemas.py1786762%13, 16, 19, 25–30, 33, 36, 42, 45–48, 51, 54, 64–67, 70–73, 79, 85, 88–91, 94, 101–102, 105–108, 111, 114, 127, 142, 180, 184, 208, 214, 217, 220, 228–231, 235, 240, 244–246, 249, 257–258, 261, 264, 277–280, 284, 288–290, 293
   statefulset_schemas.py1866167%15–18, 21, 31–36, 42, 49–53, 56–58, 61–63, 66–70, 73–75, 78–80, 83, 90–93, 99–102, 105, 132, 149, 154, 158, 164, 167–170, 173, 176, 190–193, 196–201, 207, 225, 230, 234, 262, 266, 290
   storage_schemas.py1797956%13, 16, 25–30, 33, 36, 42, 48–53, 59, 67–70, 74, 79–80, 83, 89, 92–95, 98, 104–105, 108–111, 114–116, 122, 130–131, 135, 140, 145–148, 154–155, 158, 164, 181–184, 193, 197, 203–205, 211, 214, 228–231, 244, 250–252, 255, 258, 266–269, 273, 277–279, 282
acto/kubectl_client
   kubectl.py231822%8–14, 23–29, 37–44
acto/kubernetes_engine
   base.py532258%30, 34, 38, 42, 46, 49–67, 79
   k3d.py856820%17, 21–39, 45, 54–78, 81–97, 100–110, 117–133, 137–139
   kind.py993268%51–53, 57–58, 84, 90, 99–102, 108–112, 115–116, 119–131, 139, 144, 147, 158
acto/monkey_patch
   monkey_patch.py792470%9, 18, 36, 39, 41, 45–56, 76, 90–95, 106
acto/parse_log
   parse_log.py781976%71–74, 84–86, 89–91, 96–98, 116–124
acto/post_process
   post_diff_test.py40723542%64–65, 68, 78, 93–94, 133–134, 205, 215, 219, 224–233, 268, 270, 272–276, 281–298, 306–314, 317–342, 349–358, 361–423, 437–441, 461, 469–501, 504–521, 525–574, 586, 590, 596–597, 599–620, 630–666
   post_process.py1042279%51, 55, 67, 77–88, 100, 103, 138–142, 146, 150, 158, 162
   test_post_process.py28196%53
acto/runner
   runner.py32229010%29–73, 76–78, 92–147, 150–159, 162–193, 200–228, 235–258, 261–267, 270–292, 296–301, 312–317, 331–336, 349–441, 446–450, 456–466, 477–507, 512–545
acto/schema
   anyof.py442055%26, 30–38, 41, 44, 47–49, 52, 55–60
   array.py702959%31, 43–52, 57, 59–62, 71–78, 81–83, 86–88, 91, 94, 97, 103
   base.py1025744%13–15, 18–20, 23, 26–37, 40, 43, 46–48, 51–61, 64–74, 77, 80–86, 95, 100, 105, 110, 114, 118, 142, 145–149
   boolean.py271063%16, 20, 23, 26, 29–32, 35, 38
   integer.py26965%17, 21–23, 26, 29, 32, 35, 38
   number.py301163%30–32, 35–37, 40, 43, 46, 49, 52
   object.py1174859%44, 46, 51, 66–75, 80, 83, 94–109, 112–120, 123–126, 129, 132, 148, 151–158, 168
   oneof.py443032%13–19, 22, 25–27, 30–38, 41, 44, 47–49, 52, 55–60
   opaque.py17665%13, 16, 19, 22, 25, 28
   schema.py42881%21, 25, 31–34, 49–50
   string.py27967%25, 29–31, 34, 37, 40, 43, 46
acto/utils
   __init__.py13192%11
   acto_timer.py312229%10–15, 19, 22–33, 38–40, 44–47
   error_handler.py433323%13–30, 38–52, 56–74
   k8s_helper.py645120%21–27, 39–45, 57–61, 73–79, 83–91, 95–102, 106–127
   preprocess.py695816%17–73, 93–141, 147–192
   process_with_except.py990%1–13
   thread_logger.py15380%9, 18, 28
TOTAL8142421348% 

Tests Skipped Failures Errors Time
139 0 💤 0 ❌ 0 🔥 1m 40s ⏱️

@tylergu tylergu merged commit 48eaa35 into main Dec 13, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant