Purpose:
Defining the tests and designs Coevolve tests for every new software version we use as recommended for our customers.
Tests listed here are either included for critical functions or due to previous problematic bugs in VMware approved code.
How To Use:
1. Use of this document during testing: This Document as it stands should be copied and titled with the version being tested and as tests are completed marked off by the engineer who conducted the test. If there are known changes a separate KBA should be created and a link to the KBA included in the new document.
i. [Not Implemented] Tests should link to KBA with full test guide.
2. Updates to this document: This document should be updated to match the main criteria for our designs, with stability criteria listed, and if there are particular bugs that need to be re-tested on future versions. The Addition of Bug testing to this document should have a line diagram included and the engineer who added the test criteria.
i. Minimum Stability window should be at least 19 hours or polling rate * 65536 seconds ( a 3 second polling rate = 3*65536 = 196608 seconds = ~2.3 days )
3. De-prioritization of Tests: Any Bug test should be de-prioritized after 4 successful main versions are released without re-occurrence. a test in this state only needs to be tested on each new release train. e.g. if a test succeeds all testing on 5.x.x.x then it can be de-prioritized and only tested again on the first version version of code evaluated for 6.x.x.x, 7.x.x.x and so forth.
Primary Feature and Design Testing:
| Feature | Criticality | Stability Metric | Tests | Succeeded | Tester/s |
|---|---|---|---|---|---|
| Cloud VPN Static Tunnel | Must Pass | Data Reachable | ICMP from CRC LAB to Azure Hub Resources | TRUE/FALSE/WORKAROUND |
|
| Cloud VPN Dynamic Tunnel | Must Pass | Traceroute Length | Trace Route from CRC Lab to Azure Lab Spoke Use of Azure Hub and Gateway must be tested |
||
| HA Stability | Must Pass | Edge Uptime without incident must exceed 7 days | Setup and daily use test VCO monitoring |
||
| HA Failover |
Must Pass |
Edge Hands over appropriately Recovery Time |
Expected HA Failover ( VCO commands or software upgrade) Unexpected HA Failover (Power / Data Loss) |
||
| Client Services |
Should Pass Must Have Workaround |
All services respond for IPv4 |
DHCPv4, DNS Proxy, |
||
| Dynamic Route Sharing | Should Pass Must Have Workaround |
Route share uptime exceeds 3 days | Two Edges sharing Loopbacks 1 sharing via BGP other via OSPF |
||
| SSH Access | Can Fail | Internet and Local SSH tested | SSH to Azure edges for internet, SSH to CRC LAB for Local | ||
| Software Downgrade | Should Pass Must Have Workaround |
Software Package succeeds after < 3 attempts | Downgrade Virtual standalone Azure. Downgrade HA at CRC LAB. |
||
| Analytics Data Capture | Should Pass Should Have workaround |
Data available before < 6 Hours | Have user behind CRC LAB Confirm VCO Stats, Confirm ENI Stats. |
Bug Testing:
| Issue | KBA / Bug ID | Added By and from Version | Stability Criteria | Tests | Tester |
|---|---|---|---|---|---|
| ICMP Probes drop after time period |
BUG - 122528 |
David M. 4.5.2 |
Routes from ICMP probes are reachable after 3 days. |
Setup the Dynamic Routing test but add a ICMP probe to a static route |
|
| 5X0 devices Switch ports unusable |
BUG - 125647 |
David M. 5.2.0.0 |
Link Detection and Traffic flow works without console access. |
Test connectivity to a 5X0 device LAN ports. |
Low Priority Testing:
| Issue | Deprioritized By and from Version | Stability Criteria | Tests | Tester |
|---|---|---|---|---|