Start Time: 10/22/2020 9:45 am
Maintenance Completed: 11/02/2020 12:00 pm
We are investigating sporadic system-wide slowness. We are unable to determine the root cause at this moment.
Issue Symptoms: Degradation
Service Type: Production
Comments:
10/23/20, 10am - A faulty link was identified between two networking devices. The link was one of two that connected the devices, so it was shut down administratively. That has resolved the immediate issue. We will leave this posting up until lunch in the event the issue returns. Technicians are being dispatched to test the cable and replace is necessary.
10/23/20, 8:30am - The networking vendor did not immediately identify an issue. Technicians from all affected areas are regrouping this morning to continue troubleshooting.
10/22/20, 8:15pm - Our investigation throughout the day has directed us towards a network device that services both our storage and virtualization environments. A support call has been opened with our network vendor to investigate the degradation.
10/22/20, 11:50am - We will perform an emergency update to the storage system at 6pm tonight. We have modified some settings as suggested by the vendors and other resources to alleviate the degradation.
10/22/20, 11:00am - We have received some recommendations from the storage vendor. The virtualization vendor escalated the ticket to their storage team. Our monitoring has not indicated additional slowness since 10am.
10/22/20, 10:25am - On the phone with vendor support with our virtualization vendor and storage vendor. We are still investigating.
Contact the ITS Service Desk for more information or to report additional impacts.
