Outage of AdvicePay Application and API
Incident Report for AdvicePay
Postmortem

AWS S3 was having issues with very high latency on the day of the outage. Additionally, AdvicePay was using a default HTTP client without a timeout to request firms' logo images from S3. The combination of those two issues led to a denial of service due to starvation of connections on our servers. AWS has since resolved things on their end. To protect against this issue in the future, AdvicePay has implemented timeouts for all HTTP clients that we use for requesting any resource.

Posted May 18, 2023 - 13:02 MDT

Resolved
At 11:37 AM MDT on 4/19/2023 we received an alert from our uptime monitoring service that the AdvicePay application was unreachable. Functionality was restored as of 11:50 AM MDT. We are investigating the underlying cause.
Posted Apr 19, 2023 - 11:37 MDT