We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.

Type of Event:
S2 – Visitor (PXC) dashboard intermittent performance Issues

Services/Modules Impacted:
Visitor Dashboard

Root Cause:
The servers were operating at full capacity when the service job was simultaneously handled by both API servers, leading to an increase in server traffic. This situation temporarily saturated the servers, causing them to drop incoming connections. Additionally, the extra load extended the completion time for existing requests, which intermittently affected dashboard performance.

Remediation:
Eptura proactively restarted the API services to address an intermittent issue, providing immediate relief. This strategic approach allowed us to conduct an in-depth investigation to devise a more permanent solution. We identified a specific configuration issue within the API servers, which our CloudOps team promptly addressed, optimizing the service configurations to prevent jobs from being simultaneously picked up by the API servers. This enhancement ensures smoother operations and enhanced reliability.

Timeline:

All times listed in UTC
9:40 a.m.: Monitoring alerts indicated that API services were not responding.

10: 15 a.m.: Eptura restarted the API services and restored the functionality.

12:06 p.m.: An incident was reported, prompting Eptura to start the investigation.

13:33 p.m.: Eptura updated the Visitor status page to reflect the incident and investigation.

15:00 p.m.: Eptura restarted the services but in vain. Eptura engaged Micorosft’s Network team to aid the investigation.

16:00 p.m.: Eptura deployed additional API nodes and restarted services to resolve the issue.

16:18 p.m.: System performance remained impacted due to a backlog of pending API requests.

18:30 p.m.: Eptura confirmed that the backlog cleared and resumed monitoring.

5:30 a.m.: Monitoring alerts indicated that API services were not responding.

07:15 a.m.: Eptura team restarted the services on API nodes to resolve the issue and resumed monitoring.

19:44 p.m.: Following a successful monitoring period, Eptura marked the incident as resolved.

Total Duration of Event:
8 hours 35 minutes

Preventive Action:
We have enhanced our infrastructure by provisioning additional servers to efficiently manage the daily requests. Furthermore, the Eptura CloudOps team has implemented advanced internal monitoring systems. These systems are designed to better monitor errors and dropped connections from a load balancer perspective, enabling us to take swift and effective action whenever necessary. This proactive approach ensures a smoother and more reliable service experience.