AWS has solved their issues and our Engineering team is ensuring that our queue is clean. Results are now processing and showing in the Schedule Now workflow. We will continue to monitor the situation.
Posted Jun 13, 2023 - 14:39 PDT
Update
AWS team reports: "Many AWS services are now fully recovered and marked Resolved on this event. We are continuing to work to fully recover all services."
Posted Jun 13, 2023 - 14:06 PDT
Update
AWS team reports:
Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard. Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support. We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services.
Posted Jun 13, 2023 - 14:02 PDT
Update
AWS team reports: "We are beginning to see an improvement in the Lambda function error rates. We are continuing to work towards full recovery."
Posted Jun 13, 2023 - 13:42 PDT
Update
AWS team reports: "We are continuing to work to resolve the error rates invoking Lambda functions. We're also observing elevated errors obtaining temporary credentials from the AWS Security Token Service, and are working in parallel to resolve these errors."
Posted Jun 13, 2023 - 13:16 PDT
Update
AWS team reports: "We are continuing to experience increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. We have identified the root cause as an issue with AWS Lambda, and are actively working toward resolution. For customers attempting to access the AWS Management Console, we recommend using a region-specific endpoint (such as: https://us-west-2.console.aws.amazon.com). We are actively working on full mitigation and will continue to provide regular updates."
Posted Jun 13, 2023 - 12:38 PDT
Update
AWS team reports: "We have identified the root cause of the elevated errors invoking AWS Lambda functions, and are actively working to resolve this issue."
We are working to get updates from their team ASAP.
Posted Jun 13, 2023 - 12:16 PDT
Investigating
Some users are reporting seeing "Oh No! We Are Having A Problem Finding Good Results." in the Schedule Now workflow. Our Engineering team is investigating this now.
Posted Jun 13, 2023 - 12:13 PDT
This incident affected: Goodtime Hire (Dashboard (a.goodtime.io)).