{"page":{"id":"01J16DCZR42EPRD7WB12BB78T0","name":"Materialize","url":"https://status.materialize.com/","updated_at":"2026-11-10T22:16:12Z"},"incidents":[{"id":"01M73J7SC61WHPK1GS5ZQP51P1","name":"Cloud Connectivity Issues Due to Authentication Provider Outage","status":"resolved","created_at":"2026-11-10T20:26:20Z","updated_at":"2026-11-10T22:16:12Z","resolved_at":"2026-11-10T22:16:12Z","impact":"major","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01M73RGYKZG515XVCDGMGB7NYS","body":"Authentication has recovered. Console sign-in and new connections using app passwords are fully restored. ","created_at":"2026-11-10T22:16:12Z","display_at":"2026-11-10T22:16:12Z","incident_id":"01M73J7SC61WHPK1GS5ZQP51P1","status":"resolved","updated_at":"2026-11-10T22:16:12Z"},{"id":"01M73KZDG84T8DDN4ANFYK2G23","body":"Our authentication provider has fixed the issue, and console and pgwire logins are recovering. Some users may still see slow or failed logins for a short time. We’re keeping this incident open while we confirm all services are restored.","created_at":"2026-11-10T20:56:43Z","display_at":"2026-11-10T20:56:43Z","incident_id":"01M73J7SC61WHPK1GS5ZQP51P1","status":"monitoring","updated_at":"2026-11-10T20:56:43Z"},{"id":"01M73J7SC6NPA1C6YBMZFK23JS","body":"Starting at approximately 20:20 UTC, an outage at our upstream authentication provider is preventing some users from authenticating to Materialize Cloud. This affects all Cloud regions.\nWhat’s affected\n\n• Logging in to the Materialize console\n\n• Opening new pgwire connections, including connections that use app passwords\n\n• Tools and services that open new connections or reconnect, such as connection poolers, BI tools, dbt runs, and other scheduled jobs. These may see authentication errors or timeouts.\n\nWhat’s not affected\n\n• Clusters are running normally.\n\n• Sources are still ingesting data, and sinks are still writing data, so we don’t expect any impact to data freshness.\n\n• Self-managed deployments are not affected.\n\nWhat you can do\nIf you have pgwire connections that are already open, we recommend not restarting them or the services that use them until this is resolved. New connection attempts may fail until our provider recovers.","created_at":"2026-11-10T20:26:20Z","display_at":"2026-11-10T20:26:20Z","incident_id":"01M73J7SC61WHPK1GS5ZQP51P1","status":"identified","updated_at":"2026-11-10T20:51:05Z"}]},{"id":"01M1PJHPS1ETAZWAGNP4602S80","name":"Degraded availability for Cloud customers in us-west-2 due to upstream AWS credential service disruption","status":"resolved","created_at":"2026-09-04T16:03:09Z","updated_at":"2026-09-04T16:05:18Z","resolved_at":"2026-09-03T21:43:00Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01M1PJHPS15YZB1948KJY25T9M","body":"Between 21:12 UTC and 22:18 UTC on September 3, AWS experienced elevated error rates in its Security Token Service (STS) in the us-west-2 region, which AWS reported on its Health Dashboard. Materialize environments in that region use STS to obtain credentials for object storage access, and requests for those credentials failed or timed out during the event.\n\n \n\nSome environments in the region experienced degraded performance during this window. Query results remained correct throughout, and no data was lost.\n\n\nThe upstream issue was resolved by AWS at 22:18 UTC and all affected environments recovered.","created_at":"2026-09-04T16:03:09Z","display_at":"2026-09-03T21:43:00Z","incident_id":"01M1PJHPS1ETAZWAGNP4602S80","status":"resolved","updated_at":"2026-09-04T16:03:09Z"}]},{"id":"01M0FNXWRHMET88K4MW3KVX2CS","name":"Upstream vendor causing authentication related issues","status":"resolved","created_at":"2026-08-20T13:31:51Z","updated_at":"2026-08-20T14:39:55Z","resolved_at":"2026-08-20T14:39:55Z","impact":"major","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01M0FSTHANN347Z0NER73Q627M","body":"Upstream authentication vendor has fully resolved their incident. All impacted systems have recovered.","created_at":"2026-08-20T14:39:55Z","display_at":"2026-08-20T14:39:55Z","incident_id":"01M0FNXWRHMET88K4MW3KVX2CS","status":"resolved","updated_at":"2026-08-20T14:39:55Z"},{"id":"01M0FQG603V5NEQHNM2SYS9VEY","body":"We’re seeing pgwire authentication recover and are continuing to monitor the situation to ensure service remains stable.","created_at":"2026-08-20T13:59:19Z","display_at":"2026-08-20T13:59:19Z","incident_id":"01M0FNXWRHMET88K4MW3KVX2CS","status":"monitoring","updated_at":"2026-08-20T13:59:19Z"},{"id":"01M0FP8WXBCM5ETVD0BNBJE7W6","body":"We’ve determined that this issue is related to an outage of our authentication provider and is limited to pgwire authentication. Sources and sinks continue to run as expected, and we do not expect this issue to impact data freshness.\n\nWe are continuing to monitor the situation and work toward a resolution.","created_at":"2026-08-20T13:37:52Z","display_at":"2026-08-20T13:37:52Z","incident_id":"01M0FNXWRHMET88K4MW3KVX2CS","status":"identified","updated_at":"2026-08-20T13:37:52Z"},{"id":"01M0FNXWRH16N4YY8YRFNB3MEM","body":"We are seeing issues across the platform potentially caused by an outage of an upstream vendor. We are actively investigating and looking into mitigations.","created_at":"2026-08-20T13:31:51Z","display_at":"2026-08-20T13:31:51Z","incident_id":"01M0FNXWRHMET88K4MW3KVX2CS","status":"investigating","updated_at":"2026-08-20T13:31:51Z"}]},{"id":"01KTM4NWS60D77RM64VXADRZH6","name":"Networking issues affecting source ingestion","status":"resolved","created_at":"2026-06-08T17:32:24Z","updated_at":"2026-06-17T17:05:52Z","resolved_at":"2026-06-09T17:05:00Z","impact":"none","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01KVB8QRYSGYSBCHESTPX84YDX","body":"\nThe network change affecting source ingestion has been reverted and all affected sources have recovered.","created_at":"2026-06-17T17:05:52Z","display_at":"2026-06-09T17:05:00Z","incident_id":"01KTM4NWS60D77RM64VXADRZH6","status":"resolved","updated_at":"2026-06-17T17:06:10Z"},{"id":"01KTM91MZQ311GRBKT5YQG49W3","body":"We have rolled out a fix for the network change that caused delayed or stalled ingestion on some sources. Affected sources should now be reconnecting and catching up. We are monitoring to confirm full recovery and will post a final update once we have verified things are stable.","created_at":"2026-06-08T18:48:44Z","display_at":"2026-06-08T18:48:44Z","incident_id":"01KTM4NWS60D77RM64VXADRZH6","status":"monitoring","updated_at":"2026-06-08T18:48:44Z"},{"id":"01KTM6HGCJXDH9GEYGTJ1WHBC1","body":"We have identified the cause of the stalled ingestion issue affecting some sources, related to a recent network change on our side. A fix is in progress, and adding our current egress IP ranges (\u003chttps://materialize.com/docs/ingest-data/network-security/static-ips/\u003e) to your allowlist will restore ingestion in the meantime.","created_at":"2026-06-08T18:04:58Z","display_at":"2026-06-08T18:04:58Z","incident_id":"01KTM4NWS60D77RM64VXADRZH6","status":"identified","updated_at":"2026-06-08T18:04:58Z"},{"id":"01KTM4NWS63FNKMMRT5VTFWD54","body":"We are currently investigating networking issues that are affecting ingestion for certain sources. Some sources may experience delayed or stalled data ingestion as a result. Our team is actively looking into the cause and we will provide an update as soon as we have more information.","created_at":"2026-06-08T17:32:24Z","display_at":"2026-06-08T17:32:24Z","incident_id":"01KTM4NWS60D77RM64VXADRZH6","status":"investigating","updated_at":"2026-06-08T17:32:24Z"}]},{"id":"01KR2SDG2V6HNRW2RWZ51H0DN7","name":"Degraded performance of us-east-1","status":"resolved","created_at":"2026-05-08T03:17:06Z","updated_at":"2026-05-09T17:53:00Z","resolved_at":"2026-05-09T17:53:00Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01KR6XY2060N75CSNJ9XHC38RR","body":"We are no longer observing impact to Materialize services, and this incident is now resolved.","created_at":"2026-05-09T17:53:00Z","display_at":"2026-05-09T17:53:00Z","incident_id":"01KR2SDG2V6HNRW2RWZ51H0DN7","status":"resolved","updated_at":"2026-05-09T17:53:00Z"},{"id":"01KR2TGE8Z6XZQ0C1F96PY3SNV","body":"We are continuing to monitor as things recover","created_at":"2026-05-08T03:36:11Z","display_at":"2026-05-08T03:36:11Z","incident_id":"01KR2SDG2V6HNRW2RWZ51H0DN7","status":"monitoring","updated_at":"2026-05-08T03:36:11Z"},{"id":"01KR2SDG2V5YQA2G5NGZZXRVXS","body":"Due to an upstream thermal event, some clusters in us-east-1 may be experiencing degraded performance or replica restarts. We have redirected new node provisioning to healthy datacenter targets, and are monitoring the situation.","created_at":"2026-05-08T03:17:06Z","display_at":"2026-05-08T03:17:06Z","incident_id":"01KR2SDG2V6HNRW2RWZ51H0DN7","status":"identified","updated_at":"2026-05-08T03:17:06Z"}]},{"id":"01KJR2B3TGJGTBAPKGRNVXEN9G","name":"Degraded Performance – Cluster Creation for Replicas ≥ 1600cc","status":"resolved","created_at":"2026-03-02T20:02:57Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2026-03-03T15:46:53Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01KJT62YF21TGN0ESKZV16CK28","body":"We have identified and resolved the issue impacting cluster creation for large legacy cc replicas.\n\nCluster provisioning is now functioning normally. M.1 cluster sizes were not affected during this incident.","created_at":"2026-03-03T15:46:53Z","display_at":"2026-03-03T15:46:53Z","incident_id":"01KJR2B3TGJGTBAPKGRNVXEN9G","status":"resolved","updated_at":"2026-03-03T15:46:53Z"},{"id":"01KJRR4MGGA2EDQMJMF2H88MH5","body":"We are continuing to investigate an issue impacting cluster creation for legacy cc replicas sized greater than or equal to 1600cc. Customers may experience delays or see clusters remain in a pending state during provisioning.\n\nWe have confirmed that this issue does not affect M.1 cluster sizes. Only legacy cc clusters are impacted at this time.\n\nOur engineering team is actively working to identify the root cause and implement a mitigation. We will provide further updates as we have more information.","created_at":"2026-03-03T02:23:53Z","display_at":"2026-03-03T02:23:53Z","incident_id":"01KJR2B3TGJGTBAPKGRNVXEN9G","status":"investigating","updated_at":"2026-03-03T02:23:53Z"},{"id":"01KJR2B3TGDVJJG5NSXE7RSTPW","body":"We are currently investigating an issue impacting cluster creation for replicas sized greater than or equal to 1600cc. Customers may experience delays or clusters remaining in a pending state during provisioning. Smaller replica sizes are not affected.\n\nOur engineering team is actively working to identify the root cause and mitigate the issue.","created_at":"2026-03-02T20:02:57Z","display_at":"2026-03-02T20:02:57Z","incident_id":"01KJR2B3TGJGTBAPKGRNVXEN9G","status":"investigating","updated_at":"2026-03-02T20:02:57Z"}]},{"id":"01KABFMYYKG980ERHHV0YH4WM5","name":"Cloudflare Incident Causing Intermittent Connectivity and Auth Failures","status":"resolved","created_at":"2025-11-18T12:39:49Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-11-18T14:57:24Z","impact":"none","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01KABQGWVD64F5A1TV7J0BEKBS","body":"This incident has been resolved.","created_at":"2025-11-18T14:57:24Z","display_at":"2025-11-18T14:57:24Z","incident_id":"01KABFMYYKG980ERHHV0YH4WM5","status":"resolved","updated_at":"2025-11-18T14:57:24Z"},{"id":"01KABFMYYKD0YPNE8MT318CX84","body":"The global Cloudflare incident (\u003chttps://www.cloudflarestatus.com/incidents/8gmgl950y3h7\u003e) has caused a partial connectivity and authentication outage for about ~10 minutes. Systems have recovered since then. We are monitoring the situation.","created_at":"2025-11-18T12:39:49Z","display_at":"2025-11-18T12:39:49Z","incident_id":"01KABFMYYKG980ERHHV0YH4WM5","status":"monitoring","updated_at":"2025-11-18T12:39:49Z"}]},{"id":"01K8RC1FF255XJC0XK2KTC116H","name":"Authentication Service Degradation","status":"resolved","created_at":"2025-10-29T16:15:29Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-10-29T19:49:22Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01K8RR93K73TF7TXZ11BHTCNF3","body":"The authentication provider issue has been fully resolved.","created_at":"2025-10-29T19:49:22Z","display_at":"2025-10-29T19:49:22Z","incident_id":"01K8RC1FF255XJC0XK2KTC116H","status":"resolved","updated_at":"2025-10-29T19:49:22Z"},{"id":"01K8RDRRTKRZXS0E9TZRBM9WRM","body":"Authentication services have been restored, and users should no longer experience login or session issues. We are continuing to monitor the system to ensure stability.","created_at":"2025-10-29T16:45:41Z","display_at":"2025-10-29T16:45:41Z","incident_id":"01K8RC1FF255XJC0XK2KTC116H","status":"monitoring","updated_at":"2025-10-29T16:45:41Z"},{"id":"01K8RC1FF2R4P5QGS6XZR46D02","body":"We are investigating an issue with our authentication provider that is causing degraded performance for login and session management. Users may experience intermittent errors or delays when signing in. ","created_at":"2025-10-29T16:15:29Z","display_at":"2025-10-29T16:15:29Z","incident_id":"01K8RC1FF255XJC0XK2KTC116H","status":"investigating","updated_at":"2025-10-29T16:15:29Z"}]},{"id":"01K8BFCJ6TA0QN5DYFJXH44NJV","name":"Admins unable to access Account Settings page","status":"resolved","created_at":"2025-10-24T16:03:50Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-10-27T02:56:00Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01K8HSG5C2YP5HY334GZE0X9SZ","body":"The issue preventing access to the Account Settings page has been resolved. User management functionality is now fully restored in the console.","created_at":"2025-10-27T02:56:00Z","display_at":"2025-10-27T02:56:00Z","incident_id":"01K8BFCJ6TA0QN5DYFJXH44NJV","status":"resolved","updated_at":"2025-10-27T02:56:00Z"},{"id":"01K8BFCJ6TKAPJKBY6T309ATR8","body":"We’re investigating an issue where the Account Settings page, which renders the admin portal from our authentication provider, is not loading. As a result, admins are currently unable to manage users. Please [contact our support team](https://materialize.com/docs/support/) if you are critically impacted by this issue.","created_at":"2025-10-24T16:03:50Z","display_at":"2025-10-24T16:03:50Z","incident_id":"01K8BFCJ6TA0QN5DYFJXH44NJV","status":"investigating","updated_at":"2025-10-24T21:39:05Z"}]},{"id":"01K80C0EJ9B4F8TQD4V9VHWDBR","name":"Service degraded in us-east","status":"resolved","created_at":"2025-10-20T08:33:09Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-10-20T23:12:52Z","impact":"critical","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01K81YB8JQJXFDNE5CZYN3VT0B","body":"We are seeing full recovery for our us-east-1 region","created_at":"2025-10-20T23:12:52Z","display_at":"2025-10-20T23:12:52Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"resolved","updated_at":"2025-10-20T23:12:52Z"},{"id":"01K81CF3B2QT8NMEKB3GDRDV1A","body":"us-east-1 clusters are now operating with potentially degraded performance. We have implemented several fixes and are monitoring.","created_at":"2025-10-20T18:00:23Z","display_at":"2025-10-20T18:00:23Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"monitoring","updated_at":"2025-10-20T18:00:23Z"},{"id":"01K8137W31M40X1GNKD7CNV5MA","body":"We're still experiencing degraded service in us-east regions.","created_at":"2025-10-20T15:19:09Z","display_at":"2025-10-20T15:19:09Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"monitoring","updated_at":"2025-10-20T15:19:09Z"},{"id":"01K80SC5A9CMWK5MX163TRAKTC","body":"We're recovering from the AWS outage and are monitoring all environments.","created_at":"2025-10-20T12:26:44Z","display_at":"2025-10-20T12:26:44Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"monitoring","updated_at":"2025-10-20T12:26:44Z"},{"id":"01K80N2FKWM2QZNHE1RGBJ998R","body":"We're recovering from the AWS outage and identified some environments requiring additional maintenance and are monitoring all environments.","created_at":"2025-10-20T11:11:33Z","display_at":"2025-10-20T11:11:33Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"monitoring","updated_at":"2025-10-20T11:11:33Z"},{"id":"01K80E7N77CG2QPJGQ3ZXAYCE6","body":"We're impacted by an AWS outage in us-east regions. We're monitoring the situation.","created_at":"2025-10-20T09:12:02Z","display_at":"2025-10-20T09:12:02Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"identified","updated_at":"2025-10-20T09:12:02Z"},{"id":"01K80C93ZREAY0MJ7V20YZ5EG5","body":"We're monitoring an issue with AWS that affects Materialize: [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status \"https://health.aws.amazon.com/health/status\")","created_at":"2025-10-20T08:37:53Z","display_at":"2025-10-20T08:37:53Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"investigating","updated_at":"2025-10-20T08:37:53Z"},{"id":"01K80C0EJ99RF2KRN4HWN9CMBQ","body":"Console and API endpoints unavailable.","created_at":"2025-10-20T08:33:09Z","display_at":"2025-10-20T08:33:09Z","incident_id":"01K80C0EJ9B4F8TQD4V9VHWDBR","status":"investigating","updated_at":"2025-10-20T08:33:09Z"}]},{"id":"01K75J0W053A7X05SYN9FKRC7J","name":"Sink metrics are not showing progress in the console.","status":"resolved","created_at":"2025-10-09T22:38:44Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-10-09T22:51:02Z","impact":"major","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01K75JQC6DK6185DAKF2MVW8Q2","body":"A fix for this has been released and sink metrics are now displaying accurately.","created_at":"2025-10-09T22:51:02Z","display_at":"2025-10-09T22:51:02Z","incident_id":"01K75J0W053A7X05SYN9FKRC7J","status":"resolved","updated_at":"2025-10-09T22:51:02Z"},{"id":"01K75J0W051J0TBEE7CBD938YF","body":"Sink metrics in the Materialize console indicate that no messages are being sent.  Sinks are still making progress and can still be tracked via `mz_internal.mz_sink_metrics` .  The cause has been identified and a fix is in progress.","created_at":"2025-10-09T22:38:44Z","display_at":"2025-10-09T22:38:44Z","incident_id":"01K75J0W053A7X05SYN9FKRC7J","status":"identified","updated_at":"2025-10-09T22:38:44Z"}]},{"id":"01K74TP8J6E653DKMAWSMAR9Y3","name":"Postgres sources stuck in snapshotting","status":"resolved","created_at":"2025-10-09T15:50:59Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-10-09T19:31:27Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01K7579XQT3DBQW75K6G0VVCXN","body":"A fix for this has been released and Postgres snapshotting metrics are now displaying accurately. ","created_at":"2025-10-09T19:31:27Z","display_at":"2025-10-09T19:31:27Z","incident_id":"01K74TP8J6E653DKMAWSMAR9Y3","status":"resolved","updated_at":"2025-10-09T19:31:27Z"},{"id":"01K74VAB37CVYDBPKBCFR1YYRQ","body":"We've confirmed this issue is only affecting Postgres snapshotting metrics and source availability is not impacted. Our engineering team is currently working on a fix.","created_at":"2025-10-09T16:01:57Z","display_at":"2025-10-09T16:01:57Z","incident_id":"01K74TP8J6E653DKMAWSMAR9Y3","status":"identified","updated_at":"2025-10-09T16:01:57Z"},{"id":"01K74TP8J6G82DPKTG7CKA87B4","body":"We've identified an issue that is causing all Postgres sources to appear to be stuck in snapshotting. We believe this issue is isolated to metrics and that there is no impact to source availability. ","created_at":"2025-10-09T15:50:59Z","display_at":"2025-10-09T15:50:59Z","incident_id":"01K74TP8J6E653DKMAWSMAR9Y3","status":"investigating","updated_at":"2025-10-09T15:50:59Z"}]},{"id":"01K18WP6RVX5KT5BDJFRSTNQQ4","name":"New environment creation unavailable","status":"resolved","created_at":"2025-07-28T16:08:42Z","updated_at":"2026-03-11T16:01:53Z","resolved_at":"2025-07-28T20:58:12Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01K19D89REDVT0964JBG2PMQ84","body":"This issue has been resolved. ","created_at":"2025-07-28T20:58:12Z","display_at":"2025-07-28T20:58:12Z","incident_id":"01K18WP6RVX5KT5BDJFRSTNQQ4","status":"resolved","updated_at":"2025-07-28T20:58:12Z"},{"id":"01K18WZ8M0DZ5YA7CHGKGPWN71","body":"The issue affecting environment creation has been fixed. We are now monitoring to ensure full functionality is restored.","created_at":"2025-07-28T16:13:38Z","display_at":"2025-07-28T16:10:00Z","incident_id":"01K18WP6RVX5KT5BDJFRSTNQQ4","status":"monitoring","updated_at":"2025-07-28T16:14:14Z"},{"id":"01K18WP6RVXXNKYGE3HF9XJS83","body":"We've identified an issue that is preventing new environments from being created. Our engineering team is actively working on a fix.","created_at":"2025-07-28T16:08:42Z","display_at":"2025-07-28T16:02:00Z","incident_id":"01K18WP6RVX5KT5BDJFRSTNQQ4","status":"investigating","updated_at":"2025-07-28T16:14:05Z"}]},{"id":"01JXDAVTBJCFY6HA8M98SQP110","name":"Partial Freshness Data in Console Unavailable This Week for the EU regions","status":"resolved","created_at":"2025-06-10T16:30:36Z","updated_at":"2026-03-11T16:01:52Z","resolved_at":"2025-06-13T06:18:30Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JXKZ15PVBGCFVVG025HFB7P5","body":"All freshness data should be available in the Console.","created_at":"2025-06-13T06:18:30Z","display_at":"2025-06-13T06:18:30Z","incident_id":"01JXDAVTBJCFY6HA8M98SQP110","status":"resolved","updated_at":"2025-06-13T06:18:30Z"},{"id":"01JXDAVTBJBTE4N26XHNSEPA9N","body":"Between **June 11th (Wednesday) and June 13th (Friday)**, **users will not see Freshness values in the Console** for environments in the EU regions for the last day. This is due to internal changes affecting the availability of certain metrics. Specifically, the following system tables will start showing null lag values after this period:  mz_internal.mz_wallclock_global_lag and mz_internal.mz_wallclock_lag_history. These null values indicate the object is currently unqueryable due to hydration.","created_at":"2025-06-10T16:30:36Z","display_at":"2025-06-10T16:30:36Z","incident_id":"01JXDAVTBJCFY6HA8M98SQP110","status":"identified","updated_at":"2025-06-12T09:10:18Z"}]},{"id":"01JW6PC2TBX9DY23R5AZ7D0Y40","name":"Potential corruption of source data in cloud environments","status":"resolved","created_at":"2025-05-26T16:21:20Z","updated_at":"2026-03-11T16:01:52Z","resolved_at":"2025-05-27T03:51:21Z","impact":"none","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JW7XVGXS016174A59DMMW7XC","body":"All impacted customers have been notified. Recreating sources resolves any corruption. Working on a retrospective now.","created_at":"2025-05-27T03:51:21Z","display_at":"2025-05-27T03:51:21Z","incident_id":"01JW6PC2TBX9DY23R5AZ7D0Y40","status":"resolved","updated_at":"2025-05-27T03:51:21Z"},{"id":"01JW6PC2TB0MJF537JK0GH2WSB","body":"An issue causing potential data corruption has been identified and resolved. While you may not be impacted, recreating your sources will ensure you do not have any corrupted data.","created_at":"2025-05-26T16:21:20Z","display_at":"2025-05-26T16:21:20Z","incident_id":"01JW6PC2TBX9DY23R5AZ7D0Y40","status":"identified","updated_at":"2025-05-26T16:21:20Z"}]},{"id":"01JVZ2CSRDJCW6MMVQJB4EJED2","name":"Authorization Errors","status":"resolved","created_at":"2025-05-23T17:17:31Z","updated_at":"2026-03-11T16:01:52Z","resolved_at":"2025-05-23T18:55:57Z","impact":"none","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JVZ81163ZRQ7BA9V3QV3AWZP","body":"Authentication has been fully restored","created_at":"2025-05-23T18:55:57Z","display_at":"2025-05-23T18:55:57Z","incident_id":"01JVZ2CSRDJCW6MMVQJB4EJED2","status":"resolved","updated_at":"2025-05-23T18:55:57Z"},{"id":"01JVZ35V56FV6NB766CSMZTEZT","body":"The issues were caused by an incident with our authentication provider. We have confirmed that access is now restored. We are closely monitoring the situation to ensure everything remains stable.","created_at":"2025-05-23T17:31:12Z","display_at":"2025-05-23T17:31:12Z","incident_id":"01JVZ2CSRDJCW6MMVQJB4EJED2","status":"monitoring","updated_at":"2025-05-23T17:31:12Z"},{"id":"01JVZ2CSRDG0ATB576PKFN6GFH","body":"We are currently experiencing authorization issues that may prevent some users from accessing certain services. Our team is investigating the problem and working to resolve it as quickly as possible.","created_at":"2025-05-23T17:17:31Z","display_at":"2025-05-23T17:17:31Z","incident_id":"01JVZ2CSRDJCW6MMVQJB4EJED2","status":"investigating","updated_at":"2025-05-23T17:17:31Z"}]},{"id":"01JQTB11VQNNEHSQNP5062S9BD","name":"Oauth via Google Sign-in Unavailable","status":"resolved","created_at":"2025-04-02T04:09:24Z","updated_at":"2026-03-11T16:01:51Z","resolved_at":"2025-04-02T04:46:45Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JQTD5EMW88F7B8G8ZGX38TTT","body":"Login via Google OAuth has been restored.","created_at":"2025-04-02T04:46:45Z","display_at":"2025-04-02T04:46:45Z","incident_id":"01JQTB11VQNNEHSQNP5062S9BD","status":"resolved","updated_at":"2025-04-02T04:46:45Z"},{"id":"01JQTD34YY7NTSTQYAH7AR5XYV","body":"Login attempts via Google OAuth have begun working again. Continuing to monitor to ensure the issue has resolved completely.","created_at":"2025-04-02T04:45:30Z","display_at":"2025-04-02T04:45:30Z","incident_id":"01JQTB11VQNNEHSQNP5062S9BD","status":"monitoring","updated_at":"2025-04-02T04:45:30Z"},{"id":"01JQTB11VQ2YH1ZSK6V08XGQCK","body":"Some customers are seeing: \"Access blocked: Authorization Error\" when attempting to sign in via Google","created_at":"2025-04-02T04:09:24Z","display_at":"2025-04-02T04:09:24Z","incident_id":"01JQTB11VQNNEHSQNP5062S9BD","status":"investigating","updated_at":"2025-04-02T04:09:24Z"}]},{"id":"01JPB7D6J3P95SFAV020BYYMT2","name":"Partial outage","status":"resolved","created_at":"2025-03-14T21:01:50Z","updated_at":"2026-03-11T16:01:50Z","resolved_at":"2025-03-14T21:45:00Z","impact":"major","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JPCWWKYGYCBRXGZ80Z022E0Y","body":"This issue has been resolved","created_at":"2025-03-15T12:36:29Z","display_at":"2025-03-14T21:45:00Z","incident_id":"01JPB7D6J3P95SFAV020BYYMT2","status":"resolved","updated_at":"2025-03-15T12:37:31Z"},{"id":"01JPB8GVBVF6PPAH5S73RJ9ZGA","body":"We have deployed a fix to the small number of customer regions that were seeing an outage in us-east-1 and eu-west-1 and service has been restored.","created_at":"2025-03-14T21:21:18Z","display_at":"2025-03-14T21:21:18Z","incident_id":"01JPB7D6J3P95SFAV020BYYMT2","status":"monitoring","updated_at":"2025-03-14T21:21:18Z"},{"id":"01JPB7P3KWSZJPSTH1NV74WKBV","body":"We believe we have identified the cause of the issue that has triggered an outage for a small number of customers in us-east-1 and eu-west-1 and are working on a fix. We expect the fix to be released within 3-4 hours.","created_at":"2025-03-14T21:06:41Z","display_at":"2025-03-14T21:06:41Z","incident_id":"01JPB7D6J3P95SFAV020BYYMT2","status":"identified","updated_at":"2025-03-14T21:06:41Z"},{"id":"01JPB7D6J3WHQQ3PQX57BHD2CP","body":"We are currently investigating the outage of a few customers in us-east-1 and eu-west-1","created_at":"2025-03-14T21:01:50Z","display_at":"2025-03-14T21:01:50Z","incident_id":"01JPB7D6J3P95SFAV020BYYMT2","status":"investigating","updated_at":"2025-03-14T21:01:50Z"}]},{"id":"01JKK4SFWDKQZANAH9SDPR0HFR","name":"Disk usage metrics missing in us-east-1","status":"resolved","created_at":"2025-02-08T16:02:57Z","updated_at":"2026-03-11T16:01:50Z","resolved_at":"2025-02-13T10:00:53Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JKZC24FHWC1NY9HEW3FBYA51","body":"This issue has been resolved. The bug introduced by our cloud provider has been addressed, and disk usage metrics in us-east-1 are now fully operational. We will continue to monitor for any further anomalies","created_at":"2025-02-13T10:00:53Z","display_at":"2025-02-13T10:00:53Z","incident_id":"01JKK4SFWDKQZANAH9SDPR0HFR","status":"resolved","updated_at":"2025-02-13T10:00:53Z"},{"id":"01JKTYKVYN6D27EATW9B80WADR","body":"This issue has been identified as a bug introduced by our cloud provider. We are actively monitoring updates from them and are exploring potential mitigations on our end while we await resolution. ","created_at":"2025-02-11T16:48:56Z","display_at":"2025-02-11T16:48:56Z","incident_id":"01JKK4SFWDKQZANAH9SDPR0HFR","status":"identified","updated_at":"2025-02-11T16:48:56Z"},{"id":"01JKR4GJRPBN5QFGRKZ3339AWH","body":"We are continuing to investigate an issue impacting disk usage metrics in us-east-1. ","created_at":"2025-02-10T14:34:17Z","display_at":"2025-02-10T14:34:17Z","incident_id":"01JKK4SFWDKQZANAH9SDPR0HFR","status":"investigating","updated_at":"2025-02-10T14:34:17Z"},{"id":"01JKNSJEM2A4SWBDMVE0C3TWS2","body":"We are investigating an issue causing incomplete metrics in our us-east-1 region. Our team has identified the problem and is working on a fix. ","created_at":"2025-02-09T16:44:35Z","display_at":"2025-02-09T16:44:35Z","incident_id":"01JKK4SFWDKQZANAH9SDPR0HFR","status":"investigating","updated_at":"2025-02-09T16:44:35Z"},{"id":"01JKK4SFWD3HB26ZBHS3R512ME","body":"We've identified an issue with our metrics exporter causing disk usage metrics to fail in us-east-1. Our engineering team is actively investigating. ","created_at":"2025-02-08T16:02:57Z","display_at":"2025-02-08T16:02:57Z","incident_id":"01JKK4SFWDKQZANAH9SDPR0HFR","status":"investigating","updated_at":"2025-02-08T16:02:57Z"}]},{"id":"01JHP0FP8R67TSY8NN6Z5BS8QF","name":"Partial Degradation of Privatelink Connections","status":"resolved","created_at":"2025-01-15T22:14:01Z","updated_at":"2026-03-11T16:01:49Z","resolved_at":"2025-01-16T03:10:11Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JHPHDZQB91R44Q3XDJ3EB9PY","body":"This issue is resolved. ","created_at":"2025-01-16T03:10:11Z","display_at":"2025-01-16T03:10:11Z","incident_id":"01JHP0FP8R67TSY8NN6Z5BS8QF","status":"resolved","updated_at":"2025-01-16T03:10:11Z"},{"id":"01JHP1XB3W4NSJZC1A9T49BWPW","body":"The fix has been rolled out to Production. Any affected Privatelink connections should be resolved. Environments that are not set up to auto-accept connections will need to re-accept in AWS to regain connectivity.","created_at":"2025-01-15T22:38:57Z","display_at":"2025-01-15T22:38:57Z","incident_id":"01JHP0FP8R67TSY8NN6Z5BS8QF","status":"monitoring","updated_at":"2025-01-15T22:38:57Z"},{"id":"01JHP0FP8RDG476Y4KZKFJEW0W","body":"We've identified an issue with creating and altering Privatelink connections. Any recently created or altered Privatelink connection may be degraded. We are working on a fix now. ","created_at":"2025-01-15T22:14:01Z","display_at":"2025-01-15T22:14:01Z","incident_id":"01JHP0FP8R67TSY8NN6Z5BS8QF","status":"identified","updated_at":"2025-01-15T22:14:01Z"}]},{"id":"01JGMN875JZFD66MYKBBF9NJ0S","name":"Cluster Creation Outage","status":"resolved","created_at":"2025-01-02T23:22:45Z","updated_at":"2026-03-11T16:01:49Z","resolved_at":"2025-01-05T15:22:05Z","impact":"minor","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JGVGY84DXMSRHJGQZ89HQ8BZ","body":"Cluster creation is fully restored. ","created_at":"2025-01-05T15:22:05Z","display_at":"2025-01-05T15:22:05Z","incident_id":"01JGMN875JZFD66MYKBBF9NJ0S","status":"resolved","updated_at":"2025-01-05T15:22:05Z"},{"id":"01JGN2RTA0EWQV1HKBEWVA74VM","body":"We've identified the problem and implemented a fix allowing for the creation of new clusters. We will continue monitoring to ensure the issue is fully resolved.","created_at":"2025-01-03T03:19:01Z","display_at":"2025-01-03T03:19:01Z","incident_id":"01JGMN875JZFD66MYKBBF9NJ0S","status":"monitoring","updated_at":"2025-01-03T03:19:01Z"},{"id":"01JGMN875JCGJNJX311C1NWTW4","body":"We have identified an issue that is preventing customers from being able to deploy new clusters. Our Engineering team is currently working on a fix.","created_at":"2025-01-02T23:22:45Z","display_at":"2025-01-02T23:22:45Z","incident_id":"01JGMN875JZFD66MYKBBF9NJ0S","status":"investigating","updated_at":"2025-01-02T23:22:45Z"}]},{"id":"01JA8QBTA799WRHNNB1B8CXBX9","name":"Partial External Network Outage","status":"resolved","created_at":"2024-10-15T18:34:36Z","updated_at":"2026-03-11T16:01:48Z","resolved_at":"2024-10-15T18:16:00Z","impact":"major","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01JA8QBTA7MM8287JWSCGRE876","body":"Network configuration was updated, all external services are again reachable, the incident has been resolved.","created_at":"2024-10-15T18:34:36Z","display_at":"2024-10-15T18:16:00Z","incident_id":"01JA8QBTA799WRHNNB1B8CXBX9","status":"resolved","updated_at":"2024-10-15T18:42:45Z"},{"id":"01JA8QBTA71NP68M9DRZ97SBAQ","body":"The node that we route outbound network traffic through (e.g. for Sources and Sinks that do not use AWS's PrivateLink) was rolled. The new node that took over had a configuration only partially applied which prevented some traffic from reaching outside of our network.","created_at":"2024-10-15T18:34:36Z","display_at":"2024-10-15T17:54:00Z","incident_id":"01JA8QBTA799WRHNNB1B8CXBX9","status":"identified","updated_at":"2024-10-15T18:42:38Z"}]},{"id":"01J3V8QGGK80QYRKYXRGWYD5ZC","name":"Delays bringing new cluster replicas online in aws/us-east-1","status":"resolved","created_at":"2024-07-27T23:34:32Z","updated_at":"2026-03-11T16:01:48Z","resolved_at":"2024-07-28T03:13:05Z","impact":"major","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01J3VN7P8P1KDQC3C6X513BZMN","body":"Our Engineering team has fully resolved the issue preventing the creation of new clusters.","created_at":"2024-07-28T03:13:05Z","display_at":"2024-07-28T03:13:05Z","incident_id":"01J3V8QGGK80QYRKYXRGWYD5ZC","status":"resolved","updated_at":"2024-07-28T03:13:05Z"},{"id":"01J3VHTHM0BDA9SPJYQMP8ATF6","body":"We have implemented a fix for this issue and are monitoring to ensure that new cluster replicas are successfully created.","created_at":"2024-07-28T02:13:29Z","display_at":"2024-07-28T02:13:29Z","incident_id":"01J3V8QGGK80QYRKYXRGWYD5ZC","status":"monitoring","updated_at":"2024-07-28T02:26:54Z"},{"id":"01J3V8QGGK66EESZP4CFT6JVXX","body":"We’re investigating reports of delays bringing new cluster replicas online.","created_at":"2024-07-27T23:34:32Z","display_at":"2024-07-27T23:34:32Z","incident_id":"01J3V8QGGK80QYRKYXRGWYD5ZC","status":"investigating","updated_at":"2024-07-27T23:34:32Z"}]},{"id":"01J16DFBW50WNXNP07KK3263NP","name":"AWS outage in US-EAST-1","status":"resolved","created_at":"2024-06-25T00:42:01Z","updated_at":"2026-03-11T16:01:43Z","resolved_at":"2021-12-08T04:00:08Z","impact":"critical","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01J16DFBW5S97HS57RVFWA4D70","body":"The incident has been resolved.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-08T04:00:08Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"resolved","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW55JPE4DAB0WRZ1K2S","body":"Error rates have decreased and the web console is working again. You may still experience slowness or intermittent failures. We will continue to monitor the situation.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-08T00:35:22Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"monitoring","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW5G282RZ5ZJNX9RD28","body":"We are continuing to see issues with the web console and connectivity in the AWS US-EAST-1 region.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-07T20:44:23Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"monitoring","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW5ANHMM1QM5WWJW46M","body":"We are continuing to see failures in the web console.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-07T19:24:48Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"monitoring","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW5PWSSQ6WHH59GD5XG","body":"Access to the web console seems to be restored, but we will continue to monitor the situation.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-07T19:13:09Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"monitoring","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW52QCDWG238TN7SSVT","body":"The issue has been identified.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-07T18:34:14Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"identified","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW5ZS6ARK09F8PB7391","body":"We are continuing to investigate this issue.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-07T18:34:02Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"investigating","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFBW5V5AFPP09PXF010KT","body":"We are seeing intermittent networking failures in the AWS US-EAST-1 region. This is causing downtime of the Materialize Cloud web console and restarts of some customer deployments that run in the US-EAST-1 region.","created_at":"2024-06-25T00:42:02Z","display_at":"2021-12-07T18:12:44Z","incident_id":"01J16DFBW50WNXNP07KK3263NP","status":"investigating","updated_at":"2024-07-01T12:15:30Z"}]},{"id":"01J16DFDH944NGJC4902GZPCD5","name":"New deployments taking too long to become healthy.","status":"resolved","created_at":"2024-06-25T00:42:03Z","updated_at":"2026-03-11T16:01:43Z","resolved_at":"2021-10-21T00:29:44Z","impact":"none","page_id":"01J16DCZR42EPRD7WB12BB78T0","incident_updates":[{"id":"01J16DFDH9VH81899BQDP87VPC","body":"The incident has been resolved.","created_at":"2024-06-25T00:42:03Z","display_at":"2021-10-21T00:29:44Z","incident_id":"01J16DFDH944NGJC4902GZPCD5","status":"resolved","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFDH9F1HN104C45ACBVPW","body":"We have identified the cause of the startup delays. New deployments are unable to schedule due to a problem with the network configuration of new EC2 nodes. The team is actively working on a fix.","created_at":"2024-06-25T00:42:03Z","display_at":"2021-10-20T23:27:39Z","incident_id":"01J16DFDH944NGJC4902GZPCD5","status":"identified","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFDH9X36YV5THAT39J06P","body":"We are continuing to investigate the issue.","created_at":"2024-06-25T00:42:03Z","display_at":"2021-10-20T23:22:29Z","incident_id":"01J16DFDH944NGJC4902GZPCD5","status":"investigating","updated_at":"2024-07-01T12:15:30Z"},{"id":"01J16DFDH9KJ662BG4GJ51GH66","body":"We are currently investigating an issue creating new deployments in Materialize Cloud. These new deployments are taking too long to become healthy after creation. We will post updates as we have them.","created_at":"2024-06-25T00:42:03Z","display_at":"2021-10-20T23:05:56Z","incident_id":"01J16DFDH944NGJC4902GZPCD5","status":"investigating","updated_at":"2024-07-01T12:15:30Z"}]}]}