Resolved -
This incident has been resolved
Aug 8, 01:25 PDT
Update -
Thank you for your patience. The main query-intensive jobs are through now, but it also took quite a bit longer than before. Catch up for subsequent jobs may therefore take up to 3am PT to complete. Our sincere apologies for the delay and we will continue to share updates as we monitor the job completion.
Aug 7, 21:00 PDT
Update -
We are still making progress on re-running the exports. We should start to see some exports completing but the expected completion time for the full system to rerun has extended to 9pm PT. We will continue to keep you updated and aim to have the next update at 9pm PT!
To clarify:
1. This only impacts customers who use Statsig warehouse native. If you are on Statsig cloud, you can ignore this status page.
2. For WHN customers, once we finish rerunning the final jobs, Statsig will automatically take care of refreshing the data for all experiments that are configured to incrementally refresh metrics.
Aug 7, 17:57 PDT
Monitoring -
This issue only impacts customers on Warehouse Native on the echidna pipeline.
Around 11 am PT, we have identified that WHN accounts will have event and exposure export delays from August 6. No data is lost as the job rebuilds the full day from source so there will be no gaps once the post-revert run lands.
Everything through August 5 is intact.
Around 1pm PT, we have implemented a fix that should have mitigated this issue.
We expect exports to complete around 7pm PT.
Aug 7, 16:10 PDT