All services are online

Upon further investigation we were unable to work through the backlog of predictions. Backlogged predictions have been cancelled. It will take some time before the prediction IDs report failed in the replicate web console.

Users may resubmit any of these cancelled predictions.

All predictions that have been submitted since the last update at Sep 26 2024 at 11:52pm UTC are unaffected by this cancellation of predictions.

Queues for predictions remain fairly high for black-forest-labs/flux-schnell and meta/meta-llama-3.1-405b-instruct. All other models should be behaving normally.

The database upgrade is now complete. We saw a period of about 3 minutes where the website was unavailable, and all of the affected queues have recovered.