Our database server crashed unexpectedly on friday night and we were down over most of the weekend.
Incident Timeline
| Date | Time | State |
|---|---|---|
| 2018-06-15 | 16:20 | Database server becomes unresponsive. This additionally takes down a couple of file systems that Galaxy uses |
| 2018-06-15 | 16:36 | Processes become unresponsive, monitoring data stops coming in |
| 2018-06-15 | 17:44 | Galaxy stops responding |
| 2018-06-15 | 22:56 | We notify our database administrator about the issue |
| 2018-06-17 | 20:45 | Email received that our database admin has finished repairing the server |
| 2018-06-17 | 21:04 | We manually restart the Galaxy server processes |
| 2018-06-17 | 21:15 | The first jobs run successfully again |
Resolution
We are discussing our long term options to preventing, or at least ameliorating similar issues in the future. We are investigating the possibly of keeping Galaxy online but read-only, and how this might impact our users.