We had a temporary quota issue today when one of our developers working on automating UseGalaxy.eu Quota Increase
Application
made a mistake in the code which resulted in the legacy-users quota membership being removed.
Incident Timeline
| Time | State |
|---|---|
| 12:50 | Developer noticed mistake as it happened during development, but too late. |
| 12:51 | First user complaint received |
| 13:08 | Issue resolved |
Root Cause
The script was intended to only synchronize quotas named auto_###, but due to
a bug it attempted to synchronize all quota groups. It defaulted to
synchronizing an empty set of users and completely removed the user
associations for the legacy-users quota.
Resolution
We extracted the relevant table from a database dump and proceeded to import those rows back into the table. The backup was missing one row but we have identified the missing data in admin discussions.
Going forward we will be deploying a test.usegalaxy.eu and developing against
that server to avoid similar problems.