1. Affected nodes rebooted

   Hide full report Show full report

   The affected nodes have been rebooted.

   The situation is back to normal.

   We are halting the NVMe migrations for now and will investigate further.

2. OSD situation fixed, restarting nodes

   Hide full report Show full report

   The full OSD situation has been rectified.

   A few nodes (around 5%) have stalled I/O. Those are being rebooted.

3. Single OSD filled

   Hide full report Show full report

   We have found that a single OSD has filled up to 95%, after which Ceph (the storage system) stops accepting writes.

   However, there is more than enough space on the other drives, so the balance is off. We are investigating why.

4. OSD full

   Hide full report Show full report

   We were doing several migrations to NVMe. We have done four of those in the past few weeks, with no problems. But during this migration, one physical disk (OSD) filled up. It seems like too much data was being written at the same time.

   We are working on lowering the OSD's usage.

5. Storage issues

   Hide full report Show full report

   We are seeing issues on our new NVMe storage. We're working on it.

Experiencing an issue?