There is an issue which does not disrupt the main hosting service

  1. Affected nodes rebooted

    The affected nodes have been rebooted.

    The situation is back to normal.

    We are halting the NVMe migrations for now and will investigate further.

  2. OSD situation fixed, restarting nodes

    The full OSD situation has been rectified.

    A few nodes (around 5%) have stalled I/O. Those are being rebooted.

  3. Single OSD filled

    We have found that a single OSD has filled up to 95%, after which Ceph (the storage system) stops accepting writes.

    However, there is more than enough space on the other drives, so the balance is off. We are investigating why.

  4. OSD full

    We were doing several migrations to NVMe. We have done four of those in the past few weeks, with no problems. But during this migration, one physical disk (OSD) filled up. It seems like too much data was being written at the same time.

    We are working on lowering the OSD's usage.

  5. Storage issues

    We are seeing issues on our new NVMe storage. We're working on it.

Experiencing an issue?

Question

Response within 4 working hours (working days from 9:00 AM to 5:30 PM Amsterdam time)

Free

Ask your question

Urgent problem

Response within 1 hour (24/7)*

*Charges may apply for this service.

Call +31 (0)40 711 44 96