[MLB-WIRELESS] *tap tap* Is this thing on?

[Steven Haigh]

Well. That was fun.

A quick run down on what happened:

1) The system (for reasons yet unknown) decided to remount the /  
partition as read only - and would not change to read-write. Problems  
with this means that I couldn't restart services, or really do  
anything. A standard reboot failed due to things not being able to be  
written.

2) The system was power cycled from a remote power switch that is  
onsite but never came back up.

Today I had someone look at the system to find it stuck at an fsck  
prompt noting that some problems had been found with the file system.  
The normal fsck was run and now the box is up and everything should  
start being happy again.

To help with this in the future, I have secured and configured a  
serial console on the box that is active from the moment GRUB loads -  
so I should be able to access the box even if it fails to boot via the  
serial console - as long as it gets to the GRUB boot loader. I'm in  
the middle of testing this at the moment, and all looks good.

I'm also going to check through the system to see if there is any  
indication why / was rendered read-only with no apparent trigger.

The RAID arrays held together fine - so whatever happened seemed to be  
on the filesystem level.

--
Steven Haigh

Email: netwiz at crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897