After the last couple of disk failures and subsequent problems with the RAID card in beastie (one of two servers that comprise ftp.freebsd.org), Scott Long sent me another RAID card by way of Mike Silbersack to EuroBSDCon. On Monday, I finally built up the courage and got beastie removed from the DNS round-robin so I could have some peace and quiet time in a service window. In theory, the new card should be able to read the configuration from the disks and take it from there, but somehow I’m a tiny bit paranoid and I don’t really trust that. So I had borrowed a bit on our new tape robot and flushed all the data to tape.
Yesterday, I took the machine down and exchanged the cards and rebuilt the RAID set. The card did read the configuration alright, but I removed it and built a new one. So far so good. Next, trying to boot the machine. All went well until it detected the second RAID set. The commands to the second card timed out and the machine couldn’t boot. Somehow, I got the machine booted after several tries and just let it sit there to let it sync the disks with the new config. Late afternoon, Scott came online and suggested to remove aacp from the kernel config as it might be causing problems and we don’t use it anyway. Beastie booted happily and now it was time to restore the data from tape. Writing to 8 disks in a RAID5 set obviously isn’t the easiest thing to do, so I only got about 4MB/sec out of it while we found out that the backup server only had a 100mbit card as we pushed 10MB/sec to it.
This morning the restore finally finished and after a final reboot, just to be on the safe side, everything is running again. Please welcome the new and improved beastie!