Hi all! I promised Mal I would give him a detailed writeup on how to do forensics on our MySQL database, but there's no reason not to share it with everyone on the tech team. Because it contains system internal info, I've locked it to tech members only. Here's a rough cut at some relevant documentation I put in the wiki. http://confluence.mickaboo.org/display/mbadmins/Recovering+older+ASM+Data All in all, our backup and retention policy worked very well this week. Although a former admins volunteer had deleted a record more than 2 months ago before someone pointed it out, we were able to go back and reconstruct the full record, as well as identify who deleted it and at what time. The only changes I intend to make as a result of this test is to increase the time during which we'll store individual incremental (daily) SQL dumps. They're REALLY small now, so it doesn't make much of a storage difference if we keep them 1 month vs 4 or so, but it made a huge difference during this incident that I was able to grab the Incrementals from November 09. So unless it proves to be a disk space problem, I'm going to keep the incrementals probably 6 months going forward, before erasing them. Full monthly dumps get stored indefinitely as is our current policy.