May 2011 - Techlist - mickaboo.org

[Mickaboo Techlist] I'm also on Vacation this week
by Matt Linton 21 May '11

21 May '11

Hi all! I'm also going out of town on vacation early Monday and, while I'll have the digital tethers with me, I won't be answering them much or in a timely manner. If you start to miss me (ha!) while I'm gone, feel free to pull up these two segments of a TV interview I did for Mickaboo on a small local channel in San Jose. Go easy on me, it was a half hour interview so I had plenty of time to screw up at least one or two of the things I said there. Socrates enjoyed it though. http://www.youtube.com/watch?v=LGVlkXFnMsM (part 1) http://www.youtube.com/watch?v=GqhDlN0_CD4 (part 2) If a disaster strikes, give my cell a call. Otherwise I leave you in Walt, Chloe and Waynes' capable hands. :)

1 0

[Mickaboo Techlist] Welcome to two new Tech volunteers!
by Matt Linton 18 May '11

18 May '11

Hi all; I just wanted to extend a welcome to our two new tech team volunteers, Rachel and Allison. I'll let them introduce themselves when they get some time. :) Rachel/Allison - This is our primary communication method for the technical issues at Mickaboo outside of our ticketing system (more on that later). I'm trying to arrange a time to get as many of us together as possible so we can go over everything currently in the air and on the roadmap. How's everyones' availability look for having a Tech Team get together at the summer picnic? http://confluence.mickaboo.org/display/MBPublic/2011+Summer+Picnic

2 1

Re: [Mickaboo Techlist] [Ticket#0186694] Petfinder.com Pet Inquiry: Ozzie PFId#15570988
by AMuse 16 May '11

16 May '11

Hmm. Looks like petfinder import may be broken AGAIN. I'll check with them and see what's up. On 5/16/11 9:17 AM, Mickaboo Companion Bird Rescue wrote: > Just an FYI .... Ozzie still shows up on the Petfinder site > > http://www.petfinder.com/petdetail/15570988 > > > -- Walt > ============================== > Mickaboo Companion Bird Rescue > "Don't buy, don't breed....... > ADOPT! > > AMuse<amuse(a)foofus.com> wrote: > >> Looks like our auto-export script died. I'll restart it, should fix up our >> records. >> >> >

1 0

[Mickaboo Techlist] Post-mortem of the server outage on Friday
by Matt Linton 09 May '11

09 May '11

As promised, here's the post-mortem: ------------------------------------------------------------------------ At 0600 on Friday, May 6th, a power outage in Fremont CA affected our datacenter provider, Hurricane Electric. According to HE, a power surge preceding the outage damaged their battery and generator backups, resulting in lost power (normally emergency power would kick in). The surge(s) affected our server and damaged the primary disk. Our server has two disks - one primary, and one backup disk that is normally kept offline unless doing a backup. Since the backup disk was NOT mounted, it was not affected by the power cycling. Since I was already on my way to a commitment in Modesto, CA that day, I was unable to travel to the datacenter and swap the disks until about 1330. When I arrived at the datacenter I swapped disks and brought the server online. Immediately after coming online, all our services (confluence, asm, mailing lists, etc) came back online - but with months-old copies of their databases due to the way that backups are made (quarterly full backups of the operating system and configurations, but monthly+daily backups of the data). This caused some people to temporarily reappear on the discuss and admins lists, and for Confluence/ASM/etc to show only January's copy of the data. After about two hours I finished restoring the actual database backups, bringing our services current to May 6 @ 2am. However, the Confluence incremental backups for May 1-6 would not restore for some reason, and we lost those days. /I still haven't found out what is wrong there; the data exists but Confluence will not import it properly. / Since email and OTRS were UP but not fully restored for a period of about 4 hours, some tickets that we received in that 4 hour period have been lost. Emails sent to us while the server was down were NOT lost, because SMTP re-sends undelivered emails up to 5 days after a mailserver crash. *Overall*: The response, time frames, actions and amount of data loss are entirely consistent with our Mickaboo Tech Contingency Plans document on the wiki ( http://confluence.mickaboo.org/display/mbadmins/Mickaboo+Tech+Contingency+P… ).

1 0

Re: [Mickaboo Techlist] [Ticket#0181939] New email to Technical issues (what else seems mi [...])
by Matt Linton 06 May '11

06 May '11

Nope, go for it and make your re-edits. I'm exhausted and going to bed, so I'm not getting anything else restored tonight. On 5/6/11 9:59 PM, Lee, Pamela A wrote: > Soo, for tomorrow's event... is it going to hurt your efforts if I edit the events/blog pages? > > > Pamela A Lee > 415.436.5712 > Home Loans Data Steward and Planning Team > "Coming together is a beginning. Keeping together is progress. Working together is success." - Henry Ford

1 0

Re: [Mickaboo Techlist] [Ticket#0181939] New email to Technical issues (what else seems mi [...])
by Matt Linton 06 May '11

06 May '11

Yes, the Confluence restore isn't going totally smoothly like OTRS and ASM (seem to) have. I'm not sure what's up, as I restored from the same known good backup set, but Confluence is missing May 1 - 6 still. I can SEE the data in the SQL index manually but the GUI interface is not displaying it. (WTF?). I'm working on trying to get back every bit of data, but all told if I totally fail then the only thing we'll have lost is 6 days of Confluence edits. FYI I'm still working on a written post-mortem but it looks like the root cause is that Hurricane Electric (our colo provider) experienced a big power surge due to a Fremont power outage (PG&E caused?) and their UPS failed to regulate the power like it was supposed to. Our server and dozens of others today experienced failed disk drives. There were a lot of very angry geeks in the datacenter alongside me today. :P On 5/6/11 9:20 PM, Mickaboo Workflow System wrote: > Hi Matt, > > there is a new ticket in "Technical issues"! > > "Lee, Pamela A"<pamela.a.lee(a)bankofamerica.com> wrote: > > <snip> >> Looks like the May 2011 blog entries are missing as of this moment. >> Ditto the "events" page edits during May aren't reflected in what I see >> right now. >> >> >> Pamela A Lee >> 415.436.5712 >> Home Loans Data Steward and Planning Team >> "Coming together is a beginning. Keeping together is progress. Working >> together is success." - Henry Ford >> >> >> ---------------------------------------------------------------------- >> This message w/attachments (message) is intended solely for the use of >> the intended recipient(s) and may contain information that is >> privileged, confidential or proprietary. If you are not an intended >> recipient, please notify the sender, and then please delete and destroy >> all copies and attachments, and be advised that any review or >> dissemination of, or the taking of any action in reliance on, the >> information contained in or attached to this message is prohibited. >> Unless specifically indicated, this message is not an offer to sell or a >> solicitation of any investment products or other financial product or >> service, an official confirmation of any transaction, or an official >> statement of Sender. Subject to applicable law, Sender may intercept, >> monitor, review and retain e-communications (EC) traveling through its > <snip> > > To view this ticket, visit: > https://mickaboo.org/otrs/index.pl?Action=AgentZoom&TicketID=82328

1 0