The MacDeskBlog

Michelle spent the week clearing out the front bedroom, pulling the carpeting out of the front bedroom, pulling staples and tackstrips, and cleaning up debris. On Saturday, she spent the day attaching pigtails in the outlet boxes downstairs, attaching outlets to the pigtails, and pulling some wires.

Steve finished framing in the downstairs with help from a new guy, Aldo. He needs to check his work upstairs and add a few studs in for sheet rock backing, then we’ll be ready for framing inspection.

Reuben and Orlando spent the week removing the remaining siding from the upstairs, drilling holes for me to run wiring through, and nailing on straps.

I spent Saturday moving electrical outlets upstairs so they are 18″ up instead of in the baseboards, re-wiring circuits upstairs, and wiring circuits downstairs.

Aiming for framing, shear wall, and plumbing inspections this week, and rough electrical inspection next week.

I finally got to see my new floor yesterday. Yippee! I spent the weekend wiring downstairs and Michelle spent Saturday stripping the wainscoting upstairs.

Lumber arrived mid to late morning on Friday. By the time we got there Saturday morning, the office wall and the bathroom/utility room wall were both finished. This was great news for me, since it meant I could run wiring throughout. Michelle had already nailed up most of the outlet and switch boxes, and had run some of the wires between the boxes. I nailed up the outlet boxes which go into the new walls, and pulled the remaining wiring through to connect up the boxes.

Then I ran the wires to connect each circuit to the new service panel which isn’t mounted yet. The hard part of that turned out to be Sunday morning, when I discovered that the 1″ holes I asked the crew to drill through all of the studs at 18″ high are only large enough to handle about 3 pieces of 12-2 w/ground Romex. Great for most places, but insufficient in the north wall, where I had to run circuits for the office, the family room, the utility room & bathroom, the master bedroom, and a dedicated 10-3 w/ground for the dryer outlet. I had to drill additional holes through the floor joists most of the way. Fishing cables through holes in the corners was also not really in the fun column, but everything came through okay ultimately.

The closet walls didn’t go up though, as Steve underestimated the number of studs he needed. They should go in Monday afternoon or Tuesday at the latest. Since they’re still laying on the floor or not built yet, I couldn’t finish the garage circuit nor the family room circuit, but the last few strands will go in quickly once the walls go up.

It was pointed out to me by a lucky guy with a new G5 and 30″ Cinema Display that he could not change his preferences on the site. So, I looked into it and saw pretty quickly that preferences were saving at all.

It turned out to be a little database permission problem, which I fixed. If you have any similar problems, please let me know.

Ryan

I caught up on filling in the missing pictures. There were two which I couldn’t figure out which picture was supposed to be posted, so they’ll get mixed into the queue going forward. There were two others where I made my best guess.

The database and file system locked horns again tonight so I had to restart the database. Grr. Oh well. I’ll need to figure out some way to identify that problem and automatically restart the database in the short to medium term, and some way to isolate and correct the problem in the long term.

Not tonight though. It’s after midnight. Well past my bedtime. G’night.

I filled in a bunch of the missing pictures tonight, and found the names of the five which eluded me previously. I think I have 42 left fill in. Friday is looking pretty reasonable at this point to finish that process.

The database hung again tonight. It looked like it hit a write lock or a file lock for some reason. Had to restart it manually. :-\

On the house front, we passed rebar inspection for the slab floor this afternoon. We’re clear to pour tomorrow.

The first morning didn’t go too well on the new server. MySQL was eating up too much CPU and kjournald compounded the problem. The server load got up to 40 around 8AM PDT. So, there were a few reboots as I tried in vain to lower the amount of journaling being done for ext3. Once again, Rob Egan came to the rescue with insights and suggestions. Something he said made me suspect the MySQL settings were not scaled properly, so I ratcheted them down quite a bit. Combining that with the filesystem changes, the server seems to be behaving better now than it is was all morning.

I’m interested to see how it does over the next week or so. I’ll probably need to do some more tweaking of various things. I’ll try to keep downtimes under 15 minutes each, if possible. And most tweaks only require a few seconds to stop and start MySQL or Apache.

I other news, the 13 hours of sleep I got Saturday night paid off in the form working on the server until 2AM Monday morning without kicking my ass today. All of the pictures which were scheduled for April 28th-May 5th have been re-queued for this week, starting today. I got a few of the missing pictures back in for March and April too. I think I’ll be able to get them all in by the end of the week. In the meantime, paging back through prior weeks will display fewer pictures than normal until you get back to March 12th and before.

The good news is that with about an hour of hands-on help from Rob Egan, plus some sporadic IM help for the rest of the afternoon, we managed to get the new Dedicated Server up to date using yum and rpm! It then took me over a week to get almost everything up and running the way I like. The phpAdsNew gave me fits for a few days, but I ultimately tracked the problem down. I don’t have MySQL replication working yet, which makes me a bit nervous, but I do have full database backups running nightly at least.

So onto the not so good news. I haven’t rebuilt the past 2 months of missing data yet, and I haven’t filled in the queue going forward yet. Also, as mentioned in the last post, I think I lost some blog entries, and definitely all of the blog user accounts and comments. I’m not worried about the blog entries and comments. Also, not all of my other sites are back online yet.

I could try to rebuild all of the missing data, and I’d likely get most of it correct. However, that wouldn’t really be very fair to people whose pictures were posted the last few days before the crash, since the pictures didn’t get a full week up on the front page.

On the other hand, I could skip rebuilding the missing data, and just post everything again as if it were new. However, that wouldn’t really be very fair to all of the users, since you’d be getting pictures you’d already seen before.

So, what I’ll probably do is rebuild the database as best I can from March 13th through April 27th, and repost pictures which originally went up April 28th through May 4th as if they had never been posted. Any pictures which I’m uncertain about whether or not they were posted previously will also get treated as new.

I have picture titles, categories and artist initials for all of the missing period except for March 22, 23, 30 and April 10. If you still have email announcements covering any of those days (daily or weekly), please send them along to me at the feedback address.

I hope to finish recovering from this by the end of this week. We’ll see how well I managed that. Work has been insanely busy since the middle of last month, which dragged out this process already. This week promises more of the same.

Thank you for the notes of support,

Ryan

As you probably noticed, the server died last Wednesday. Since I was in the process of building the replacement server, I wasn’t as diligent in my backup procedures. One of the things that I was trying to do with the replacement server was upgrade MySQL because the database replication was not working from the virtual dedicated server to the new dedicated server and a database upgrade will provide more powerful replication.

Alas, the virtual dedicated server had a catastrophic failure. Core systems files vanished. The entire database vanished. Backups of the database vanished. Backup scripts vanished.

So that’s the bad news. The good news is that the database is the only piece that wasn’t being replicated. Copies of all of the files were already on the new dedicated server (and two other copies as well). Even the database story isn’t entirely bad. I still have the backup from the last time I manually replicated it to my development server. The bottom line is that I lost all of the database changes from early March through last week. I’m going to try to figure out which pictures I posted in that time frame and re-post them for the same days. For pictures I can’t figure out, I’ll just queue them up for the future.

The other good news is that MacDesktops is finally on the new dedicated server, and I have most of the services I need up and running on it. If you notice anything isn’t quite right, please let me know.

On the blog front, I’m pretty sure that I had 24 or 25 blog entries previously. The entire blog database was new since the backup I had, so it looked like I was going to lose the entire thing. Fortunately, I was using Feedburner to syndicate the site, and it had most of the posts cached. I was able to import 17 posts. I don’t know which posts are missing, if any. Maybe the count included posted comments which I DID lose along with the users. I probably won’t try very hard to recover the last bit that’s missing. Getting as much as I did back through Feedburner was more than I hoped for.

My employer gave everyone in my department one of these spiffy Victorinox watches a couple of years ago. Most things people get from work are pretty ho-hum, along the lines of t-shirts, pens and other mundane items that you already have a ton of. Every now and then, though, work surprises you with something that you really like. For me, this has happened thrice in a bit over a decade. Sure, I kept all of the t-shirts that I get, and I even wear most of them. I also received a really nice jacket from work last year, which I wear almost daily and really appreciate. But, even the jacket doesn’t click with me like a particular sweatshirt did back in 1997, or a duffle bag that same year, or like this watch did. The duffle bag was the first non-shirt item given by a tiny consulting firm I worked for. I can’t really put my finger on why the sweatshirt and the watch reside on a different plane of appreciation from the rest of the stuff. Obviously, it’s not about the monetary value of the objects, since the sweatshirt wasn’t very expensive, and the jacket is more valuable than the watch. I think there was something intangible in the receipt of each item which made it more special. The surprise I felt in getting the item is somehow embedded in the item itself now.

I do know one of the reasons that I liked the watch so much immediately. I had just gone scuba diving for the first time shortly before work gave me the watch, and I had decided to get certified to dive. I was thrilled when I flipped over the watch and saw the words “WATER RESISTANT 100 METERS” on the back. It instantly became my dive watch in addition to being my everyday watch. I hadn’t actually worn a watch every day for many years with the exception of about a year when I wore a Nike watch that Tracey gave me (I lost it, I think at the gym). I wore this watch in Maui when I dove there a couple of years ago. I wore it diving in Bonaire and Key West last year. And of course I wore it diving in Maui last week.

I don’t know what changed in Maui this time, but my Victorinox is longer water tight. I went on more dives and deeper dives in Bonaire last year, so it’s not like the diving should have done it in. I noticed a little bit of fog on the screen by mid-week last week, and it got a little worse with each dive after that. Mind you, it is a quality Swiss time piece, so it is still keeping time. I’m just having trouble telling what time that is. Okay, that’s not quite true, since the hands are plenty big, telling the time isn’t really a problem. I just don’t enjoy reading the time off of it now, through all of the condensation inside. And, the date counter is really hard to read now.

Anyway, I’m pretty sure that Victorinox gives lifetime warranties on most or all of their products. I’m hoping to get it repaired or replaced under warranty. In the meantime, I’m just wearing it with the sweat inside, silly as that is. If I stop wearing it just because it’s flawed right now, I think I’ll diminish the attachment I have to it. It’s not like I need to wear a watch at all. For years, I just used my pager as my watch, and now I carry a Blackberry around all the time anyway. But I like to wear this watch.

Rational people don’t always behave rationally. This is just one of those times for me. Come to think of it, there are other irrational things that I’m doing these days too. Not all of them innocuous. Time for some introspection induced by a faulty watch. Go figure.