The MacDeskBlog

This has taken me at least a month longer than I had hoped, but I’m finally ready to start beta testing the new main server. The new server is a Mac Mini provided by the folks at Visual Seed. Expect to see new ads from them soon. Images are still and will continue to be hosted by the good folks at Import GSM.

I’m heading off on vacation for the next week, or I would try to swap the new server into production. Of course, that would be somewhat foolish without more testing, so it’s probably a good thing that I didn’t get to this point two days ago.

Known issues on the new server:
1. ads may not always appear on the pages,
2. changing your site preferences might fail with a Couldn’t Connect to Database error,
3. site might be a little bit sluggish when fetching ads,
4. all blog related links will take you back to http://MacDesktops.com/ instead of leaving you on http://ws2.macdesktops.com/

Why those issues aren’t problems:
1. ad requests go through an ssh tunnel to the production database, which will reside on the new server.
2. database writes have the same limitation.
3. same thing.
4. limitation in WordPress. Won’t be an issue when ws1 becomes MacDesktops.com.

Please take a stroll through the new server. If it works well for you, feel free to use it instead of the production server. And if you encounter any other problems or odd behavior on the new server, please send me a note and head back to the production server. As always, you can reach me at the feedback address at macdesktops.com, even if the javascript mailto link doesn’t work.

Another storm blew through town over the weekend and knocked out power to the DNS servers. Though the web servers were still up and running on different connections, nobody could get to them. I migrated the DNS late Sunday night to reduce the potential points of failure. And, I’m still working on migrating the pages off of the remaining DSL line as well, though that process has hit a couple of speed bumps.

I tried turning on Performance caching a week or two ago. What a disaster. MacDesktops content is simply not compatible with performance caching. I tried to turn it back off again a couple of days into the experiment, but it appears that if any domain name has it turned on, then all domains are affected by it. Or maybe changing the setting within the Server Admin (Tiger Server) application just doesn’t change the setting to the webperfcache application on the fly.

Anyway, I managed to kill webperfcache today, and the server load dropped immediately from about 20 to about 6. It’s down below 3 now that we’re well outside of peak hours. My apologies for taking the site offline for a several minutes during that process. I hope that this will translate into an improved user experience. MacDesktops is still being served over those two unimpressive DSL connections, so performance won’t improve tremendously, regardless. But it should get at least a bit better.

I am in talks with a few different companies about hosting and hope to get that resolved very soon. Cross your fingers.

GoDaddy hit me with a $900 charge for data transfer overage last month. No warning. They charge $29 for 500GB of transfer per month, and then jack it up to $6/GB for overage transfer. With no warning. They were willing to charge me only $450 for the overage, but not willing to go lower because I could have seen the overage through their exceedingly cumbersome administration console. I was supposed to notice that bandwidth usage had double for the last week of my month for that, and remember that my month ends on the 18th.

I can see that the server went over the bandwidth limit. I can accept that I should pay more for the month. The problems I have with this situation are:
A. $6/GB is an unreasonable rate for overage which $0.058 is the base pre-paid rate;
B. GoDaddy does not make it clear to customers that the overage rate is more than one hundred times the base rate;
C. GoDaddy does not warn of overage before the fact, they expect users to constantly check through GoDaddy’s administration console (which is junk) to make sure everything is fine (which is junk);
D. GoDaddy does not pre-authorization the overage;
E. GoDaddy does not provide any way to make the prepaid bandwidth a cap for the month, they just start charging the overage rate.

So, I cancelled my servers.

I apologize for the inconvenience. Please feel free to contact GoDaddy.com and tell them how screwed up their business is and that their handling of MacDesktops is preventing you from using their services. The billing manager’s name is Mark Danzan. If you talk to him and he tries to tell you that their billing model is like a cell phone plan, please make sure you sympathize mention that you too have a cell phone plan which charges $30/month for 500 minutes and charges $6/min thereafter! Or feel free to contact me and tell me how ridiculous I’m being.

I am working on another hosting arrangement for the future. In the meantime, I am very close to having the site up and running on my home servers, with images spread across two DSL lines.

It is clear to me that I need to redesign the site now, as in very soon. I keep receiving emails from people who think the site is broken because they keep seeing just the message box at the top of the page, not realizing that the pictures are still there, just a little bit lower. This is an extremely clear indication that the expanded message box is causing problems for users, and is therefore bad design.

I’m wasting too much space at the top, and creating a false bottom by the clean break all the way across the page below the message box.

Anyway, I have big big plans for the redesign. I’ll try to break them into manageable chunks. Whenever possible, I’ll rollout incremental releases. Hopefully, this will mean incremental improvements over the coming months more than it will mean constant change and confusion.

My impending ability to work on the redesign is directly related to the improved stability of the server since yesterday. I may have fixed that bug right before I posted about needing help isolating the bug. The bit of code that I changed really shouldn’t have been able to cause the behavior I was seeing, so I’m not declaring the bug squashed yet. However, the simple fact that there haven’t been any slow database queries since I made the change yesterday, compared to a few dozen per day since I moved onto the Linux servers in early May, has me hopeful.

Also worth mentioning is that I have not cut over to the new, beefier server yet. It’s merely awaiting my copying about 20GB of pictures and site files over to it, and configuring the database and web server. Maybe this weekend, or maybe next week.

Recapping:
1. Scroll down
2. I am aware that the site design desperately needs an update
3. I actually think that I’ll be able to start that soon
4. Server stability and performance is improving, but
5. The never-responding-search bug might still be out there, so tell me if you encounter it!
6. New server still on the way, and soon.

There’s a bug in the site right now, and I can’t reproduce the problem in order to find the bug.

Symptom
You submit some request and never get a response from the server.

What to do
Use the email link at the top of the MacDesktops home page (”Let me know”) to tell me exactly what you were trying to search for, and to send me the URL which appears in your browser when it’s not getting any response. To help me the most, your email should include:
1. The full, ugly URL.
2. Which fields you entered things into.
3. What you entered into those fields.
4. Which browser you are using, including version number.
5. Which operating system you are using, including version number.

Hopefully, a couple of those emails will enable me to reproduce the problem and squash the bug.

Thanks for your help,

Ryan

It was pointed out to me by a lucky guy with a new G5 and 30″ Cinema Display that he could not change his preferences on the site. So, I looked into it and saw pretty quickly that preferences were saving at all.

It turned out to be a little database permission problem, which I fixed. If you have any similar problems, please let me know.

Ryan

The first morning didn’t go too well on the new server. MySQL was eating up too much CPU and kjournald compounded the problem. The server load got up to 40 around 8AM PDT. So, there were a few reboots as I tried in vain to lower the amount of journaling being done for ext3. Once again, Rob Egan came to the rescue with insights and suggestions. Something he said made me suspect the MySQL settings were not scaled properly, so I ratcheted them down quite a bit. Combining that with the filesystem changes, the server seems to be behaving better now than it is was all morning.

I’m interested to see how it does over the next week or so. I’ll probably need to do some more tweaking of various things. I’ll try to keep downtimes under 15 minutes each, if possible. And most tweaks only require a few seconds to stop and start MySQL or Apache.

I other news, the 13 hours of sleep I got Saturday night paid off in the form working on the server until 2AM Monday morning without kicking my ass today. All of the pictures which were scheduled for April 28th-May 5th have been re-queued for this week, starting today. I got a few of the missing pictures back in for March and April too. I think I’ll be able to get them all in by the end of the week. In the meantime, paging back through prior weeks will display fewer pictures than normal until you get back to March 12th and before.

The good news is that with about an hour of hands-on help from Rob Egan, plus some sporadic IM help for the rest of the afternoon, we managed to get the new Dedicated Server up to date using yum and rpm! It then took me over a week to get almost everything up and running the way I like. The phpAdsNew gave me fits for a few days, but I ultimately tracked the problem down. I don’t have MySQL replication working yet, which makes me a bit nervous, but I do have full database backups running nightly at least.

So onto the not so good news. I haven’t rebuilt the past 2 months of missing data yet, and I haven’t filled in the queue going forward yet. Also, as mentioned in the last post, I think I lost some blog entries, and definitely all of the blog user accounts and comments. I’m not worried about the blog entries and comments. Also, not all of my other sites are back online yet.

I could try to rebuild all of the missing data, and I’d likely get most of it correct. However, that wouldn’t really be very fair to people whose pictures were posted the last few days before the crash, since the pictures didn’t get a full week up on the front page.

On the other hand, I could skip rebuilding the missing data, and just post everything again as if it were new. However, that wouldn’t really be very fair to all of the users, since you’d be getting pictures you’d already seen before.

So, what I’ll probably do is rebuild the database as best I can from March 13th through April 27th, and repost pictures which originally went up April 28th through May 4th as if they had never been posted. Any pictures which I’m uncertain about whether or not they were posted previously will also get treated as new.

I have picture titles, categories and artist initials for all of the missing period except for March 22, 23, 30 and April 10. If you still have email announcements covering any of those days (daily or weekly), please send them along to me at the feedback address.

I hope to finish recovering from this by the end of this week. We’ll see how well I managed that. Work has been insanely busy since the middle of last month, which dragged out this process already. This week promises more of the same.

Thank you for the notes of support,

Ryan

As you probably noticed, the server died last Wednesday. Since I was in the process of building the replacement server, I wasn’t as diligent in my backup procedures. One of the things that I was trying to do with the replacement server was upgrade MySQL because the database replication was not working from the virtual dedicated server to the new dedicated server and a database upgrade will provide more powerful replication.

Alas, the virtual dedicated server had a catastrophic failure. Core systems files vanished. The entire database vanished. Backups of the database vanished. Backup scripts vanished.

So that’s the bad news. The good news is that the database is the only piece that wasn’t being replicated. Copies of all of the files were already on the new dedicated server (and two other copies as well). Even the database story isn’t entirely bad. I still have the backup from the last time I manually replicated it to my development server. The bottom line is that I lost all of the database changes from early March through last week. I’m going to try to figure out which pictures I posted in that time frame and re-post them for the same days. For pictures I can’t figure out, I’ll just queue them up for the future.

The other good news is that MacDesktops is finally on the new dedicated server, and I have most of the services I need up and running on it. If you notice anything isn’t quite right, please let me know.

On the blog front, I’m pretty sure that I had 24 or 25 blog entries previously. The entire blog database was new since the backup I had, so it looked like I was going to lose the entire thing. Fortunately, I was using Feedburner to syndicate the site, and it had most of the posts cached. I was able to import 17 posts. I don’t know which posts are missing, if any. Maybe the count included posted comments which I DID lose along with the users. I probably won’t try very hard to recover the last bit that’s missing. Getting as much as I did back through Feedburner was more than I hoped for.