Adieu to the database errors!

Because Pete Stevens at Mythic Beasts is brilliant, the period the site was down for while he moved our database and web servers lasted for under 5 minutes (and in the time between his bringing everything back up, dialling my phone number and my picking up, we had 30k hits, which graphically illustrates our need for the move…)

We think everything is working smoothly now, and all the functionality with commenting and posting on the forums should be there. If you see anything peculiar, please leave a comment here or email us using the contact page (link at top of page).

Liz again, 15:34h. Because so many of you were asking for details here and on Twitter about what our server setup is, Pete’s just mailed me this:

Thanks for the nice comments. Since they’re all going to ask, feel free to post on my behalf,

The new box is dedicated to raspberrypi.org. There’s a fairly quick six disk RAID10 array with battery backed cache and enough RAM to cache everything on disk without really running short. Some server porn from /proc

# cat /proc/meminfo  | head -1
MemTotal:       99202196 kB

# pvscan
PV /dev/sda5   VG mainvg   lvm2 [4.05 TiB / 3.95 TiB free]

# cat /proc/cpuinfo | grep processor | wc
8      24     112

# w
15:12:37 up 17:08, 10 users,  load average: 1.39, 1.09, 0.86

Whilst the box doesn’t have that much CPU, everything should be served direct from memory so we should be able to keep those CPUs nice and busy. I have a plan to add lots of quad core web processing boxes on the front if it turns out to be necessary.

Now being the generous lot you are you’re probably about to hang me for incorrectly configuring the kitten morphing plugin for wordpress that I didn’t know about, and the few minutes where we probably threw comments away while I did the db dump/restore. Flame on…

Pete

It’s almost as if he’s read the comments section here before…but as far as we are aware, all kittens are being morphed correctly. 

111 comments

Avatar

hooraaayyyyyy :-)

Avatar

That’s very similar to the noise I’ve been making for the last few minutes.

Avatar

i noticed that site is loading faster

Avatar

Great :) Next on the list is to ditch the crummy forum software and get a design without 100 pixels padding on the content area :p

Top work guys – Mythical Beast are awesome…might have to get a Mac Mini collocated there :D

Avatar

[quote]might have to get a Mac Mini collocated there [/quote]
Have you ever heard Sheldon about the guys @mac stores?

http://www.youtube.com/watch?v=-Q9goT1-jrY

Avatar

Make sure you tell mythic beasts that you came to them from raspberrypi.org so the foundation can get three months free hosting

Avatar

Speed increase is fantaaaaaastic!

Avatar

Hooray.

Only comment is that when browsing mobile I don’t see the password field here, so I can’t post…

Avatar

Well done Pete, and Liz

Avatar

Thank you! it is suuper fast now :D

Avatar

From my end doing admin stuff, it feels like upgrading from a tricycle to an F19. I’m almost as excited about this as I was about launch…

Avatar

Wow, you actually replied one of my posts? Wow, that’s flattering! and yes, I dont even doubt it, you passed from a LowEnd VPS to a Dedicated, isnt it? it feels like you had the 15 dlls/yr VPS and now a dedi.

btw, your password no-spam plugin is broken… sometimes it doesnt accept the pwd… or it is my cache?

Avatar

I had trouble with that password too yesterday.

Avatar

Hmm – might be our cache, actually. Let me just whitelist you, and tell me if you have any more problems.

Avatar

testing password acceptance…

Avatar

LIz, for me, firefox under Linux it’s quite structural that it almost never works. I have to reload the page and then immediately submit the quote and password for it to work….

Avatar

Hmmm. That post was more or less the first that worked…..
And this one is back to the old: “hit back and enter the password” (which would clear the comment already typed, and not help getting the comment posted.)

Avatar

Hey – some tricycles are rather nice thank you very much ;)

I do 40mph+ every day on my way home from work…
http://www.icetrikes.co.uk

Forums are somewhat more browseable now though :D

Avatar

I misread that last line as “I’m almost as excited about this as I was about lunch…” and I thought: typical foodie :D

Avatar

Lunch was, in fact, some most excellent home-made gazpacho poured over a bit of shredded white crab meat. I was pretty excited about that too.

Avatar

Oh NOES !
It’s faster ! It’s doesn’t crashes anymore !
What on earth we will going to do without those insightful posts asking if the forum runs on a RPi or a BBC Micro ?

May Raspberry Pi become so popular and widespread that this will be the first upgrade of many to follow.

Avatar

Link to Shop still reports:

Down for Maintenance
This store is currently unavailable. Please visit our main website at http://www.raspberrypi.org.

Avatar

That’s because it’s closed until mid month, when we’re getting someone in to run it. Nothing to do with this server move.

Avatar

Can you please add “Will be closed until end of March” to that notice on raspberrypi.com – it will save you a lot of twitter & forum replies :)

Avatar

Make that closed until the end of June. Then people will get a surprise when it opens earlier.

Avatar

Oh No! How am I supposed to get my full-color sticker to go on my Pi? Sorry, the white silk-screen does not do much for me ;)

PS: I keep on getting “Error 1: Click back and type in the password.” Twice as much work to post. Also, the comment window does NOT auto-scroll down very well when you type. I have to keep hitting the down arrow to see what I am typing. Firefox 10 on Red Hat Enterprise 5.

Avatar

I always have this error while reading, but if I reload the page and immediately post a comment it works. (maybe the password has a timeout I don’t know).

… coping this text to the clipboard, just in case it fails again :) …

Avatar

Comments doesnt work … get an error

thats why wrote an mail to you

Avatar

Wow, that’s fast now.

May I ask on what kind of machine the site now runs? And what was used before?
Oh, and can we get some statistics please?

Avatar

How can the server have 99GB RAM ?

Avatar

96 GB of RAM (most likely 12 sticks of 8 GB) – minus about 1.4 GB used by the kernel (and video card ?) * 1024 * 1024 = 99202196 KB

Avatar

Well, to be fair it’s only 94 (and a half-ish), but seeing as how I can easily buy 8 GB sticks of RAM, and most servers have about 8 ram slots, then I can easily get 64 GB of RAM. And it’s been a while since I looked for server RAM so it’s not unreasonable for there to be 12 GB sticks of RAM (12*8=96 GB with a little loss bringing it down to ~94.6)

Avatar

There were 99GB of RAM in the box, but then they took some down and passed them around. Now there are 94GB of RAM in the box.

Sorry.

Avatar

The server still has 6 free RAM slots, currently it has 6x16GB although we’d need to install another processor to use them.

Avatar

SSD drive on a server would help even more. We saw about a 500% increase in db IO just upgrading to a simple Intel SSD.

Avatar

RAM is even better.
cp my_ginormous.cnf my.cnf

Avatar

If you don’t let the DB server hit the disk, you have no speed to gain at all from faster disks.

Avatar

Mythic Beasts are brilliant; they host the O2 Joggler forums where I spend a fair bit of time, and they never miss a beat.

In fact, while you’re waiting on your Raspberry Pi in the mail, you could grab a Joggler as a friend for it. 1080p on a picture frame format device? Yes! :)

Avatar

I thought you were serving this site off of some thousand Pis. Ain’t that the one true reason we have to wait for ours?

Avatar

I can’t really feel any speed difference (maybe because of my location), but I’m glad I won’t get those nasty database errors anymore.

Avatar

That’s some good server porn right there.

Pete Rocks!

Avatar

Mmmmmm, server porn.

Avatar

Porn is just going to make the load problem much worse.

Avatar

I’d have liked a full and complete /proc/cpuinfo though…

Avatar

Hi! Sorry if this is a known issue. I’ve been lurking for a while and decided to register for the furums, the only issue (however slight) i’ve seen so far is that avatar selection is not working.

It says:
“Sorry, uploads disabled! Storage location does not exist or is not writable. Please contact a forum Admin.”

Other than that congrats on the release and new server update. I quite enjoyed the 6AM rush on that eventful morning, looking forward to getting my pre order :)

Avatar

Avatar uploading was never enabled – use gravatar.

Avatar

psergiu, odd because after I registered a few days ago from the forum page I clicked “profile” (next to logout) and then select your avatar. I’ve never visited gravatar.

Avatar

eek, just noticed the avatar I upload that way only appears in the forum, not over here.

Avatar

I really love that somehow the phrase “kitten morphing” is important to this website, and propose that this delightful lingo is somehow included in RasPi hardware/software. Perhaps an animated .gif Easter egg?

Avatar

This is a test of the Emergency Broadcast System. If this had been an actual emergency, you would have been instructed to tune to your local Emergency Broadcast Station. This is only a test.

Server porn! As George Takei of Mr. Sulu fame would say, “Oh, MY!!!” OK, _now_ you folks are starting to really “get” us! :)

Forum responses are lightning fast, now, even out here in a remote area in California at the bitter end of a wimpy cell phone link. Thank you, thank you, thank you!!!

Well, let’s see if this makes it on-the-air! Fingers crossed … and Post Comment …

Awwww, crap! The dreaded “Error 1: Click back and type in the password.” error. Well, at least the error comes back lightning fast now. I only take back some of my plaudits … for now. And Post Comment again … and again … and …

Avatar

“Error 1: Click back and type in the password.”

One workaround is to log into the forum page first. If you do the reply box will show your login and you get to skip password etc.

Liz, I can log out from the reply box. It would be brilliant if we could log in from the reply box or register there.

Avatar

You mean Mr. Sulu, of George Takei fame? The man has over 1.25 million followers on Facebook, most of whom are too young to remember Mr. Sulu.

Avatar

You know that these commands accept file as argument and reading them with cat is a waste (forcing the kernel to fread them twice (second time in ram, sure) and forking a process)?

head -1 /proc/meminfo
grep processor /proc/cpuinfo| wc

If it was my sysadmin, I’d be worried about his skills.

Avatar

And as predicted, here’s this comment thread’s first troll. Thanks for your not-at-all-helpful input, Tom. I hope getting it off your chest made you feel better.

Avatar

But Pete might have been running # cat /proc/meminfo | head -1 thousands of times per second for weeks even.;) #TrollSocialSkills4Hire

Avatar

But Liz! It’s terribly inefficient! Clearly that should have been handcoded in assembly for optimal performance and minimal memory footprint…

Think of the kittens!

Avatar

I generally do things the multiple-command way too; I find command lines easier to edit when each command does a single thing separated by pipes – I use multiple “grep” processes with simple matches rather than one complicated regex too. That also has the beneift that it scales across multiple cores better :-)

Avatar

Seconded

Avatar

The previous command I’d typed was ‘cat /proc/meminfo’, by my count ‘up arrow’ ‘| head -1, is 10 additional keypresses whereas your command requires 21 key presses. Since this is an educational project I’ll expect 2000 words in my email by Monday with a detailed calculation of the number of cpu cycles consumed in each case taking into account both the command run and the cycles consumed to process the shell into which it was typed. For additional marks you can include the cycles taken by my laptop and the two ssh processes in order to communicate the extra key presses over the server. Further marks will be available for a reasonable estimate of the additional load on network devices for the additional internet traffic.

I particularly look forward to your discussion regarding the relative costs of cycles on (a) the server cpu, (b) my laptop cpu, (c) embedded network processors, and the cost utility function you suggest to weigh off usage between them.

Avatar

That’s a proper response!

Of course the cost of your time also needs to be factored in, and the fact that both (system cost and human cost) require an opportunity analysis to determine whether the cost in terms of time/CPU was actually “prepaid” (i.e. didn’t stop anything else happening because it used idle time)

Avatar

I thought about that but the MBA course is next week.

Avatar

As somebody who has has spent far too many hours of their life trying to translate ‘geek’ into ‘bean counter’ and vice versa my opinion would have to be that the ‘Cost of Tom’ far outweighs any possible benefits.

Avatar

Masterful serve. :-)

Avatar

For me at least, as a poor simple sysadmin these past 19 years, treating everything as a pipeline, with “cat” as the simplest way to start a pipeline from a file, frees me from a small cognitive load that allows me to get on with caring about stuff that actually matters. If there was a gigabyte of stuff squirting through that pipe, or that pipeline was going to be run in an inner loop of some long-running job, that’s when the caring would start.

Instead, I’ll concentrate on getting 1300TB of data migrated from our old storage system to the new one, at 500MB/sec… that is a task worth tuning.

Avatar

Tks Pete :)

Avatar

Finally Pete unveils the specs for the secret Model C! ;)

Absolutely great work! – everything is super fast again just like in ye olde days

Avatar

I thought your servers were running on a Raspberry Pi.

Avatar

Excellent work – thanks ! :D

Avatar

Smokin’ !

Avatar

The site is definitely faster, no DB errors.

Be sure to remember RAID is not a backup, seen a number of people bit by confusing the two.

Avatar

Well, I’m sure the professional webhosting company looking after the server probably know what they are doing….

Avatar

Well, you were also sure that the web-site would do just fine on the product release time, and also you were sure that your designated distributor’s site could handle the traffic … :)

Avatar

They aren’t in charge of the distributor’s websites, and they knew that their site would be swamped.

Avatar

Not to mention the fact that our site *did* do fine at release – we had realised we wouldn’t cope with the traffic with the regular site, so we swapped out to a static page for the day.

Avatar

I’d recommend trying http://www.shopify.com/ as a storefront.

Avatar

Thank you for upgrading the server!

Avatar

Maybe a picture of the server?

Avatar

Since Pi-day is next week, I’m thinking there should be a contest for most creative kitten morphing submission, with the winner announced at Pi-time (3/14 1:59:26). The prize being up to Liz and the gang, whilst a Model A or B would be obvious, perhaps some of Liz’s favorite chocolates? :)

Avatar

Pi-time if you’re in the US and you write dates month before day, perhaps. Putting my picky hat on, I’d also say that if you’re going to use the 24-hour clock (given that you haven’t specified AM or PM, you must be), then it should be written as 01:59:26 (note leading zero).
However, if you’re here you must be, at least in part, a bit of a geek. Being so, you would know that there is an international standard for date notation. Using this, the time you suggest should actually be written as:
20120314015926
So, nothing special there then!

;-)

Avatar

To get pickier: Wouldn’t you *have* to use US dates for “Pi-time?” Day before month can’t work — there aren’t 14 months in a year. Also, moving the slash — 31/4 can’t happen – only 30 days in April. And… pickier yet: shouldn’t pi time really be 3/14 1:59:27? <– rounded up; as the next digit is a '5'. At least that's what I learned way back in school… (and back then I had pi memorized to almost 100 places; I can only remember about 25-30 or so now – still more than good enough for the math I can't/don't do any more! ;-) )

BTW, great job on the server upgrade!

Avatar

Off topic, but the RPi Fedora Remix download instructions are *buried* in the forums:

http://www.raspberrypi.org/forum/distributions/fedora-arm-on-raspberry-pi/page-7#p49878

No links on the download page.

Avatar

Sweetness!

Avatar

Feel free to mis-morph all the kittens you want. I’m a dog person.
Woof!

Avatar

cat /proc/cpuinfo | grep processor | wc
vs
grep -c processor /proc/cpuinfo

Avatar

Great! Firebug reports I’m getting consistent http://dazzlepod.com/ip/raspberrypi.org/

Avatar

Hmm, that comment looks wrong. I posted “..consistent 900ms loading time..”

Avatar

I want a morphed kitten! Wait… what do the kittens morph into?

Avatar

Evil kittens.

Avatar

Is the kitten morphing native in Puppy Linux?

Avatar

Does this morphing kitten involve a box, a radioactive source and small flask of poison?

:)
ps makes my 32gb server seem woefully inadequate…

Avatar

Thanks, Pete! Forum works like brand new.

…oh, wait… I guess it is! :-D

Avatar

Sounds great with faster servers, but have you considered a reverse proxy as Vernish or something similar?

Avatar

If you read the source to this page you’ll notice that Wp-Supercache reports that it takes 0.5s to generate the page, and most of the rest of the time we’re serving statically cached copies. Some rough benchmarking on the server reports that we can serve something in the region of 10,000 hits/second of cached pages and about 15/s of the uncached pages. Varnish would allow us to serve the cached pages more quickly at the expense of a more complicated setup, and to be fair, serving the cached pages quickly hasn’t historically been the issue…

Avatar

Plus, of course, we can’t cache the forum (which is a big database user) because users need to be able to see new replies on it!

Avatar

As another commenter said, changing from Apache to something like Nginx will improve performance significantly. (I was recently looking at some benchmarks between different webservers, you really do need to think about something other than Apache for high load sites)
Also, using Microcaching for things like forums also help. http://fennb.com/microcaching-speed-your-app-up-250x-with-no-n
Most users won’t know that they are viewing “old” cached information, so you can serve “old” (say upto a minute out of date) cached information to most users, and only server newer information to users who post. (Simple check if a user has posted as normally you create a cookie when they post)
Also, if you look at Pagespeed reports, there is heaps more you can be doing to decrease server load. Like allowing the browser to cache some items such as pictures and some of the javascript files (just need to set a Expiration as none is being sent currently).

Other than Pete, I’m sure there are some keen enthusiasts here who are also system admins that would be willing to help a charity…. You know where to contact me.

Avatar

100 GB Ram, that means, that they put 400 board B in cloud … :)

Avatar

Which also tots up to $10,000 (about half the likely cost of such a pimped out server) and a cumulative 280 GHz which is likely to be about 10 times that of the server.

Of course this is a silly comparison in a thousand other respects, but still, perhaps there is a viable role for Rpis in web clusters of some kind, at least static ones.

What about comparative figures for power consumption?

Avatar

What is the purpose of 4TB of hard drive if all the DB fit in less than 100GB?

Avatar

Futureproofing. Disk space is pretty cheap, might as well put it in there whilst you are upgrading the server. Then you don’t need to go offline to add them later.

Avatar

It also happens that often if you buy enough disks to get a certain number of IOs per second from them in a RAID setup, you end up with a lot of capacity on that RAID array without even trying. Disk capacity has grown like crazy but IOPS per disk has grown at a much lower rate… a cluster filesystem that I help run has more space for metadata storage that we’ll ever use, just because of the number of disks required to get the IOPS numbers we wanted.

Avatar

Sooo much faster now, thank you so much, well done Pete !

Avatar

It doesn’t help the server when you use WordPress…

Avatar

I really hope you are not paying for that!

Avatar

For what?

Avatar

For that server. Some of mythic beast servers cost 6K pounds a month! We are paying 0.3K to a local company for dedicated hosting with half the ram and reasonable disk space.

Avatar

Well, we have to pay something….not the sort of thing you get for free.

Avatar

Mate, you need to fix your software and site. I have Nginx on a 1.6Ghz Atom with 1GB RAM that pushes 1500+ requests per second for static HTML.

Is there really any reason why you need 96GB RAM or even a database to serve what essentially should be a few static pages? I absolutely guarantee you I could manage your traffic on a sub-£100 VPS no problem whatsoever.

Avatar

Switching from Apache to Nginx will give you a speed boost. Or putting some Varnish servers in front of the Apache.

Avatar

Please get facts straight before telling other people how to do their jobs. It’s not just a static blog site – it’s has a very active forum and a commenting system. Please refrain from insulting people, it WILL get you banned.

Avatar

Wow, that sure is alot of RAM!
Hopefully the servers’ usage will get nowhere near that amount!
Can’t wait for the RPi’s to come back in stock and me get my hands on one.
As for @Ryan, has he not seen the comments system, or the Forum, or the Powered by WordPress banner?

Leave a Comment

Comments are closed