[MLB-WIRELESS] Server Virtualisation - (was RE: possible vpn and public ip address allocation option)

[Steven Haigh]

On 16/03/2010, at 10:58 PM, <mw at freenet.net.au> <mw at freenet.net.au> wrote:

> Good evening! :-)
> 
>> A number of comments about this:
> 
> And some responses...
> 
>> 1) For what you want to do, nobody should need root access. If they do,
>> then you're doing something wrong. There is no reason why someone who is
>> working on a wiki needs root level access to the operating system. Again,
> 
> I'm not talking about application administration at a user level, I'm
> talking about it from a development level.  Take a radius server for
> example.  This is something that needs to be set up and installed on a raw
> system from scratch.  How can that be done without shell access?  OK, shell
> access is not root access - that's a matter of semantics I guess.

There is a huge difference between the two. Root access implies that you have complete access to the system and are able to make changes to everything from the network interface to low level hard disk access. For obvious reasons, this is always limited to a very small number of people.

A shell account or access is just that - a shell. It is usually limited in privileges that stop people breaking things they shouldn't be breaking :)

> The thing is, if I want to build a radius server under debian (becau se
> that's what I am most familiar with) then the last thing I want to happen is
> for someone else to run a package installer that breaks all my setup work.
> And I don't want to do something to the system that will break someone elses
> work.

I agree - we had this issue a number of server iterations ago when we were using pre-built mapserver packages that were not updated with the times. This meant the entire system was running an out of date and unsupported distribution and was not being security patched. This has now changed - but mapserver is not available for this distro and it seems nobody has bothered with the pain to make it a static binary to be called as a CGI.

> The obvious advantage of a VM is that it is safely partitioned into discreet
> functional units.  And we can get many discrete servers onto a single 1RU
> server when hosting/colo space is tight.

True. The problems in the hosting situation though are many. We have 1 IP. This means you either need to get into VERY ugly configurations when you have - say - a node DB with a web interface the wiki with a web interface on different VMs.

> 
>> 2) I'd be more concerned at the moment with fixing things like spammers
> who
>> have automated the process of creating accounts and spamming the wiki such
>> as: http://www.melbournewireless.org.au/wiki/?diff=Apple This is much more
>> of an issue than having VMs everywhere and trying to reinvent the wheel.
> 
> Sure - that needs to be done as well.  But who are *we* to determine what
> members are going to do?  Is it not better to see a show of hands as to who
> has an interest in developing additional features?  Sure, state a problem
> and ask for volunteers, but that's only half the solution to getting some
> real progress in the online services for MW.

The big problem I've seen over time is that people come up with great ideas, sometimes a small but working thing is put in place, but then people wander off and everything becomes unmaintained and nobody quite knows what the mystical code does or how to fix it! There have been quite a number of things like this happen in the past - which I guess is par for the course in the type of organisation we're involved in! :)

>> 3) You seem to overlook the overhead in VMs. Unless you throw serious
>> hardware at it the performance penalty is more than acceptable for doing
>> all but toying. Neither the current server nor the newer one being built
>> supports hardware virtualisation so there is a rather large performance
>> drop in CPU cache speeds between tasks etc etc. You basically lose ~10-20%
>> CPU performance even if using hardware virtualisation. From what I have
>> seen this gets even worse when multiple VMs are very busy.
> 
> Hey, that's what you are saying.  From my experience, it usually works in
> reverse.  I rarely see a physical server using even more than 10 or 20
> percent of the available resources.  It makes sense to get the most workload
> out of what you have available, and if I can consolidate 5 servers onto one
> hardware, then it's a no-brainer from my perspective.  Not only do I save
> money on hardware, but I also save HEAPS of cash on power - it's amazing how
> much electricity one server full of disk spindles and fans uses up!
> 
> Furthermore, if you are paying for rack space, then there is further
> opportunity for serious savings there too.
> 
> And what do you call 'busy'?  According to the current web stats, the
> February daily average was a bit over 16000 hits (which is the busiest of
> the last 12 months) which equates for one hit, on average, every 11 seconds.
> That's hardly 'busy' and I doubt if that would put an average CPU at more
> than 5% load at any time throughput the day, even with a database query
> behind every single request. 
> 
> And if this mailing list also running on that system, then even in times of
> the <ahem> 'massive' post numbers we've seen recently, that would hardly be
> enough to spike the CPU over 10% in total - if we are absolutely fortunate!
> ;-D

I could beat myself in the head with a stick 12 times a day. That's only once every 2 hours - but you can bet if they happen all at once I'll have one hell of a headache :)

The current P3 800Mhz system that is in the colo has had multiple times over the last several months where it has reached a load average of 12 on a 5 minute average. Sadly, I haven't been able to catch what is happening at these times as it seems to be without a definite pattern. It may be a wave of spam at the same time as the MW crons fire at the same time as something else happens - but measuring load on a 24 hour period is very deceiving.

>> 4) Nobody working on a user space project (snmp, wiki etc etc) should need
>> root access. I know I mentioned this twice, but its such an important
> thing
>> I thought I would mention it twice.
> 
> Heheheh - neither here nor there.  If all resource components were
> virtualised on independent VMs, we could:
> 
> a.  assign a sysadmin for each and every function: db, web (even a web
> server for each app - wiki, 'locfinder', etc), mail, radius, dns, snmp
> monitor, etc etc

I can see this becoming a pain right from the start. There is a sysadmin for every function and that user is called root ;) The hard part is that people don't seem to look at these tasks with much glamour and therefore they get ignored. For example, the error_log for the MW web site grows by an average of 60-80MB per day. Most of these are none critical and therefore just get ignored. Ideally the code should be cleaned up - but that hasn't happened over the last few years or so of this happening.

> b.  make VM images available for individual developers to happily hack away
> at - learn the ropes, and make real contributions.

This is a nice thought - but what would someone gain over installing linux on any other machine in existence? There is already an API in place (that is currently down due to HTTPS being on v2.wireless.org.au until the complete migration happens) that allows people to check user auth for MW members vs the MW web site. It uses standard HTTP calls and has been fully tried and tested. It takes the username and password of a user and gives a yes or no reply based on if the user/password combo is correct, and is or is not a financial member.

> c.  save real $$ on colo and power and hardware

MW doesn't have any costs for power or colo or hardware. These have been provided free of charge to MW for the last 10 years ;)

-- 
Steven Haigh

Email: netwiz at crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299