Before you read this post, you should know that I work as a network administrator guy for a small local IT company. I am there to cater to the needs of everyone else, as all my other “collegues” in other companies. Well, today was the day when technology decided not to help me out…
Everything was fine this morning. The servers were up and running, so I had some spare time to plan the work on a few projects we’ve got running with customers (While I’m the network administrator of the company, I’m also the only tech guy capable of designing and implementing computer networks and everything related. There’s the manager but he’s far too busy with… well, manager things). As I was thinking that this would be an easy day – preparing for future work makes work easier – problems started to come in. First, there was a SQL Express database that started to moan. Now, I’m far form a SQL server expert. Yeah, yeah, point your finger at me and say it’s my responsibility to know about it. Well, either way, this particular server has been up and running for the last four weeks. It’s a server used by our “temporary development team”; a bunch of students working on a graduation project. All went fine, until today; when they received complaints about the database in question not having an owner. As uneducated on the topic as I am, I always assumed that the person who creates the database is the owner. A quick peek learned me that the database was indeed “ownerless”. So I fixed that problem and had to move on to the next request.
That request came from another student; one of the two students that are studying to be come network administrators. Now, I’m not the kind of network admin to play nice with students – they’re of no benifit to me and only create more problems and work for me – but this particular guy is okay. He’s always been friendly, open for suggestions but smart enough to find it’s own path; and he’s somehow found a way to not annoy me and get things done.
So when he asked me if he could have a server to test things on, I gladly helped him out. Now, when I say “server” I mean “Virtual environment which will be deleted the minute his internship at our company ends”. He asked for an Ubuntu “Server”; so I copied one of the existing servers in our VMWare Server 2 environment. That took me longer than I was comfortable with, because for some reasone the virtual machines were being “locked up” even after closing them. I had to delete certain folders before I could copy them; and then noticed something disturbing. The status indicators for the server – the physical machine – were spiking. Processor usage was at 100%; and RAM usage was somewhere around 80%. A quick analysis learned me that one of the virtual machines, a stripped Debian machine, was using the maximum of it’s resources; and apparently he was killing the whole box while doing so – I don’t know how that’s possible since you assing only a certain % of physical ram and processor power, but it really happened. It ended up freezing all of the virtual machines on the box; and the box itself. None of the Virtual Machines on it could be managed and the physical server wouldn’t listen to the commands either.
so I had to do it; I had to restart the server. Luckily this server is only a production / testing server; mostly for web appliances so restarting the server wasn’t that big of a deal – I hoped. Before I could restart the virtual machines, however, I had to leave as I had to deliver something to a customer. I had to fit one iMac and one Fujitsu Siemens Tower server in my little car (google VW Polo to find out how little). After a half hour drive, we came up to the company in question, and after we rang the bell and waited for 5 minutes I realized our dear customer did it again. It was the third time (out of four) that they just weren’t there when we had an appointment. So, we returned to the office, after wasting 1 hour and a half.
Back at the office, bad news arrived. Some people wanted to see a demo I prepared; a Magento based webshop. While it’s not a “paying” customer per sé but a business partner of my company’s owner; the manager still insisted that “I showed him the demo as soon as possible.” Only one little problem… the demo was, and is hosted on one of the Virtual machines that had to be restarted. Most of the virtual machines started just fine and didn’t seem to have suffered any permanent damage from the cold restart, but one Virtual Machine got hit bad – a not so important svn server that’s only used for document sharing between 2 or 3 people. I had to ignore that problem, in favor of the demo, though.
Not that the demo ever took place. The business partner in question was about to leave and wanted to see the demo all “quickish”; but we kept loosing the connection to the demo site. I later found out what the problem was; 2 minutes after he left. By starting ALL my Virtual Machines on that box; I started a few that have the same IP adress; making it nearly impossible to view the demo store. Bij the time I solved all the issues around the VMWare server, it was time to go home. That svn server will have to wait until tomorrow.
In IT, things never go as planned…