Viktor Petersson.com
Archive for the ‘Howto’ Category
There’s no doubt that virtualization and the cloud is here to stay. So you migrated your entire architecture to the cloud and everyone is happy. Eventually, you’ll come to a point where you start decommission servers.
If this was an on-premise server, all you had to do was to powered it off and perhaps put it to use elsewhere (or if virtualized, simply delete it). In the cloud however, it’s tempting to do the same.
What people don’t think about however is that most cloud vendors use regular magnetic disks. This means that when you delete a virtual drive, it will be provisioned to someone else. Normally, the first thing the next person who gets provisioned your old disk blocks (or parts of it) would do is to format it and fill it with data.
However, if this person is a malicious user, s/he could restore what was written to those disk blocks, just as s/he could with a magnetic drive that has been formatted.
Therefore, before I decommission any drives in the cloud, this is what I do:
- Power off the system
- Change the boot device to a Live CD (most linux-distributions will do)
- Run shred on the device
- Power off the system and delete the drive
While shredding the drive will take a fair amount of time, we know that even if a malicious user is provisioned the same disk blocks, they won’t find any of your data.
Rack-servers make your life easier. They allow you to stack a bunch of servers into a small area while still keeping them well organized. Unfortunately, most server-vendors know this, and will charge you an arm and a leg for them. I recently had to build up a small virtualization-farm, and I really wanted to keep all server in a rack without having to spend a fortune on servers.
Enter HP ProLiant DL-120. It might not look too impressive, but for the price-point (I paid ~$800 each), it’s not bad 1U server. However, we haven’t started with beefing it up.
I love Munin. It’s a great monitoring tool, quick to set up, and doesn’t come with too many bloated requirements. It’s also very flexible and easy to write plugins for.
On Ubuntu, Munin comes with most (non-custom) plugins I use. Unfortunately, the Nginx-plugins (nginx_status and nginx_request) are incorrectly documented. In the header of the plugins, one can read that the default URL is ‘http://localhost/nginx_status‘, which certainly makes sense. The plugin even documents how one were to set this up. However, the plugin logic tells a different story. In nginx_requests, we can see that it relies on ‘hostname -f’ to look up the hostname that it connects to (which in 99.999% of all production servers isn’t ‘localhost’).
The fix however, is very easy. All you need to do is to add the URL to the Munin’s plugin configuration file. This can be done using the following command:
if [[ $(cat /etc/munin/plugin-conf.d/munin-node | grep "nginx") = "" ]]; then echo -e "\n[nginx*]\nenv.url http://localhost/nginx_status" >> /etc/munin/plugin-conf.d/munin-node; fi |
This can be run as many times as you’d like, as it only appends the config-snippet if it’s not there already.
In my case, I wanted to push this out using Puppet, so I have the following block in my Puppet Munin-module:
package { 'munin-node': ensure => 'present', } service { 'munin-node': ensure => 'running', hasrestart => 'true', hasstatus => 'true', enable => 'true', } file { 'nginx_request': ensure => 'link', path => '/etc/munin/plugins/nginx_request', target => '/usr/share/munin/plugins/nginx_request', require => Package['munin-node'], } file { 'nginx_status': ensure => 'link', path => '/etc/munin/plugins/nginx_status', target => '/usr/share/munin/plugins/nginx_status', require => Package['munin-node'], } # Fixes a bug in the plugin and configures it to poll using localhost exec { 'activate_nginx_munin': command => 'bash -c if [[ $(cat /etc/munin/plugin-conf.d/munin-node | grep "nginx") = "" ]]; then echo "\n[nginx*]\nenv.url http://localhost/nginx_status" >> /etc/munin/plugin-conf.d/munin-node; fi', user => 'root', require => [ File['nginx_status'], File['nginx_request'], ], path => [ '/usr/sbin', '/usr/bin', '/sbin:/bin', ], } |
(I also use a Puppet-template for Munin’s config-file, but that’s beyond the scope of this article.)
I hate iTunes with passion. It’s bloated and annoying. However, since it is a major revenue stream for Apple, they try hard to push it into your face as often as possible.
Since I started using Spotify a few years back, I don’t think I’ve used iTunes to play music once (only to upgrade iOS-devices before the over-the-air-updates came out).
As long as you don’t touch any other Apple products, Spotify will take over the media control-keys, but as soon as you start one of Apple products (like Keynote or Quicktime), Apple hijacks the media control-keys. It’s beyond frustrating. Instead of starting (or pausing) the music in Spotify, iTunes pops up and it makes me furious every time.
Fortunately, I just found a solution. It isn’t perfect, but it’s better than anything else I’ve found.
- Start by going to “System Preferences” -> “Keyboard” and select “Use all F1, F2, etc. keys as standard function keys”
- Next, download the application Shortcuts from Mac App Store
If you haven’t heard of Bottle, it’s a lightweight web framework for Python. It is perfect if you have a small project that requires a web interface, but you don’t want to go all in with a complex framework like Django.
Since Bottle is so lightweight, it doesn’t always have all the features you need built-in. One thing that I was missing was access control. For instance, what if you want to limit access to an admin-page to a certain IP? Sure, if you’re running you’re app behind a full-fledge webserver like Nginx or Apache, you can use it to limit access, but that doesn’t work if you’re deploying to something like Heroku.
more>>
Some time ago, I posted a question on Serverfault about how people managed and deployed ports in a large environment. Despite a fair number of comment, nobody seemed to really have the anser to this (or at least that suited my needs). It simply appears to be the case that the built-in tools in FreeBSD (primarily portsnap and portupgrade) are not built for a shared ports-tree.
Having a local ports-tree on each node seemed both wasteful and inefficient to me, yet this what was I had to resort to.
With that decided, how do you optimize the setup given this new setup?
more>>
Let me first admit that I am new to Munin. I’ve played around with most monitoring tool, but never Munin for some reason. I really don’t know why, since it appears to be a great tool. As a result, this might be obvious to seasoned Munin-users, but it wasn’t to me at least.
It appear as most stock-plugins are configured in /etc/munin/plugin-conf.d/munin-node. This isn’t the case for the Memcached plugin. Hence this post.
Start by grabbing the Memcached plugin from here. Copy these files to /usr/share/munin/plugins.
Next, install the required Memcached-perl plugin:
sudo apt-get install libcache-memcached-perl |
Now to the strange part. In order for this plugin to work, we need to include the hostname and the port of Memcached in the filename. We do this we create a symlink to the plugin that includes this info (eg. ‘memcached_traffic_127_0_0_1_11211′ if Memcached is listening on 127.0.0.1:11211).
Since I deployed this on a few servers, I wrote a simple shell-script that does this:
#!/bin/bash for i in $(find /usr/share/munin/plugins/memcached*); do ln -s $i /etc/munin/plugins/$(basename $i)127_0_0_1_11211 done |
After you’ve installed the plugin, let’s make sure it worked.
#!/bin/bash for i in $(find /etc/munin/plugins/memcached*); do sudo munin-run $i done |
Assuming you didn’t get any errors when running the script above, go ahead and restart Munin-node (service munin-node restart).
Update: It appears as this is a pretty common approach to pass variables to Munin-plugins.
In this day and age, many people use multi screens in their setup. I’m one of those people. One thing that really bugs me in OS X however, is that you cannot set one wallpaper to expand beyond one screen.
Let’s say you have two screens with 1920×1080 resolution and you have found a wallpaper of the appropriate dimension (ie. 3840×1080). Interfacelift got plenty of them.
Now, in order to properly use these, you need to slice the image into two images (one for each screen). Sure, you can do this in Photoshop, but that feels like an overkill.
Instead, let’s use the good ‘ol tool ImageMagick (you can install it using Homebrew).
convert source.jpg -crop 1920x1080 sliced.jpg |
And voilá — you know got two images. One for each screen:
- sliced-0.jpg
- sliced-1.jpg
That wasn’t very hard, was it? If you have screens of different resolutions, that is a bit trickier, but ImageMagick should still be able to do it.
I’m a big fan of FreeBSD. However, as painful it is to admit, it isn’t always the best OS to run in the cloud. Compared to Linux, you will get worse network and disk performance even with Virtio installed. There are also other issues. For instance, it is likely that you won’t get CARP to fully work (while this works perfectly fine with OpenBSD’s CARP, and Linux’s VRRP). I have written about workarounds for this issue in the past, but they do not seem to work equally well in FreeBSD 9.0.
Luckily, there is a userland implementation of CARP called UCARP that works better than CARP. It’s also very similar to CARP when it comes to configuration.
more>>
We‘ve been using MongoDB in production for about six months with YippieMove. It’s been an interesting experience and we’ve learned a lot.
Contrary to many MongoDB deployments, we primarily use it for storing files in GridFS. We switched over to MongoDB after searching for a good distributed file system for years. Prior to MongoDB we used a regular NFS share, sitting on top of a HAST-device. That worked great, but it didn’t allow us to scale horizontally the way a distributed file system allows.
Enter MongoDB. Just like most people playing around with MongoDB, we started out with a simple Replica Set, but are now in the process of switching to a sharded setup.
In the post, I will go over some of the things we’ve learned when using MongoDB.
more>>