Thomas Gericke

Es gibt Datenbanken, die stehen unter grosser Last, weil die Queries sehr komplex sind. Andere Datenbanken sind immens gross. Eine meiner Datenbanken hat sehr viele Zugriffe, sowohl lesend als auch schreibend. Heute wurde der Server an die Lastgrenze getrieben, was mich dazu zwang, Optimierungen vorzunehmen.

1. INSERT DELAYED und UPDATE LOW_PRIORITY

Bisher habe ich mich immer davon fern gehalten, diese Funktionen zu nutzen, einfach weil ich keine Veranlassung dazu sah. Warum schreibende Zugriffe verzögern, wenn sie doch auch unmittelbar vollzogen werden können? Die Tatsache, dass mehrere hunderte gleichzeitiger Zugriffe sich gegenseitig behinderten, brachte mich dazu, meine Denkweise zu überdenken.

Hier geht's weiter... » // Read more... »

Habe heute Abend Woopra getestet. Geboten wird ein Dienst, welcher Echtzeit-Statistiken von Webseiten generiert. Hierzu wird ein Javascript-Code-Schnipsel auf der gewünschten Website eingebunden, welche einen Connect auf die Woopra-Server macht und dabei Server- und Client-Daten übermittelt. Nette Idee, funktioniert auch. Für mich dennoch untauglich. Mehr dazu weiter unten.

Die Bedienung der Woopra-Website ist einfach und intuitiv. Zunächst muss ein Profil angelegt und per E-Mail-Verifizierung bestätigt werden. Dann können die Websites (anhand von Domains) angelegt werden, die getracked werden sollen.

Im Quellcode der fraglichen Webseiten muss nun der kleine Javascript-Schnipsel eingebunden werden. Sofort darauf werden die Live-Statistiken entweder über das Dashboard der Woopra-Website oder aber über ein entsprechendes Desktop-Programm ausgegeben. Ich hatte den Mac OS X Client getestet. Dieser schaut sehr ansprechend aus und stellt die Informationen nett dar.

Für mich kommt Woopra in dieser Form nicht in Frage, denn es deckt nicht mal die Hälfte meiner Anwendungsfälle ab. Konkretes Problem ist, dass ich zahlreiche Seiten betreibe, die keinerlei oder besondere Header-Informationen senden und mit der Javascript-Ausgabe im BODY nicht klar kommen. Spätestens bei einem 301-Header ist hier Schluss, das funktioniert nicht mit Javascript.

Das erinnert mich daran, dass ich mit (m)einem unlängst eingeschlafenen eigenen Webanalyse-Tool weitermachen sollte. Bereits vor einem Jahr hatte ich eine lauffähige Version, die in ähnlicher Weise wie Woopra arbeitet und auch viele brauchbare Daten bereitstellt. Aus Zeitgründen war dieses Projekt eingeschlafen. Vielleicht mache ich bei Gelegenheit damit mal weiter und erfinde Google Analytics neu 🙂

Auf dieser Seite wird einwandfrei beschrieben, wie man (s)einen Firefox (oder Iceweasel) dazu bewegen kann, PDF-Dokumente inline (direkt im aktiven Tab) im Browser zu öffnen.

Hi there!

Do you know this problem?! You have to get up at some certain time in the morning, so you set up your alarm clock to give you a friendly “rrrrrrring-rrrrring”. But you simply don’t wake up! Your alarm clock rings and rings and rings for like hours but you simply don’t hear it. Sometimes, you use the snooze button, but you cannot even remember you did it.

Well, that’s like how I feel sometimes in the morning. Bad thing, if you have some kind of meeting or appointment. Several times in the past, I sent automated e-mails to my co-workers to give me a wakeup call on my cellphone, this surprisingly always works. But it’s annoying. For me and my co-workers.

Here comes the solution:

use nagios to give you a highly sophisticated wakeup call.

Imagine, you have a linux box which is under surveillance of a nagios remote monitoring system. And imagine furthermore, this nagios system is configured to send out voice calls in cases of critical states. All you need is a new plugin (most commonly via NRPE) which tells nagios to initiate your wakeup call.

This is how I do it:

1. Add the following line to your NRPE config file on your linux box (e. g. /etc/nagios/nrpe.cfg):

command[wakeupcall]=/usr/lib/nagios/plugins/wakeupcall

2. Create the plugin and place it on your linux box at /usr/lib/nagios/plugins/wakeupcall

#!/bin/bash
#
# nagios nrpe plugin to initiate a wakeup call
# 2009, Thomas Gericke, thomas (at) thomasgericke (dot) de
#

FILE="/etc/wakeupcall/wakeupcall"

if [ -e $FILE ]; then
        echo "Wakeupcall needed. Gonna hit ya!"
        exit 2
else
        echo "No wakeupcall needed. Sleep well."
        exit 0
fi

3. Restart your NRPE service on your linux box (e. g. /etc/init.d/nagios-nrpe-server restart)

4. Set up the service check in your nagios configuration on your nagios server

I assume, you already have the host configured on your nagios system and you already are familiar with nagios’ notifications. Here’s the service check you need:

define service {
        use                     generic-service
        max_check_attempts      1
        notification_options    c
        service_description     Wakeup Call
        host_name               YOUR.HOST.NAME
        check_command           check_nrpe!wakeupcall
        contact_groups          YOUR_CONTACT_GROUP
}

I hope you’re familiar with nagios’ service check templates and such stuff.

5. Reload your nagios on your nagios server (e. g. /etc/init.d/nagios reload)

6. Create the wakeupcall config directory on your linux box (mkdir /etc/wakeupcall/)

7. Set up a file for an at-job on your linux box (e. g. ~/add.wakeupcall)

touch /etc/wakeupcall/wakeupcall

8. Set up an at-job on your linux box once you need to be woken up

at -f ~/add.wakeupcall 06:40

Your linux box will create /etc/wakeupcall/wakeupcall at 6.40AM (the very next time this moment occurs), nagios will realize it and will initiate the notification. If you have configured your nagios server to send out voice calls to your cellphone, you will receive a wakeup call.

Have fun, comments are appreciated! 🙂

Hi there!

Because some of you asked, how I realized the grabbing and thumbnailing of whole websites (here’s an example and I wrote about that in this post), this is a brief HOWTO.

Imagine, you have a Linux system without graphical support. How do you display complex graphical content and make a screenshot? Here it comes: grabbing websites on a Linux system is quite simple.

Prerequisites:

1. a Linux operating system (Debian is fine)
2. khtml2png (I used khtml2png_2.7.6_i386.deb from here)
3. a running X server (Xvfb does it for me)
4. kdelibs4c2a
5. libkonq4

This is it!

The trick now is: on a system working as a server, you usually don’t want to have a running X server. So, I just installed Xvfb, which is a “Virtual Framebuffer ‘fake’ X server”. It is running in the background and khtml2png uses its display.

First, install Xvfb and several libs:

apt-get install xvfb kdelibs4c2a libkonq4

Hit ‘y’ to solve dependencies!

Now, get khtml2png from http://sourceforge.net/projects/khtml2png/ and install it:

dpkg -i khtml2png_2.7.6_i386.deb

Then, start your ‘fake’ X server:

/usr/bin/Xvfb :2 -screen 0 1920x1200x24

Of course, you may reduce the resolution to your needs. But remember the display number (:2) you set for Xvfb.

And finally, you may use khtml2png to fetch any website you like:

/usr/bin/khtml2png2 --display :2 --width 1024 --height 768 http://www.thomasgericke.de/ /tmp/website.png

Don’t worry about the fact that the package is named khtml2png and the binary is called khtml2png2. It’s okay!

I have a little magical wrapper around that stuff which gets URLs out of a database and performs some checks. Images are save with wget and converted to PNG, websites are fetched with khtml2png. Both are saved and thumbnailed on-the-fly with PHP.

I call khtml2png via cron like this:

/usr/bin/khtml2png2   --display :2 \
                      --width 1024 \
                      --height 768 \
                      --time 42 \
                      --disable-js \
                      --disable-java \
                      --disable-plugins \
                      --disable-redirect \
                      --disable-popupkiller \
                      http://www.thomasgericke.de/ \
                      /tmp/website.png

My script is started every minute and checks if new URLs have to be fetched. It also checks if existing PNGs are older than 24 hours and, if so, the URL will be fetched and the PNG overwritten.

Just let me know, if you have any further questions.

Since I lead a team of highly specialized IT personnel, who are mostly certified LPI level 1, level 2 or even level 3, I decided it was my turn to give it a try. I passed LPI 101 (part one of LPIC-1) on 2009-01-28 and LPI 102 (part two of LPIC-1) on 2009-02-02.

Here’s what you might wanna know about how to get LPIC-1 certified:

I did not learn a lot, because I’m used to Unix administration since about 1995. But I can highly recommend to buy and read the following book: LPIC-1 by Peer Heinlein. If you’re experienced, most parts may sound very familiar, but do you know all switches and all parameters of certain commands? Do you know, which commands and what paramaters LPI might ask? No? I guessed.

Here’s (in a very brief abstract), what you should know before trying to get certified:

1. Hardware & Architecture
   • Do you know what interrupts are and where information are stored during the system’s runtime?
   • What is IO?
   • Do you know how to configure PCI expansion cards manually?
   • How to connect and eventually configure a USB device?
   • What is a Winmodem?
   • Can you configure serial settings for a modem or a sound card?
   • Do you know the difference between SATA, SCSI or even external USB drives?
   • How does the system connect, name and use them?
2. Linux Installation & Package Management
   • Do you know how to set up partitions?
   • And why exactly this way?
   • Why can some directories be on its own partition? Why should some be? Why must some not be?
   • You can install and use LILO and Grub?
   • What are the differences?
   • Can you unpack, compile and install program sources?
   • You know what shared libraries are and why they exist?
   • Do you know the difference between RedHat and Debian package management?
   • Do you know the most important switches and parameters (nearly all of them) of those packaging tools?
3. GNU & Unix Commands
   • Do you know what a shell environment is?
   • When and how will an environment be inherited?
   • Can you set and unset variables?
   • Do you know (including the usage) of all of the following tools? cat, cut, expand, fmt, head, hexdump, join, nl, paste, pr, sed, sort, split, tac, tail, tr, unexpand, uniq, wc. If not, get used to them!
   • Do you know how to create, delete, read, move and copy files?
   • Have you understood what STDIN, STDOUT and STDERR are and how they may be (re)directed?
   • Can you kill processes, stop them, send them into the background or get them back to the foreground?
   • Do you know what the process priority means and how it can be manipulated?
   • Do you know how regular expressions work?
   • Do you have problems using vi? If so, get used to this editor!
4. Devices, Linux Filesystems, Filesystem Hierarchy Standard
   • Do you know how to set up a partition manually?
   • Even a swap partition?
   • Can you add new swap space while your system is running?
   • Can you check, tune and repair a filesystem?
   • Do you know when and how partitions are mounted? And why?
   • Can you set up and use quotas?
   • Can you get and read quota reports?
   • Do you know what file and folder permissions are? And how they usually should be?
   • Can you modify them?
   • Do you know what special bits (files/folders) are?
   • Can you explain the difference between hard and soft links? Can you create, use and delete both of them?
   • Do you know how find, which and locate work? How and why?
5. The X Window System
   • Do you know where the X configuration is stored?
   • Have you understood the difference between a xserver, a window manager and a display manager?
   • Can you redirect the graphical output of a program onto another machine?
   • Do you know in which part of the xserver’s configuration fonts are defined?
6. Kernel
   • Do you know what kernel modules are?
   • How can you query them?
   • How do you add or remove them?
   • Do you know how to reconfigure a kernel?
   • And how to compile and install it?
   • What about LILO and Grub? Anything to do after you installed a new kernel? Why? Why not?
7. Boot, Initialization, Shutdown and Runlevels
   • What are runlevels?
   • Where are they defined?
   • What about logfiles during a system boot?
   • Do you know how to shut down a system safely? Even if still users are logged in?
8. Printing
   • (Just my opinion: printing has been a freakin’ show ever since!)
   • What printing systems do you know?
   • What commands are used to print or to (re)configure the printing queue?
   • Do you know what happens to you print job after you startet it?
   • Do you know when (and when not) it has to be converted? And how this happens?
   • Can you set up a remote printer?
   • Even a remote printer on a Windows system?
9. Documentation
   • Do you know the different type of man pages?
   • And have you ever heared of whatis or apropos?
   • Do you know which messages users get before or after the log on to a system? And why?
10. Shells, Scripting, Programming and Compiling
    • Do you know all BASH fundamentals including profile, variables, environment or built-in functions?
    • Can you write your own (simple) BASH scripts?
    • Can you let your script perform a loop?
    • Do you know how to compare strings or expressions (using test or if [ …])?
11. Administrative Tasks
    • Do you (exactly!) know what a shadow system is? What is it? How does it work? And why is it used?
    • Can you add, remove or reconfigure users or groups using tools?
    • And can you even do it right within the config files?
    • Do you know what the skeleton is used for?
    • Do you know what syslog is used for and how it works?
    • Can you read and/or rotate logfiles?
    • Even if they are stil locked by a process?
    • Do you know how to set up certian jobs which should be started once in the future or even frequently?
    • Do you exactly know the syntax of a cronjob?
    • Do you know how to make backups? How to store them and how to restore and use them?
    • Can you set the correct time and date on your system?
    • Even for the hardware clock?
12. Networking Fundamentals
    • Have you understood what an IP address is and how it looks like?
    • Can you calculate the netmask and broadcast of a network?
    • Do you know what IP, TCP, UDP, ICMP mean? And the difference?
    • Do you know the ports of the most common services?
    • Can you change the IP address of a running system?
    • Could you debug TCP connection failures?
    • Do you know what routing means and when it is needed?
    • Can you configure a modem as your PPP device?
13. Networking Services
    • Do you know how incoming connections are handled?
    • What is the difference between inetd and xinetd?
    • How could you migrate?
    • Can you block or allow certain connects using simple files under /etc/?
    • Do you know how mail aliases are handled?
    • And what if they have changed?
    • Can you start, stop, restart and even configure an Apache webserver?
    • Do you know how to mount or export filesystems using NFS?
    • Can you mount a samba share?
    • Do you know how to configure your system using DNS servers? How?
    • Can you maintain a SSH server?
14. Security
    • Do you know how incoming packets are handled by the kernel?
    • Can you add new iptables rules?
    • What are sockets?
    • Do you know netstat?
    • Do you know how to secure your system using xinetd, iptables, syslog, etc.?
    • Can you block logins to your machine using a single file?
    • Do you know how to secure access to files using permissions?
    • Do you know what the umask is?

This is wow, isn’t it?!

If you’re using a linux/unix system for quite a while, most of the questions should sound easy. But, indeed, nearly every tool has switches and/or parameters, you’ve never ever heard of. So, I highly suggest, you should read at least the man pages of every program, tool or task I mentioned above.

If you do know the (of course right) answer for every question, you should have no problems passing the tests for LPIC-1 certification.