Looking a little on the internet I found out what a "Trackback" is exactly and I also found the WP file that allows the trackback mechanism to show up as a posted comment. This means that with "Trackbacks" turned on in WordPress, cretins can leave spam comments to their heart's content by formatting a special HTTP POST request to wp-trackback.php (thus by-passing my captcha).
I have disabled pingbacks and trackbacks on my blog for now, let's see if I get any more spam in the next month or two...