/etc/logrotate.conf
I set the interval to monthly and rotate 24, but this did not help.
I seems like half of the log files are rotated by savelog which is triggered daily by cron. So I moved the file
/etc/cron.daily/sysklogd
to cron.monthly and edited the number of cycles to 120 (10 years, -c 120. This seems to do the trick. :-)