The jpeg buffer starts at 0x476AA600. When chdk not in exmem the camera can use almost the the entire buffer up to 0x47FFFFFF, which is a total of 9,787,903 bytes which equals to about 3 or 4 jpegs.
When chdk in exmem it is placed at 0x478e67e0 which makes the jpeg buffer only 2,343,392 bytes big and slows down the continuous shooting and also makes the jpeg quality override not work.
A faster SD card helps, that's why I didn't even notice that until now.