Some update (more like a blogpost).
I've been working on the movie quality control. I've found the routines that control the recording bitrate, and managed to implement CHDK's bitrate and quality overrides.
(Side note: these routines are very similar to those in Ixus30/40)
Everything would be great, if the camera would not crash when a greater amount of these overrides is in effect. I use a "grid" pattern on my monitor to create a hard to compress scene. With the bitrate set to 1x (original) the recording is stable (the highest original quality is a bitrate with average (not more than) 64kB/frame in 640x480 mode). However, I got crashes with 0.25x bitrate, 1.5x ... 2x bitrate, and with constant quality (~88-99%) filming that pattern.
I tried several things, but the unstable operation has not changed.
The trouble is, I simply can not find the reason how my modifications could have such an effect. I suspect that there's a bug in the firmware routines (or even worse, in the DIGIC itself), that I'm exposing.
The crashes seem to be memory corruption related. I either get a regular assert (= camera restarts) or an unresponsive camera with the display showing garbage (either the display is "reprogrammed" to show some random memory area or the liveview buffers are overwritten by the misbehaving movie routines).
In VGA mode, the trouble begins when the size of an individual compressed frame reaches ~204kB. In such cases (when I stopped the recording before a potential crash), those frames are corrupted and the AVI also gets corrupted (the AVI index is not affected). Interestingly, there's a constant (observed value is 0x32000 = 204800) in the parameters passed to the encoder routine (I tried to play with it without success). The issue may be related to the buffering of the frames (the recording buffer is almost 20MByte), the crash always seems to happen after the amount of recorded data exceeds the buffer size.
No new build this time (the only usable change would be that recording mode change works now).
edit:
Increasing that limit (0x32000) before the compression routine AND reporting a framesize not more than the limit to the following routines seems to stabilize the recording. (This destroys the oversized frames' lower parts of course.) Maybe the compression code can't reduce the size of those oversized compressed frames correctly? Faulty DSP code?