This is going to be very difficult. You probably would be better off using a camera system with documented interfaces. Even if you value your time at minimum wage, this is like to be a far cheaper option.
Where can I find more info about the hardware & software architecture of the SD780 IS? (emphasis on the control and data flow in video capture mode)
Disassemble the firmware dump and start hacking. Note that all CHDK supported cameras are pretty similar (even the when they changed the OS from vxworks to dryos most "camera" software didn't change much...)
The CHDK code will give you an idea of what aspects of the system are known.
Is the video compression handled by a different ASIC, or can I have access to the raw image data in video mode?
Dedicated hardware, as far as we know. "Digic" has an ARM 946e-s where the regular firmware and CHDK run, and some dedicated image processing hardware we currently know very little about. Given the relatively slowness of the general purpose CPU, it's safe to assume that the heavy lifting of video processing must happen on the other part.
Assuming I don't need to encode and store the video stream - how much CPU time should I expect to have available for image processing?
Hard to quantify, but not very much. The ARM CPU is not particularly fast. Some relevant examples in CHDK would be edge overlay, zebra, histogram and shot_histogram (histogram on the raw buffer).
