mweerden, nice to see you working on this again.
Some comments below, I'll try to give the other issues a more detailed response when I have a bit more time.
actual live image dimensions (= bitmap dimensions?)
There is no simple, general relation between bitmap and live dimensions. The CHDK code is currently very ugly in the way it deals with live view dimensions. The best thing to do would be to clean this up before trying to build a PTP interface for it, but this would be a big job.
I wonder if you'd be better off with a new ptp function that's something like "get video frames" which takes flags specifying which items you want (bitmap, viewport, bitmap palette if we can get it) and returns a structured result. The camera side could notice when aspect and possibly palette changed, and automatically send updated information. edit: batching them all together should be more efficient.
Some discussion of viewport layout is in
http://chdk.setepontos.com/index.php?topic=5415.0 and
http://chdk.wikia.com/wiki/Frame_bufferspalette (either as a whole or just CAM_BITMAP_PALETTE)
The actual camera palette would be best, but requires find the relevant function for all cameras. Also, it changes at unpredictable times, so it wouldn't be clear when you'd need to update it. Doing it every frame would be wasteful, but as above maybe the camera could notice and only send it when requested or it changes.
I would also like to note that my experience with installing and figuring out how to draw an image with IUP has been quite frustrating. In the end I just piped the data to an external program. It definitely has resulted in my not being a fan of it. Just wanted to get that of my chest.
IUP documentation isn't great. My understanding is that it doesn't really support drawing directly to a window. To do this you should either use CD
http://www.tecgraf.puc-rio.br/cd/ or the GL canvas, or roll your own function in C that draws directly the the hwnd or equivalent.
That said, I think the ability to easily receive lua values from the camera and use them directly is very high, and once you get used to it's oddities, it's pretty convenient for general gui stuff.
In include/platform.h some vid_* functions return long, some int. I don't think there is a reason, but I'd like to be sure
No point, long and int are identical. I would really like to get rid of all the longs from everything except possibly functions that are defined as 'long' in some external standard.
Proper serialisation in Lua looks nice, but also complicated/expensive. And it would indeed bump up the PTP version. Perhaps an option is to leave implicit serialisation as it is now and just provide a proper serialise function for Lua programmers to use if they want to communicate more complex things (as is the case in most languages I believe).
My current thinking is to add an option to lua exec, which uses serialize for everything (except maybe strings, because escaping them does add quite a bit of overhead for not much value.) The serialize function really isn't very complicated (currently 100 lines in chdku), and the overhead for simple values should be negligible compared to ptp overhead. It doesn't handle cyclic tables or code, but there's not much need for those. Handling cyclic tables wouldn't be that hard if you needed to, and you can just override the serialize function with a smarter one.
Sending the values as lua leaves them quite human readable.