I played with this a bit more. I implemented clean overlay on G7X, but was not able to reproduce the crash with either HDMI or analog video out. G7X does not support HDMI in rec, so it's not really equivalent.
I was initially confused for a bit how clean overlay could trigger the crash at startup, since it's only active in rec mode, but I guess the M ports start in rec by default (which is OK, the main reason we default to play on P&S is to avoid extending the lens unexpectedly)
I looked at the assert a bit more. It is the same as the one on G7X I saw when forcing analog video out. It happens in the final function called from transfer_src_overlay (g7x 100d fc0f7faa, m3 120f fc245b6, both MakeOsdVram.c Line 603)
The values that trigger the assert come from pointers that are set in transfer_src_overlay, I don't see an obvious way to check them from CHDK code.
I'm thinking about making everything call draw_restore rather than vid_bitmap_refresh, so the check disabling drawing can be in one place rather than in every affected port's vid_bitmap_refresh.
This doesn't work, draw_restore sets the guard pixel, which means the CHDK display doesn't get redrawn when it should. We could maybe use gui_set_need_restore but that puts the palette_control mod far from the eventual vid_bitmap_refresh call.
Attached is a patch that does the 3s delay without the analog video stuff.