Test code with proposed fix attached.
Using this code, I have done successful runs of 1000 continuous mode remotshoots on both a540 and d10. I didn't observe any memory leaks. On d10, free memory was only 64 bytes less than at the start of the sequence.
The fix is generic for dryos cameras, but for cameras like A540, a minor change in the filewrite hook will be needed. Only the a540 is updated in the patch.
This is untested on CAM_FILEWRITETASK_SEEKS cameras.
The patch includes debug logging code, but it will only be used if you #define DEBUG_LOGGING in platform.h In this case, you will need to ensure that _LogPrintf is available. This shouldn't be needed unless you encounter problems.
The basic approach is to make the raw hook wait for filewrite to clear ignore_current_write before allowing the hook to continue for the next image.
To do this, I added pending_image_data which tracks the data types that have been requested, and only clears them when the relevant hook is completely done with them. For jpeg, this is clearly only after the jpeg hook is done with ignore_current_write for the current image.
Tracking the "pending" status of raw and DNG header isn't strictly needed right now, but I think it may be useful in the future.
Note that we could probably get slightly better performance if there was a way not to block waiting for filewrite, but it would mean raw N+1 potentially being transferred before jpeg N was complete. Even if we had a way to do this, the current protocol couldn't support it.
As an aside, I noticed that chdkptp continuous shoot doesn't handle the file counter wrapping... the chdkptp side will end, but the camera will keep shooting.
edit:
updated patch to include a410, which is the only other vxworks cam with filewrite.
Also, we may be able to get rid of current_write_ignored for dryos cameras with this logic.