Where is the "my colors" functionality located? Is this a feature built into the camera that happens in the background or something that any user can experiment with?
It's a feature of the Canon firmware available in the Func menu (icon looks something like a bell, described on page 90 of the A3300 PDF manual I have), which allows various color effects like B&W, Sepia and various color manipulations. It applies to both live view and Canon jpeg, but does not affect CHDK raw. There isn't an option to swap R and B channels in the Canon UI, but it's possible the underlying functionality can be programmed to it. It's also possible that it only supports the options exposed in the Canon UI.
Some models do have a "color swap" option, but that does single colors, not channels.
Instead of modifying the live view data, lets say I take an approach where I manipulate the DNG data. Is it possible to view the color swapped image in the camera preview menu then?
Manipulating the CFA pattern in CHDK would only affect programs reading a CHDK DNG. Canon jpeg would be be unaffected.
If you modify the raw data in the raw hook, that is reflected in the Canon jpeg. You can do this using CHDK script:
https://chdk.fandom.com/wiki/Lua/Raw_Hook_OperationsReading / writing individual pixels from Lua is very slow, but may be useful for prototyping.
Writing dedicated C code would be somewhat faster, but processing the entire raw buffer will still take significant time. I'd guess on the order of a few seconds.
My main goal is to get an idea of what the final image might look like with swapped Red and Blue channels. I don't want to modify the actual image, unless it is easy to undo later.
If you swap it in the raw data, the saved jpeg will be changed. You could presumably reverse it with image processing software, but there would likely be some loss of quality. If you implement it in C code, you could decide whether you wanted to swap before or after CHDK raw/DNG is saved. In any case, you could reverse the process on raw data later.
If you only want to change what is seen in playback, there's another option: Like the live view, the playback frame buffer is YUV 411, and is available from the vid_get_viewport_fb_d function. Unlike the live view, it's only updated when needed, like when you switch images or do other things that erase the screen so it would be much less performance sensitive than the live view. It should be quite manageable to implement a hotkey or something the transforms the current playback frame in some way.
I also have no idea where to begin reverse engineering such things. My knowledge of assembly and low level languages is very limited.
If you have an interest in exploring the firmware, Ghidra is very helpful, see
https://chdk.fandom.com/wiki/Firmware_analysis_with_Ghidra but it's certainly something that takes significant time and technical knowledge to pick up.
