There is no difference between playback and rec modes in behaviour.
I'm not clear what you mean, it's not going to switch to playback on connecting if it's already in playback. But anyway, the point is that the automatic switch playback can interfere with switching the camera to rec from chdkptp later. To avoid that, power on the camera playback before connecting USB or power on using the playback button with the camera already connected, and only use chdkptp to switch modes while connected. Behavior varies a bit between models, but the preceding should safe on all of them.
However, the camera is not detected by the system as i. e. there is no additional camera available (only the built-in device).
If you mean in the regular UI outside of chdkptp, this is the expected and intended behavior. The inability to switch to rec mode is caused by the normal software accessing the camera. The udev stuff prevents the system from automatically accessing the it. You should see it available in the chdkptp list.
The operations in my previous reply solved the access issue.
Meaning you can use the chdkptp 'rec' button to switch to rec mode and see the live view now?
Hmmm... @reyalp I don't think OBS is the right tool to do this. I simply want the camera to act as an additional device on the system... USB might not be the brightest approach but nor is HDMI as best as I can tell...
Right, what I'm telling you is that chdkptp doesn't do that. It talks to the camera using a custom protocol and does not provide a driver that other software would recognize as a standard device.
On Linux, you can probably achieve that using some combination of chdkptp lvdump, piping to something like ffmpeg and V4L loopback, but it will be very hacky, and the resulting quality will be low. The resolution of the live view is effectively around 640x480, the frame rate is not fixed, and you are likely to see tearing.
There are some examples of piping to ffmpeg toward the end of USAGE.TXT, though you'd some additional stuff to send to V4L. There's an example in of the general approach in
http://www.gphoto.org/doc/remote/ (only for the ffmpeg + v4l related commands, I don't believe gphoto currently supports the CHDK live view protocol)
You will likely want to do this in CLI mode rather than the GUI.