Hey all, I've got this working so figured I'd share it. I wanted a solution that works outside scripting and there's currently nothing in place that reads the dial. I know there was an effort a while back to do this, but it never got integrated into the trunk and that ones based on a old version of CHDK.

I don't know if it's just my camera (beta S95 100h), but I couldn't get setting the mode by "_SetCurrentCaptureModeType(canonmode);" to work. I found the propcase values were wrong, and fixing them I can now read the value and match it with the right hackkey, but setting is a no go.

This isn't currently integrated with scripting or set by the GUI, but it could be setup that way. Here's my modifications to my <camera>/kbd.c

up top in the header:

static CapturemodeMap mode_dial_map[] = {
	{ 0          	, 0x00000000 }, // saving canonmode and canonmode for physical dial setting here.
	{ 0          	, 0x00000000 }, // saving override settings here.	
	{ MODE_AUTO  	, 0x0000000f }, // Same as mask for my camera, but not sure if this is always the case.
	{ MODE_P     	, 0x00000007 },
	{ MODE_TV    	, 0x00000005 },
	{ MODE_AV    	, 0x00000004 },
	{ MODE_M     	, 0x00000006 },
	{ MODE_C     	, 0x00000003 }, // previously unlisted - why?
	{ MODE_BLANK 	, 0x00000009 }, // previously unlisted
	{ MODE_VIDEO_STD , 0x0000000d },
	{ MODE_SCN   	, 0x0000000c }, // previously unlisted as in use a submode is actually selected
	{ MODE_LOWLIGHT  , 0x0000000e },
	{ 0, 0 }
};
// previously unlisted modes should be added in modelist.h

#define MODE_REG 1

#define KBD_MODE_SET    	mode_dial_map[0]
#define KBD_MODE_OVERRIDE  	mode_dial_map[1]
#define KBD_MODE_AUTO   	mode_dial_map[2]
#define KBD_MODE_P      	mode_dial_map[3]
#define KBD_MODE_TV     	mode_dial_map[4]
#define KBD_MODE_AV     	mode_dial_map[5]
#define KBD_MODE_M      	mode_dial_map[6]
#define KBD_MODE_C      	mode_dial_map[7]
#define KBD_MODE_BLANK  	mode_dial_map[8]
#define KBD_MODE_VIDEO_STD  mode_dial_map[9]
#define KBD_MODE_SCN   	mode_dial_map[10]
#define KBD_MODE_LOWLIGHT   mode_dial_map[11]

The code itself:

int set_mode_dial(int set_hackmode)
{
	int i = 2;  // 0 and 1 are used to hold set and override values
	while (mode_dial_map[i].canonmode){ //runs till the end -{0,0}
		if (mode_dial_map[i].hackmode == set_hackmode) { //find what's set
			KBD_MODE_OVERRIDE.canonmode = mode_dial_map[i].canonmode;
			KBD_MODE_OVERRIDE.hackmode = mode_dial_map[i].hackmode;
			return 1;
		}
		i++;
	}
	return 0;
	// takes time to go into effect, so be sure to wait until mode is finished changing.
	// change will hold until mode dial is moved or camera restarted.
}

int check_set_mode_dial()
{
	return KBD_MODE_SET.hackmode;
}

int check_override_mode_dial()
{
	return KBD_MODE_OVERRIDE.hackmode;
}

void my_kbd_read_keys()
{
	static long i=0;
	int temp;

	kbd_prev_state[0] = kbd_new_state[0];
	kbd_prev_state[1] = kbd_new_state[1];
	kbd_prev_state[2] = kbd_new_state[2];

	_GetKbdState(kbd_new_state);

	// Mode dial override
	temp = kbd_new_state[MODE_REG] & KBD_MODE_AUTO.canonmode; //save old dial position - using auto as the mask
	if (KBD_MODE_SET.canonmode != temp) {  //has the dial moved?
		i = 2;  // 0 and 1 are used to hold values
		while (mode_dial_map[i].canonmode){  //runs till the end {0,0}
			if (mode_dial_map[i].canonmode == temp) { //find what's set
				KBD_MODE_SET.canonmode = mode_dial_map[i].canonmode;
				KBD_MODE_SET.hackmode = mode_dial_map[i].hackmode;
				break;
			}
			i++;
		}
		// this is my own overrides, something different should be done here. Maybe gui control or just set like the default case.
		switch (KBD_MODE_SET.hackmode) {  //find override
			case MODE_AUTO:
				KBD_MODE_OVERRIDE.hackmode = KBD_MODE_P.hackmode; // I don't like auto mode
				KBD_MODE_OVERRIDE.canonmode = KBD_MODE_P.canonmode;
				break;
			case MODE_BLANK:
			case MODE_LOWLIGHT:
			case MODE_SCN:  //someday will make these do something interesting
				KBD_MODE_OVERRIDE.hackmode = KBD_MODE_M.hackmode;
				KBD_MODE_OVERRIDE.canonmode = KBD_MODE_M.canonmode;
				break;
			default:
				KBD_MODE_OVERRIDE.hackmode = KBD_MODE_SET.hackmode;
				KBD_MODE_OVERRIDE.canonmode = KBD_MODE_SET.canonmode;
		}
	}
	// needs to be set in every my_kbd_read_keys() pass.  Again using auto as the mask.
	kbd_new_state[MODE_REG] = (KBD_MODE_OVERRIDE.canonmode | (kbd_new_state[MODE_REG] & ~KBD_MODE_AUTO.canonmode));

// Below this is unmodified
	if (kbd_process() == 0) {
		// leave it alone...

I'm also working on a key mapping override, and that in combination with this hack we can make our own custom modes. I'm not sure how to make this universal as many camera's don't have a physical dial, but maybe someone has some ideas? Also, How does setting by propcase work?