Dark frame control

[reyalp]

I've managed to get my a540 to save a raw before and after dark frame subtraction. This should allow you to create a proper dark frame by subtracting the the after from the before, which then could be applied to other images.

With the dark function identified, it may be possible to do dark frame only (e.g. shoot_darkframe in a script), but I haven't explored that yet. If someone is interested in doing that, there appear to be some test routines that might be helpful. See FFEA31F4 for example.

very very rough hacky diff (via tortioisesvn, from juciphox branch) attached. The unprocessed image is saved with the normal raw file name, and the processed on is saved as DRK_*

The jpeg always has the subtraction done, for obvious reasons.

One thing I noticed when doing this is that the camera actually handles a third possible value of the location (0x6130) that controls dark frame subtraction. CHDK sets it to 1 or 2, but there is clearly code for handling 3 as well. I tried forcing it to 3, but wasn't really able to figure out how it's different.

edit:
test images can be found at Picasa Web Albums - reyalp - CHDK cropped, but the jpeg has original exif info. 15 sec exposure @F2.6

[LjL]

Given that the assembly hacking is beyond me for the time being, I'll ignorantly ask if what you're doing is change the OS's own routines in assembly to make it call CHDK's raw hooks twice, once before and once after the dark frame is taken...?

It's just not clear to me just how much of the firmware we can modify. I know we aren't actually (permanently) modifying any, but still we're subtly changing some, aren't we? And specifically I'm wondering wether it would be at all possible to completely skip the JPEG processing and saving routines.

[PhyrePhoX]

very cool. how did you find it?

[jetzt]

Seems that someone read my (and others) feature request and had an idea about it, awesome.
(I hope I didn't translate it wrongly in my head: this is about a darkframe without lens cap? )
 

[LjL]

Yeah, although it's also taking a "real" photograph that gets subtracted out.

[reyalp]

Given that the assembly hacking is beyond me for the time being, I'll ignorantly ask if what you're doing is change the OS's own routines in assembly to make it call CHDK's raw hooks twice, once before and once after the dark frame is taken...?

Exactly. Specifically, copying the disassembled ROM code and modifying the copy.

It's just not clear to me just how much of the firmware we can modify. I know we aren't actually (permanently) modifying any, but still we're subtly changing some, aren't we? And specifically I'm wondering wether it would be at all possible to completely skip the JPEG processing and saving routines.

It should be possible. You might have to copy a lot more ROM code into CHDK. However, you'd want to grab the exif data somehow. One thing I want to look into is writing proper CRW format images, rather than just a raw buffer dump. It appears that code to do this exists in the firmware (in the debug stuff I mentioned). That would get you the the information, and get rid of a bunch of hassle with tools having to update for every camera.

I also wonder if the imager can be told to add instead of subtract. If you can do that, you could do stacking right in memory, which would open up all kinds of neat stuff.

@PhyrePhoX
I basically looked for everywhere the dark frame control address was used, on the assumption there must be a period of time where the unsubtracted image is still in memory, and the subtracted one hasn't been taken yet. Actually I started out thinking that the firmware might do the actual subtraction (so I spent some time looking for a big loop involving the raw buffer address), but that wasn't very smart 

@jetzt
Yes that's where I'm going with this, but what I posted isn't all the way there. Currently, it gives to two raws from which you can produce a dark frame. You could do this on the camera, either automatically or via a "raw-subtract" but I haven't done that yet.

[LjL]

I'm also looking into CRW with metadata (and implemented it partially), but I don't think you need to get the EXIF data that the camera provides... I strongly suspect, as I've said in this thread, that you can reconstruct everything (and more, and better - such as giving the real shutter speed and aperture, as overridden by CHDK) with available functions.

Although the reason why I'm interested in skipping the JPEG processing is that it might speed up continuous shooting a lot - you'll say, what use if you aren't saving the shots? Well, just as you said, you could sum in memory, for a start, and maybe you could get fast burst shooting at low resolution by only making the camera save one last JPEG containing a mosaic of several (well, four 1600x1200) shots.

[reyalp]

Your right, you don't need the exif specifically, and recording the actual settings used would be a plus. However, being lazy, I like the idea of hijacking the cameras own code to write it for me. It's certainly possible to save all the same settings the camera does, it's just a matter of tracking them down.

Disabling jpeg should be fairly easy... we ought to be able to just jump past it after writing the raw. If you are lucky, you can just jump to the end of one of those functions that we've hooked, but there might be some essential housekeeping that has to happen after you take an image. Try it and see...

[ewavr]

This sounds ridiculous, but first CHDK versions wrote RAW twice -  before and after DF subtraction (first RAW was overwritted by second) - and without any capt_seq_task

[reyalp]

This sounds ridiculous, but first CHDK versions wrote RAW twice -  before and after DF subtraction (first RAW was overwritted by second) - and without any capt_seq_task

Hehe. Aside from capture twice, was it moved to solve some other problem ?

also, I have this great new invention for moving stuff around... You put some round things on a stick, and then you can just roll them instead of carrying !

[ewavr]

Hehe. Aside from capture twice, was it moved to solve some other problem ?

Yes, "old" RAW  on some cameras caused E16 (capture timeout?) error.

[reyalp]

Yes, "old" RAW  on some cameras caused E16 (capture timeout?) error.

Yeah, I actually expected my change to do something like that (or just crash). I was quite surprised that it worked at all. Do you have a link or svn rev for the old code ? I'd be interested to see where it is in relation to what I've done.

[ewavr]

Do you have a link or svn rev for the old code ?

As usually, http://tools.assembla.com/chdk - trunk before rev.167 or Grand's branch before rev.191 (without capt_seq.c files)

[LjL]

Disabling jpeg should be fairly easy... we ought to be able to just jump past it after writing the raw. If you are lucky, you can just jump to the end of one of those functions that we've hooked, but there might be some essential housekeeping that has to happen after you take an image. Try it and see...

I'd gladly try, but I think I need some kind of jump start...
I don't have IDA and I'd rather work on Linux, and right now I really just have no clue how the firmware is structured.

It would be nice if someone already into this things would spend some time improving the wiki's developers' documentation... I'm sure it's time well spent: people, like me, often just don't do things because they 1) see that few other people are doing them, so assume they're beyond themselves 2) don't quite know where to start.

Just see the topic about temperature sensors: the functions to use those have been there since forever, I think, for some models. And only those models. Now that someone suggested that they may be there in all cameras, everyone is jumping at IDA and actually finding them.

I don't think I personally need a step-by-step tutorial, as in "press this and then that", but I'd benefit from a technical explanation of how the firmware is layed out (and there is something like that, it's not bad but very limited), the difference between adding our own routines and modifying existing Canon routines (how do we actually do that, since we aren't modifying the firmware?), what I can expect to find in plain text in the firmware code, and what I should plan to reverse-engineer manually; and generally speaking, what sort of things I can expect to be able to change, and which others are set in stone (because they're not done in the ARM processor, or whatever reason).

Now I'll go back in a corner and stop hijacking this thread, but I thought I'd just throw a rock and see where it lands.

[reyalp]

Disabling jpeg should be fairly easy... we ought to be able to just jump past it after writing the raw. If you are lucky, you can just jump to the end of one of those functions that we've hooked, but there might be some essential housekeeping that has to happen after you take an image. Try it and see...

I'd gladly try, but I think I need some kind of jump start...
I don't have IDA and I'd rather work on Linux, and right now I really just have no clue how the firmware is structured.

If you look at capt_seq_task, it has a bunch of asm copied from ROM. When it's done doing it's special magic, it just jumps back into the ROM. It calls several functions made in the same way.

Somewhere in those routines is the part that saves the jpeg, and that will be somewhere after the raw hook is called.