Raw buffers in digic 6 - General Discussion and Assistance - CHDK Forum
supplierdeeply

Raw buffers in digic 6

  • 7 Replies
  • 5830 Views
*

Offline reyalp

  • ******
  • 14080
Raw buffers in digic 6
« on: 16 / April / 2017, 22:40:54 »
Advertisements
In trunk r4797 I changed the way g7x gets the active raw buffer. I suspect this applies to other digic 6 cameras (and maybe other recent cameras), so I'm posting a new thread.

Background
In the past, cameras had one or two raw buffers which were at fixed addresses and used in almost every shooting mode.

G7X uses up to 3 buffers, with the addresses and number varying depending on shooting mode, continuous shooting setting, and Canon raw setting.

Implementation
Instead of trying to trying to identify every permutation and reproduce the logic in hook_raw_image_addr, I decided to try to find where the Canon firmware stores the actual raw buffer it is using for a given shot.

I found a function sub_fc152772 (addresses refer to 100d but are mostly the same in other g7x subs) which I've called captseq_raw_addr_init

It's called in both the "quick shoot" and normal code paths, although it may be skipped in some cases. It appears to store the current raw address in struct pointed to by r1 (at offset 0x5c in the case of g7x)
The parameter in r0 is the value that ends up in the active_raw_buffer variable, generated by the preceding call to sub_fc152724. The actual raw address (along with some other values) appears to come from sub_fc184310

The struct in r1 is not at a fixed address, so cannot be accessed with a simple stubs_min define. In the normal shoot case, it's pointed to by the message struct, while in quick shoot it seems to come from somewhere else. In either case, it varies between calls.

To use this, I added a new function  captseq_raw_addr_init_my which replaces the call to captseq_raw_addr_init, and stores the current address in the global variable current_raw_addr. This in turn is used by hook_raw_image_addr

Testing a lot of different scene modes, continuous settings etc, the address obtained this seems to be correct.

This allows using CHDK raw with many mode / setting combinations where Canon raw is disabled, and allows rawop / raw histogram to work correctly.

Issues
From the disassembly, it appears there may be cases where captseq_raw_addr_init could be skipped. I didn't find any with extensive logging, but I certainly didn't try every permutation.

The raw address is only valid for a given shot, but is stored in a global where it could be used later. To avoid this, I added clear_current_raw_addr, which is called both at the start of pre-shoot, and after the raw hook. This isn't 100% effective, because there are currently some code paths that don't hit the raw hook. DNG reversing makes using an incorrect raw potentially very bad.

At the moment, if the raw address isn't valid, hook_raw_image_addr falls back to the old code, because CHDK assumes a raw address is always available. AFAIK benchmark is the only thing that assumes it can get a raw address outside of shooting. It might make more sense to use an arbitrary uncached address (e.g. MEMBASEADDR | CAM_UNCACHED_BIT) and make hook_raw_image_addr return NULL when valid raw isn't available.

The code in raw.c isn't set up deal with hook_raw_image_addr returning null either, so if captseq_raw_addr_init was skipped in some case, it would fail badly. It probably wouldn't be too hard to make it skip all the buffer stuff in this case though.

On previous ports with 2 raw buffers, it the second buffer was used to speed up DNG reversing. There is no obvious way to support this with the new method.
« Last Edit: 09 / January / 2020, 21:21:02 by reyalp »
Don't forget what the H stands for.

Re: Raw buffers in digic 6
« Reply #1 on: 17 / April / 2017, 11:05:44 »
I'll try to apply this to the Sx60hs port. I have wondered if the current raw address is available in the Edmac range somewhere....what are the actual addresses that you see on the g7x?

*

Offline reyalp

  • ******
  • 14080
Re: Raw buffers in digic 6
« Reply #2 on: 17 / April / 2017, 13:03:19 »
I'll try to apply this to the Sx60hs port. I have wondered if the current raw address is available in the Edmac range somewhere....what are the actual addresses that you see on the g7x?
If you mean raw buffer addresses, values I have seen are below, but this isn't necessarily comprehensive.

0x435ee300
0x43f12580
0x4427bfa0
0x453d6300
0x45cf6380
0x46063fa0
0x46f04300
0x471be300
0x47ada180

Note these appear in various tables in ROM (what I called raw_buffers_canon_raw and raw_buffers_jpeg in the original port)

The active_raw_buffer variable we have seems accurate, what's missing is the way to identify the correct table of addresses for a given mode. If there is a simple way to do this, it could be a better solution.

If you mean the structure that the address is stored in, it seems to be around 0x3exxx (Canon BSS, I think)
Don't forget what the H stands for.

Re: Raw buffers in digic 6
« Reply #3 on: 18 / April / 2017, 20:44:46 »
I implemented this for the SX60HS/100f, it seems to work without any problems.

I have the following addresses corresponding to those discussed above.
Code: [Select]
sx60hs/100f  g7x/100d
0xfc154bcc    0xfc152724
0xfc154c1a    0xfc152772
0xfc177cec    0xfc184310

The structure is the same with an offset of 0x5c (#92) for the raw_address ptr.


Re: Raw buffers in digic 6
« Reply #4 on: 18 / April / 2017, 20:55:44 »
Also
Quote
If you mean raw buffer addresses, values I have seen are below, but this isn't necessarily comprehensive.

0x435ee300
0x43f12580
0x4427bfa0
0x453d6300
0x45cf6380
0x46063fa0
0x46f04300
0x471be300
0x47ada180

Note these appear in various tables in ROM (what I called raw_buffers_canon_raw and raw_buffers_jpeg in the original port)
I think these must relate to this table of three addresses on sx60/100f:
Code: [Select]
on> rmem -i32 0xfc631bf4 16
0xfc631bf4 64
0xfc631bf4: 0x435ee300 0x44decf20 0x465ebb40 0x00000360
0xfc631c04: 0x00000288 0x00000000 0x00000000 0x000002ac
0xfc631c14: 0x00000200 0x0000005a 0x00000044 0x00000210
0xfc631c24: 0x0000018c 0x000000a8 0x0000007e 0x000001a8
which has at least one address in common with g7x

*

Offline reyalp

  • ******
  • 14080
Re: Raw buffers in digic 6
« Reply #5 on: 18 / April / 2017, 22:12:08 »
which has at least one address in common with g7x
Some of the addresses that aren't in common may not be raw buffers. On G7X, sub_fc184310 puts three uncached addresses in the buffer pointed to by R0. The first one is the raw buffer address, the others appear to be something else.

sub_fc152772 takes these values and puts them in the structure, at offsets 0x58, 0x5c and 0x60.

From my notes:
Code: [Select]
=call_func_ptr(0xfc184311,5167520,10,4) return string.format('0x%08x 0x%08x 0x%08x 0x%08x',peek(5167520),peek(5167520+4),peek(5167520+8),peek(5167520+12))
41:return:'0x4427bfa0 0x46063fa0 0x52640000 0xdeadbeef'
5167520 is memory I previously malloc'd and filled with 0xdeadbeef. The meaning of the following two parameters is unclear, but in canon code is small integers (first in the range 1-0x13 with not all values used, second 0-4) and influences the addresses returned.
Don't forget what the H stands for.

Re: Raw buffers in digic 6
« Reply #6 on: 23 / April / 2017, 12:08:50 »
On previous ports with 2 raw buffers, it the second buffer was used to speed up DNG reversing. There is no obvious way to support this with the new method.
The G7X seems to default to the "weak" versions of hook_alt_raw_image_addr() in generic/wrapper.c ?

Which should bypass the DNG reversal task trying to use double buffers for speed up as it returns the same buffer address as hook_raw_image_addr()

Code: [Select]
// for cameras with two (or more?) RAW buffers this can be used to speed up DNG creation by
// calling reverse_bytes_order only once. Override in platform/sub/lib.c
char __attribute__((weak)) *hook_alt_raw_image_addr() {
    return hook_raw_image_addr();
}


But if the returned value of hook_raw_image_addr() (i.e. current_raw_addr) has changed after the raw data was captured that would be bad I think.  And your new code zero's the current_raw_addr value out?

edit 2: commenting out the zeroing of current_raw_addr doesn't fix the crash

I'm getting crashing of the RevBytes task when I try to save DNG rather than RAW and I'm guessing that's the reason?  Did you modify something else I missed to prevent this from happening?

edit 1:  It's also possible I'm not getting a valid RAW buffer.  The structure offsets used in captseq_raw_addr_init() of the G16 are quite a bit different than those for the sx280, G7x, or SX60,  even though the code is almost identical. The G7X has a little bit of extra gymnastics going on mind you. (disassembly for all four cameras attached).
« Last Edit: 23 / April / 2017, 13:20:24 by waterwingz »
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline reyalp

  • ******
  • 14080
Re: Raw buffers in digic 6
« Reply #7 on: 23 / April / 2017, 15:57:54 »
The G7X seems to default to the "weak" versions of hook_alt_raw_image_addr() in generic/wrapper.c ?

Which should bypass the DNG reversal task trying to use double buffers for speed up as it returns the same buffer address as hook_raw_image_addr()
Yes. Since the logic for the "current" raw address is not understood, there is no reliable way to determine an "alternate" address. I mentioned this in the initial post.

If the camera changed the "current" address between when CHDK called hook_raw_image_addr() and  hook_alt_raw_image_addr(), that would be a problem, but that shouldn't be possible, since both are called inside the raw hook, and both the update and zeroing should only happen outside. It would probably be better to do this is a more robust way (i.e. make hook_alt_raw_image_addr return NULL when there's no alternate buffer, and update the raw / dng code accordingly)

edit:
Looking at your attached file, I agree it's odd the offsets in are so much smaller, while the rest of the function looks pretty similar.

A couple (possibly obvious) suggestions

Log the 3 values stored by your captseq_raw_addr_init without DNG enabled. See if they look like a raw address. If any of them do, try saving CHDK raw or dumping memory to see if any of them look like raw image data.

Try finding a raw buffer address by another method, like following the comments on g7x raw_buffers_canon_raw / raw_buffers_jpeg doing a full RAM dump and using binview. Once you have a valid raw address, check if it shows up in the captseq_raw_addr_init values.
« Last Edit: 24 / April / 2017, 02:33:17 by reyalp »
Don't forget what the H stands for.


 

Related Topics