Image data injection before JPEG generation for lcp profile generation

reyalp

12537

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #20 on: 27 / June / 2018, 13:32:20 »

Advertisements

Quote from: koshy on 27 / June / 2018, 07:09:10

In RawTherapee as they are already demosaiced the tools in the "Raw" tab will be disabled.

Right, that's what I meant by "if any of the corrections aren't exactly as you want". For example, RT allows you to do CA correction (there's another CA option outside the raw tab, but the results aren't quite as good), flat fielding and of course adjust demosaicing options in the raw tab. Some of the options in other tabs are also only available on raw images.

So again, it's certainly useful to have the option of going through linear DNG, but it's limiting in some ways.

It would still be a good argument for including the lens correction parameters in CHDK DNG if we can, obviously getting correction in some applications is better than none

Logged

Don't forget what the H stands for.

srsa_4c

4217

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #21 on: 27 / June / 2018, 14:06:33 »

Quote from: koshy on 27 / June / 2018, 12:52:50

Code: [Select]
exiftool -v4 FILENAME.RAW > list.txt

Thanks. I tried getting something similar but did not get to adding a number after -v.

Quote

Right... I'll do that tonight.

Well, now that I know that exiftool trick, I have something to start with - I won't be needing those processed images, so you don't have to spend time on that.
So far, it looks like VignettingCorrUnknown2 is not a straight copy from the ROM.

Logged

Menu jumping/flickering on M10?

koshy

1037

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #22 on: 27 / June / 2018, 14:51:36 »

Quote from: srsa_4c on 27 / June / 2018, 14:06:33

So far, it looks like VignettingCorrUnknown2 is not a straight copy from the ROM.

If it's the right item that seems obvious.
Field curvature correction depends on the current focal length and if there is an iris (which for the cams with native RAW there is) and such is used vignetting correction in addition depends on the aperture. With a couple of hundres zoom steps as reyalp pointed out and the number of aperture settings there would be far to many to just keep them in ROM to copy out.
They need to be generated and the really intriguing question on that end would be what Canon FW aspects control that generation. Why would they be any different for RAW than what goes into the native JPEGs? just saying that if it were possible to reverse the generation process where the output is known it might be possible to use the same FW aspects on the cameras without native RAW for the CHDK DNGs.
The reason to even ponder characterization based on image data was that I was willing to see that through for the whole lot of cameras that need it because the alternative (a in camera DNG solution) seemed lightyears away...

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

koshy

1037

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #23 on: 27 / June / 2018, 15:12:06 »

Quote from: reyalp on 27 / June / 2018, 13:32:20

So again, it's certainly useful to have the option of going through linear DNG, but it's limiting in some ways.

It would still be a good argument for including the lens correction parameters in CHDK DNG if we can, obviously getting correction in some applications is better than none

Now that is the way I see it, yeah. That glass certainly is half full rather than half empty. The basic situation also isn't a CHDK exclusive one. Anything that has an electronic viewfinder these days seems to go in that direction. Open source software might adapt eventually. And not everyone even cares about open source image editing software.

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

koshy

1037

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #24 on: 27 / June / 2018, 15:24:03 »

Quote from: reyalp on 27 / June / 2018, 13:32:20

For example, RT allows you to do CA correction (there's another CA option outside the raw tab, but the results aren't quite as good), flat fielding and of course adjust demosaicing options in the raw tab. Some of the options in other tabs are also only available on raw images.

I wrote another item but cut it out from the other response. Can't decide wether it's pointless rambling. There is a core concept that would have merit but I won't implemet it anytime soon.

-------------------

Also as soon as you start being creative about all this and have the skill, which all the gents here have... Say you really wanted the Raw Therapee stuff, I don't care (yet) personally but assuming I did...
The DNG converter interpolation item to Linear DNG does not do CA correction or anything. It does interpolation and it does (if Backwards DNG version is low enough as we have established) correct for field curvature and probably vignetting as far as denoted in the RAW file.
Now for the creative bit: In such a work flow I would initially do whatever I would want in Raw Thereapee (which I don't care for ;-)) and would write the result into a Tiff file. I would then create an automated tool that would take the base RAW image and make an intermediate linear DNG at 1.4 backwards compatibility settings. That one would be interpolated, but geometric distortion etc. would not be baked in but rather be denoted as DNG opcodes. Important.
My tool would then replace the linear interpolated image in the DNG by the contents of my Tiff from Raw Therapee with my desired CA and so forth. If that would need to be taken back to12 bit Gamma 1.0 encoded or not would be established. Once the data is back in the DNG - in whatever required form - my tool would harness DNG converter again. This time to make a linear DNG with DNG 1.1 backwards compatibility out of the already linear 1.4 based intermediate DNG. That would not interpolate again. It would however apply geometric distortion and vignetting to the data. I'd then retrieve that back into a Tiff and would re-encode to whatever tone reproduction curve it was before having put it into the DNG container. The only problem I see would be vignetting - in so far as that would depend on the data in the Tiff still being a linear response and not having been ruined by tone curves of whatever nature so there may be some restrictions to what I could do in Raw Therapy before using this tool and I'd ave to do ought else second step in Raw Therapee after the described processing. If I were to introduce such a second step I'd end with another linear DNG for RT to re-open rather than a Tiff. That kind creative combination of existing tools can do stuff today rather than in years. I try to avoid re-inventing wheels.

-------------------

This is really starting to be off topic, if it's just a curious aside that ought to be fine, otherwise it should probably be spliced into its own Lounge thread...

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

koshy

1037

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #25 on: 27 / June / 2018, 15:41:00 »

Quote from: blackhole on 27 / June / 2018, 09:28:17

Quote from: koshy on 27 / June / 2018, 08:23:09
Quote from: blackhole on 27 / June / 2018, 08:16:43
Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html
Yes, like I said no "CANON POWERSHOT G9 X MARK II" or anything recent.
The CANON POWERSHOT G9 X MARK II is supported, see here:
https://helpx.adobe.com/camera-raw/kb/camera-raw-plug-supported-cameras.html
The PowerShot G1 X Mark III is also supported from CR 10.1 onwards.

You're missing the distinction... The first item you posted was "supported lenses" the second is "supported cameras". The first item lists where Adobe produced individual *.lcp lens charaterizations, the second item lists what they support for debayering etc. Now, for any camera that has the correction of field curvature and or vignetting detailed on its RAW data (typically in the maker notes) Adobe does NOT supply *.lcp lens charaterizations as they would be redundant. That is why the CANON POWERSHOT G9 X MARK II is NOT on the first list you posted but of course is on the second. Hope that clears this up.

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

blackhole

763
A590IS 101b

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #26 on: 27 / June / 2018, 15:54:49 »

Quote from: koshy on 27 / June / 2018, 15:41:00

You're missing the distinction... The first item you posted was "supported lenses" the second is "supported cameras". The first item lists where Adobe produced individual *.lcp lens charaterizations, the second item lists what they support for debayering etc. Now, for any camera that has the correction of field curvature and or vignetting detailed on its RAW data (typically in the maker notes) Adobe does NOT supply *.lcp lens charaterizations as they would be redundant. That is why the CANON POWERSHOT G9 X MARK II is NOT on the first list you posted but of course is on the second. Hope that clears this up.

Yes, you're right, just dcp is included.

Logged

http://astrofoto.pondi.hr

srsa_4c

4217

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #27 on: 27 / June / 2018, 16:38:53 »

Quote from: koshy on 27 / June / 2018, 14:51:36

Quote from: srsa_4c on 27 / June / 2018, 14:06:33
So far, it looks like VignettingCorrUnknown2 is not a straight copy from the ROM.
If it's the right item that seems obvious.
Field curvature correction depends on the current focal length and if there is an iris (which for the cams with native RAW there is) and such is used vignetting correction in addition depends on the aperture. With a couple of hundres zoom steps as reyalp pointed out and the number of aperture settings there would be far to many to just keep them in ROM to copy out.
They need to be generated and the really intriguing question on that end would be what Canon FW aspects control that generation. Why would they be any different for RAW than what goes into the native JPEGs? just saying that if it were possible to reverse the generation process where the output is known it might be possible to use the same FW aspects on the cameras without native RAW for the CHDK DNGs.
The reason to even ponder characterization based on image data was that I was willing to see that through for the whole lot of cameras that need it because the alternative (a in camera DNG solution) seemed lightyears away...

I looked up the related routines in g7x 100d - it was easy to find as 0x4015 is not an often used constant.
The handler for tag 0x4015 is sub_FC49034C. The routine that creates the tag's content is sub_FC4F92D8. It's unfortunately rather complex - many small pieces of information is collected there.
I don't have a cam with this kind of 0x4015 tag (the EOS M10 does have a tag with the same ID but different content). Checked a JPEG only cam's fw, found no sign of this tag (not really a surprise).

edit:

Quote

Why would they be any different for RAW than what goes into the native JPEGs

The DIGIC vignetting/distortion correction unit likely takes the correction data in a different format.

« Last Edit: 27 / June / 2018, 16:46:34 by srsa_4c »

Logged

Menu jumping/flickering on M10?

koshy

1037

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #28 on: 27 / June / 2018, 17:47:48 »

Quick test to establish what to look at. Upon conversion to DNG I established that these models do contain Opcodes at all:

Code: [Select]

CANON POWERSHOT G15.CR2
CANON POWERSHOT G16.CR2
CANON POWERSHOT S90.CR2
CANON POWERSHOT S110.CR2
CANON POWERSHOT S120.CR2
CANON POWERSHOT SX50.CR2
CANON POWERSHOT SX60 HS.CR2
CANON POWERSHOT G1 X MARK II.CR2
CANON POWERSHOT G1 X MARK III.CR2
CANON POWERSHOT G3 X.CR2
CANON POWERSHOT G5 X.CR2
CANON POWERSHOT G7 X MARK II.CR2
CANON POWERSHOT G7 X.CR2
CANON POWERSHOT G9 X MARK II.CR2
CANON POWERSHOT G9 X.CR2

Attaching a text file with Exif dumped from those.
Looking for "VignettingCorr" we find

Code: [Select]

 Line 1212:   | | | 33) VignettingCorrUnknown2 (SubDirectory) -->
 Line 1278:   | | | | VignettingCorrVersion = 3
 Line 3068:   | | | 44) VignettingCorrUnknown2 (SubDirectory) -->
 Line 3132:   | | | | VignettingCorrVersion = 5
 Line 3135:   | | | 45) VignettingCorr2 (SubDirectory) -->
 Line 4610:   | | | 31) VignettingCorrUnknown1 (SubDirectory) -->
 Line 4643:   | | | | VignettingCorrVersion = 2
 Line 5986:   | | | 32) VignettingCorrUnknown2 (SubDirectory) -->
 Line 6052:   | | | | VignettingCorrVersion = 3
 Line 7555:   | | | 34) VignettingCorrUnknown2 (SubDirectory) -->
 Line 7622:   | | | | VignettingCorrVersion = 4
 Line 9134:   | | | 34) VignettingCorrUnknown2 (SubDirectory) -->
 Line 9201:   | | | | VignettingCorrVersion = 4
 Line 10980:   | | | 43) VignettingCorrUnknown2 (SubDirectory) -->
 Line 11044:   | | | | VignettingCorrVersion = 5
 Line 11047:   | | | 44) VignettingCorr2 (SubDirectory) -->
 Line 12554:   | | | 33) VignettingCorrUnknown2 (SubDirectory) -->
 Line 12621:   | | | | VignettingCorrVersion = 4
 Line 14392:   | | | 44) VignettingCorrUnknown2 (SubDirectory) -->
 Line 14456:   | | | | VignettingCorrVersion = 5
 Line 14459:   | | | 45) VignettingCorr2 (SubDirectory) -->
 Line 16098:   | | | 34) VignettingCorrUnknown2 (SubDirectory) -->
 Line 16165:   | | | | VignettingCorrVersion = 4
 Line 17513:   | | | 31) VignettingCorrUnknown1 (SubDirectory) -->
 Line 17546:   | | | | VignettingCorrVersion = 2
 Line 18889:   | | | 32) VignettingCorrUnknown2 (SubDirectory) -->
 Line 18955:   | | | | VignettingCorrVersion = 3
 Line 20067:   | | | 27) VignettingCorrUnknown1 (SubDirectory) -->
 Line 20075:   | | | | VignettingCorrVersion = 1
 Line 21404:   | | | 31) VignettingCorrUnknown1 (SubDirectory) -->
 Line 21437:   | | | | VignettingCorrVersion = 2
 Line 22782:   | | | 33) VignettingCorrUnknown2 (SubDirectory) -->
 Line 22849:   | | | | VignettingCorrVersion = 4

So that comes in versions 1 through 5.

Hunt_for_VignettingCorr.txt (1602.39 kB - downloaded 9 times.)

« Last Edit: 27 / June / 2018, 17:50:07 by koshy »

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

koshy

1037

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #29 on: 27 / June / 2018, 17:55:12 »

Quote from: srsa_4c on 27 / June / 2018, 16:38:53

The DIGIC vignetting/distortion correction unit likely takes the correction data in a different format.

Well maybe it takes the same data and puts it to different use.

Quote from: srsa_4c on 27 / June / 2018, 16:38:53

I looked up the related routines in g7x 100d - it was easy to find as 0x4015 is not an often used constant.
The handler for tag 0x4015 is sub_FC49034C. The routine that creates the tag's content is sub_FC4F92D8. It's unfortunately rather complex - many small pieces of information is collected there.

If so one or several of the "small pieces of information" used may be the needle in the haystack to point at where the DIGIC lens correction stuff is even pulled off and what common base of information there may be.

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)