Image data injection before JPEG generation for lcp profile generation

reyalp

14080

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #10 on: 26 / June / 2018, 18:11:12 »

Advertisements

Quote from: koshy on 26 / June / 2018, 17:01:25

Great you're in for a surprise. That was not an addition. It was part of the original standard.

I meant "supported" in the sense of actually being useful in real world photography workflows. I don't know how well supported it is in practice, but given that DNG is primarily used as a raw format for CFA cameras, it wouldn't be surprising if it was less well tested. DNG is an extension of TIFF so theoretically allows a vast number of baroque variations, but that doesn't mean every DNG supporting program will actually do something reasonable with all of them. For example, we've encountered a number of programs that didn't support the packed 10/12 bit data CHDK produces, even though it's completely valid.

Anyway, as I noted, the adobe DNG converter does not appear to apply distortion correction to linear DNG data so this entire tangent is irrelevant.

Logged

Don't forget what the H stands for.

koshy

1096

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #11 on: 26 / June / 2018, 19:41:41 »

Quote from: reyalp link=topic=13461.msg137541#msg137541

I meant "supported" in the sense of actually being useful in real world photography workflows.

I know it for a fact that it is more than useful, I'd label it a key item but all that takes too wide a swoop now.

Quote from: reyalp on 26 / June / 2018, 18:11:12

Anyway, as I noted, the adobe DNG converter does not appear to apply distortion correction to linear DNG data so this entire tangent is irrelevant.

Sorry, the sentence about your test slipped me by. You must have gotten it wrong somewhere. Perhaps failed to realize that the lowest "Backwards version" plays a cruical part in what Adobe does there. I made you an example that shows it nicely and has enough to reverse it.

Get it from (for like 7 days):
https://wetransfer.com/downloads/ff785fa136b30e6090c645818e7db07220180626232231/6b90a16c66f71a4bcb7a228005257cbb20180626232232/e5ccf4

starting from "CANON POWERSHOT G9 X MARK II.CR2"
we get two DNGs:
"CANON POWERSHOT G9 X MARK II CFA @1.4.dng"
"CANON POWERSHOT G9 X MARK II LINEAR @1.1.dng"
Settings used should be obvious "Backwards version" and "Linear" or not. These conversions can be automated from the comand line (on Windows / Mac / VMs / not in Wine)

"CANON POWERSHOT G9 X MARK II CFA @1.4.dng" has Canon's embedded lens correction information converted to the DNG form.
"CANON POWERSHOT G9 X MARK II LINEAR @1.1.dng" has it readily applied.

Sending both through dcraw for demosaicing we get two Tiffs:
"CANON POWERSHOT G9 X MARK II CFA @1.4.tiff" is distorted as dcraw does not understand ANY lens correction information
"CANON POWERSHOT G9 X MARK II LINEAR @1.1.tiff" is corrected as DNG converter did apply the correction.

Hence linear DNGs can be used an intermediate to any raw conversion software that does not support embedded lens correction information, which was my point. This applies to everything micro 4/3 for example. Didn't expect that to cause any confusion. Same story with CHDK Badpixel information (about the possible use of the linear DNG intermediate to have it all baked in for whereever).

Maybe worth pointing out again that the goal is the same adjustment as the camera used for a JPEG. Of course if the camera embedds this information in its proprietary RAW image things are much easier than for us with CHDK but even that might eventually work out.

« Last Edit: 26 / June / 2018, 19:44:21 by koshy »

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

reyalp

14080

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #12 on: 26 / June / 2018, 23:50:39 »

Quote from: koshy on 26 / June / 2018, 19:41:41

Sorry, the sentence about your test slipped me by. You must have gotten it wrong somewhere.

Thanks, you're right, I thought I had 1.1 selected, but re-running the program it was set to 1.4. Converting with 1.1 using the latest DNG converter did apply distortion correction.

Raw therepee 5.4 displays the resulting DNG as all black though, so I'm not entirely convinced on the compatibility front. Weirdly, it correctly displayed files converted to linear with an earlier version of the adobe converter.

IMO there are significant disadvantages to go going through a debayered "raw". Aside from time and storage, if any of the corrections aren't exactly as you want, you lose quality.

It's good to have the option, but a standalone profile would probably be preferable if available.

Quote

Of course if the camera embedds this information in its proprietary RAW image things are much easier than for us with CHDK but even that might eventually work out.

I could be wrong, but my impression is that only the lens identification is stored on Canon raw files, and the correction data is packaged in their software. That would explain why both Canon and 3rd parties issue software updates for every new camera. The formats also originated on ILC cameras where the lens characteristics might not be available to the firmware.

The CRW and CR2 formats have been reverse engineered quite a bit, e.g. http://lclevy.free.fr/cr2/ https://sno.phy.queensu.ca/~phil/exiftool/canon_raw.html

Logged

Don't forget what the H stands for.

koshy

1096

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #13 on: 27 / June / 2018, 07:09:10 »

Quote from: reyalp on 26 / June / 2018, 23:50:39

IMO there are significant disadvantages to go going through a debayered "raw". Aside from time and storage, if any of the corrections aren't exactly as you want, you lose quality.

I said intermediate didn't I... This needs to be more verbose. Do not delete originals, do not store these files. They'd be written to a temporary folder in the course of opening the image. As for "if any of the corrections aren't exactly as you want" none of them allow for any user inetraction in Camera RAW for example. The stuff that gets applied at this stage is under the hood stuff as far as Adobe is concerned. In RawTherapee as they are already demosaiced the tools in the "Raw" tab will be disabled.

Quote from: reyalp on 26 / June / 2018, 23:50:39

I could be wrong, but my impression is that only the lens identification is stored on Canon raw files, and the correction data is packaged in their software. That would explain why both Canon and 3rd parties issue software updates for every new camera. The formats also originated on ILC cameras where the lens characteristics might not be available to the firmware.

I'll read up on it in case of Canon but I note there are zero newer "Powershot" lcp files shipped by Adobe.
They are all here on Win: C:\ProgramData\Adobe\CameraRaw\LensProfiles\1.0 (Library on Mac)

The distinction is this: What is applied in camera is expected to be denoted in RAW format and will be applied without asking the user and without providing an opt-out. In newer versions of Camera RAW / Lightroom there will be a message about it (see screen grab) in older "silence". Anything that is not also applied in camera - as in DSLRs - comes as an optional lcp file.

LR_ACR_Profile.png (154.5 kB, 1116x1108 - viewed 44 times.)

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

blackhole

937
A590IS 101b

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #14 on: 27 / June / 2018, 08:16:43 »

Quote from: reyalp on 26 / June / 2018, 23:50:39

I could be wrong, but my impression is that only the lens identification is stored on Canon raw files, and the correction data is packaged in their software. That would explain why both Canon and 3rd parties issue software updates for every new camera. The formats also originated on ILC cameras where the lens characteristics might not be available to the firmware.

That's right.

Quote from: koshy on 27 / June / 2018, 07:09:10

I'll read up on it in case of Canon but I note there are zero newer "Powershot" lcp files shipped by Adobe.
They are all here on Win: C:\ProgramData\Adobe\CameraRaw\LensProfiles\1.0 (Library on Mac)

The distinction is this: What is applied in camera is expected to be denoted in RAW format and will be applied without asking the user and without providing an opt-out. In newer versions of Camera RAW / Lightroom there will be a message about it (see screen grab) in older "silence". Anything that is not also applied in camera - as in DSLRs - comes as an optional lcp file.

Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html

Logged

http://astrofoto.pondi.hr

koshy

1096

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #15 on: 27 / June / 2018, 08:23:09 »

Quote from: blackhole on 27 / June / 2018, 08:16:43

Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html

Yes, like I said no "CANON POWERSHOT G9 X MARK II" or anything recent.

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

koshy

1096

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #16 on: 27 / June / 2018, 08:33:19 »

If we remove the information from the DNG:

Code: [Select]

exiftool.exe -OpcodeList3= -OpcodeList2= -OpcodeList1= "CANON POWERSHOT G9 X MARK II CFA @1.4.dng"Lightroom, CameraRAW and DNG converter (upon conversion to Linear DNG) will no longer be able to use it.
=> Image stays distorted.

ADDITION:
I expect the opcodes to stem from the Canon Maker Notes. Sadly the pragmatic proof of using:

Code: [Select]

exiftool.exe -MakerNotes= "CANON POWERSHOT G9 X MARK II.CR2"Failed as DNG Converter won't read the CR2 thereafter.

The actual suspect stays untouched individually:

Code: [Select]

exiftool.exe  -VignettingCorrUnknown2= "CANON POWERSHOT G9 X MARK II.CR2"
Warning: Sorry, VignettingCorrUnknown2 is not writable
Nothing to do.

Code: [Select]

VignettingCorrUnknown2 (SubDirectory) -->
  | | | |     - Tag 0x4015 (968 bytes, undef[968]):
  | | | |         c7f4: 05 80 00 00 c8 03 00 00 00 00 00 00 02 00 00 00 [................]
  | | | |         c804: 01 00 00 00 00 00 00 00 00 00 00 00 0d 27 00 00 [.............'..]
  | | | |         c814: 10 27 00 00 00 00 00 00 60 09 00 00 60 15 00 00 [.'......`...`...]
  | | | |         c824: 40 0e 00 00 07 00 00 00 09 00 00 00 00 00 00 00 [@...............]
  | | | |         c834: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 [................]
  | | | |         c844: 00 04 00 00 80 00 00 00 01 00 00 00 06 00 00 00 [................]
  | | | |         c854: 07 00 00 00 09 00 00 00 11 00 00 00 00 00 00 00 [................]
  | | | |         c864: 10 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 [................]
  | | | |         c874: d8 0c 00 00 07 00 00 00 b0 19 00 00 3b 00 00 00 [............;...]
  | | | |         c884: 88 26 00 00 c9 00 00 00 61 33 00 00 e2 01 00 00 [.&......a3......]
  | | | |         c894: 39 40 00 00 be 03 00 00 11 4d 00 00 95 06 00 00 [9@.......M......]
  | | | |         c8a4: e9 59 00 00 a9 0a 00 00 c2 66 00 00 3c 10 00 00 [.Y.......f..<...]
  | | | |         c8b4: 9a 73 00 00 95 17 00 00 72 80 00 00 e5 20 00 00 [.s......r.... ..]
  | | | |         c8c4: 4a 8d 00 00 76 2c 00 00 23 9a 00 00 8e 3a 00 00 [J...v,..#....:..]
  | | | |         c8d4: fb a6 00 00 67 4b 00 00 d3 b3 00 00 30 5f 00 00 [....gK......0_..]
  | | | |         c8e4: ab c0 00 00 0c 76 00 00 84 cd 00 00 18 90 00 00 [.....v..........]
  | | | |         c8f4: 11 00 00 00 00 00 00 00 10 00 00 00 00 01 00 00 [................]
  | | | |         c904: 00 00 00 00 00 00 00 00 d8 0c 00 00 07 00 00 00 [................]
  | | | |         c914: b0 19 00 00 3b 00 00 00 88 26 00 00 ca 00 00 00 [....;....&......]
  | | | |         c924: 61 33 00 00 e6 01 00 00 39 40 00 00 c5 03 00 00 [a3......9@......]
  | | | |         c934: 11 4d 00 00 a3 06 00 00 e9 59 00 00 be 0a 00 00 [.M.......Y......]
  | | | |         c944: c2 66 00 00 5d 10 00 00 9a 73 00 00 c3 17 00 00 [.f..]....s......]
  | | | |         c954: 72 80 00 00 25 21 00 00 4a 8d 00 00 cb 2c 00 00 [r...%!..J....,..]
  | | | |         c964: 23 9a 00 00 fd 3a 00 00 fb a6 00 00 f3 4b 00 00 [#....:.......K..]
  | | | |         c974: d3 b3 00 00 de 5f 00 00 ab c0 00 00 e1 76 00 00 [....._.......v..]
  | | | |         c984: 84 cd 00 00 17 91 00 00 11 00 00 00 10 00 00 00 [................]
  | | | |         c994: 80 00 00 00 00 00 00 00 00 00 00 00 d8 0c 00 00 [................]
  | | | |         c9a4: 0b 00 00 00 b0 19 00 00 15 00 00 00 88 26 00 00 [.............&..]
  | | | |         c9b4: 1e 00 00 00 61 33 00 00 28 00 00 00 39 40 00 00 [....a3..(...9@..]
  | | | |         c9c4: 2c 00 00 00 11 4d 00 00 32 00 00 00 e9 59 00 00 [,....M..2....Y..]
  | | | |         c9d4: 36 00 00 00 c2 66 00 00 3f 00 00 00 9a 73 00 00 [6....f..?....s..]
  | | | |         c9e4: 3e 00 00 00 72 80 00 00 4a 00 00 00 4a 8d 00 00 [>...r...J...J...]
  | | | |         c9f4: 4a 00 00 00 23 9a 00 00 54 00 00 00 fb a6 00 00 [J...#...T.......]
  | | | |         ca04: 54 00 00 00 d3 b3 00 00 5f 00 00 00 ab c0 00 00 [T......._.......]
  | | | |         ca14: 67 00 00 00 84 cd 00 00 6a 00 00 00 10 00 00 00 [g.......j.......]
  | | | |         ca24: 80 00 00 00 00 00 00 00 00 00 00 00 d8 0c 00 00 [................]
  | | | |         ca34: 0b 00 00 00 b0 19 00 00 15 00 00 00 88 26 00 00 [.............&..]
  | | | |         ca44: 27 00 00 00 61 33 00 00 35 00 00 00 39 40 00 00 ['...a3..5...9@..]
  | | | |         ca54: 47 00 00 00 11 4d 00 00 51 00 00 00 e9 59 00 00 [G....M..Q....Y..]
  | | | |         ca64: 5d 00 00 00 c2 66 00 00 68 00 00 00 9a 73 00 00 []....f..h....s..]
  | | | |         ca74: 73 00 00 00 72 80 00 00 7d 00 00 00 4a 8d 00 00 [s...r...}...J...]
  | | | |         ca84: 7e 00 00 00 23 9a 00 00 85 00 00 00 fb a6 00 00 [~...#...........]
  | | | |         ca94: 85 00 00 00 d3 b3 00 00 86 00 00 00 ab c0 00 00 [................]
  | | | |         caa4: 7f 00 00 00 84 cd 00 00 7e 00 00 00 0f 00 00 00 [........~.......]
  | | | |         cab4: 00 20 00 00 00 00 00 00 00 20 00 00 07 01 00 00 [. ....... ......]
  | | | |         cac4: f6 1f 00 00 0e 02 00 00 d6 1f 00 00 15 03 00 00 [................]
  | | | |         cad4: b1 1f 00 00 1c 04 00 00 7e 1f 00 00 23 05 00 00 [........~...#...]
  | | | |         cae4: 23 1f 00 00 2a 06 00 00 88 1e 00 00 31 07 00 00 [#...*.......1...]
  | | | |         caf4: a6 1d 00 00 38 08 00 00 85 1c 00 00 3f 09 00 00 [....8.......?...]
  | | | |         cb04: 3f 1b 00 00 46 0a 00 00 ef 19 00 00 4d 0b 00 00 [?...F.......M...]
  | | | |         cb14: a0 18 00 00 54 0c 00 00 36 17 00 00 5b 0d 00 00 [....T...6...[...]
  | | | |         cb24: 42 15 00 00 41 0e 00 00 6e 12 00 00 21 00 00 00 [B...A...n...!...]
  | | | |         cb34: 80 00 08 00 94 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb44: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb54: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb64: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb74: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb84: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb94: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cba4: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cbb4: a0 00 08 00 00 00 00 00                         [........]
  | | | | + [BinaryData directory, 968 bytes]
  | | | | | VignettingCorrVersion = 5
  | | | | | - Tag 0x0000 (1 bytes, int8u[1]):
  | | | | |     c7f4: 05                                              [.]
  | | | | 45) VignettingCorr2 (SubDirectory) -->
  | | | |     - Tag 0x4016 (40 bytes, int32u[10] read as undef[40]):
  | | | |         cbbc: 28 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 [(...............]
  | | | |         cbcc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
  | | | |         cbdc: 01 00 00 00 00 00 00 00                         [........]

« Last Edit: 27 / June / 2018, 15:36:18 by koshy »

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

blackhole

937
A590IS 101b

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #17 on: 27 / June / 2018, 09:28:17 »

Quote from: koshy on 27 / June / 2018, 08:23:09

Quote from: blackhole on 27 / June / 2018, 08:16:43
Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html
Yes, like I said no "CANON POWERSHOT G9 X MARK II" or anything recent.

The CANON POWERSHOT G9 X MARK II is supported, see here:
https://helpx.adobe.com/camera-raw/kb/camera-raw-plug-supported-cameras.html
The PowerShot G1 X Mark III is also supported from CR 10.1 onwards.

Logged

http://astrofoto.pondi.hr

srsa_4c

4451

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #18 on: 27 / June / 2018, 12:35:05 »

Quote from: koshy on 27 / June / 2018, 08:33:19

Code: [Select]
VignettingCorrUnknown2 (SubDirectory) --> | | | | - Tag 0x4015 (968 bytes, undef[968]):...

How are you getting this kind of listing?
Perhaps trying all this on a model that we have a fw dump from would make more sense.

Logged

koshy

1096

Re: Image data injection before JPEG generation for lcp profile generation

« Reply #19 on: 27 / June / 2018, 12:52:50 »

Quote from: srsa_4c on 27 / June / 2018, 12:35:05

How are you getting this kind of listing?

Code: [Select]

exiftool -v4 FILENAME.RAW > list.txt

Quote from: srsa_4c on 27 / June / 2018, 12:35:05

Perhaps trying all this on a model that we have a fw dump from would make more sense.

Right... I'll do that tonight.

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)