Image data injection before JPEG generation for lcp profile generation - page 2 - General Discussion and Assistance - CHDK Forum supplierdeeply

Image data injection before JPEG generation for lcp profile generation

  • 30 Replies
  • 5695 Views
*

Online reyalp

  • ******
  • 12147
Advertisements
Great you're in for a surprise. That was not an addition. It was part of the original standard.
I meant "supported" in the sense of actually being useful in real world photography workflows. I don't know how well supported it is in practice, but given that DNG is primarily used as a raw format for CFA cameras, it wouldn't be surprising if it was less well tested. DNG is an extension of TIFF so theoretically allows a vast number of baroque variations, but that doesn't mean every DNG supporting program will actually do something reasonable with all of them. For example, we've encountered a number of programs that didn't support the packed 10/12 bit data CHDK produces, even though it's completely valid.

Anyway, as I noted, the adobe DNG converter does not appear to apply distortion correction to linear DNG data so this entire tangent is irrelevant.
Don't forget what the H stands for.

*

Offline koshy

  • *****
  • 946
Quote from: reyalp link=topic=13461.msg137541#msg137541

I meant "supported" in the sense of actually being useful in real world photography workflows.
I know it for a fact that it is more than useful, I'd label it a key item but all that takes too wide a swoop now.

Anyway, as I noted, the adobe DNG converter does not appear to apply distortion correction to linear DNG data so this entire tangent is irrelevant.

Sorry, the sentence about your test slipped me by. You must have gotten it wrong somewhere. Perhaps failed to realize that the lowest "Backwards version" plays a cruical part in what Adobe does there. I made you an example that shows it nicely and has enough to reverse it.

Get it from (for like 7 days):
https://wetransfer.com/downloads/ff785fa136b30e6090c645818e7db07220180626232231/6b90a16c66f71a4bcb7a228005257cbb20180626232232/e5ccf4

starting from "CANON POWERSHOT G9 X MARK II.CR2"
we get two DNGs:
"CANON POWERSHOT G9 X MARK II CFA @1.4.dng"
"CANON POWERSHOT G9 X MARK II LINEAR @1.1.dng"
Settings used should be obvious "Backwards version" and "Linear" or not. These conversions can be automated from the comand line (on Windows / Mac / VMs / not in Wine)

"CANON POWERSHOT G9 X MARK II CFA @1.4.dng" has Canon's embedded lens correction information converted to the DNG form.
"CANON POWERSHOT G9 X MARK II LINEAR @1.1.dng" has it readily applied.

Sending both through dcraw for demosaicing we get two Tiffs:
"CANON POWERSHOT G9 X MARK II CFA @1.4.tiff" is distorted as dcraw does not understand ANY lens correction information
"CANON POWERSHOT G9 X MARK II LINEAR @1.1.tiff" is corrected as DNG converter did apply the correction.

Hence linear DNGs can be used an intermediate to any raw conversion software that does not support embedded lens correction information, which was my point. This applies to everything micro 4/3 for example. Didn't expect that to cause any confusion. Same story with CHDK Badpixel information (about the possible use of the linear DNG intermediate to have it all baked in for whereever).

Maybe worth pointing out again that the goal is the same adjustment as the camera used for a JPEG. Of course if the camera embedds this information in its proprietary RAW image things are much easier than for us with CHDK but even that might eventually work out.
« Last Edit: 26 / June / 2018, 19:44:21 by koshy »

*

Online reyalp

  • ******
  • 12147
Sorry, the sentence about your test slipped me by. You must have gotten it wrong somewhere.
Thanks, you're right, I thought I had 1.1 selected, but re-running the program it was set to 1.4. Converting with 1.1 using the latest DNG converter did apply distortion correction.

Raw therepee 5.4 displays the resulting DNG as all black though, so I'm not entirely convinced on the compatibility front. Weirdly, it correctly displayed files converted to linear with an earlier version of the adobe converter.  :-[

IMO there are significant disadvantages to go going through a debayered "raw". Aside from time and storage, if any of the corrections aren't exactly as you want, you lose quality.

It's good to have the option, but a standalone profile would probably be preferable if available.
Quote
Of course if the camera embedds this information in its proprietary RAW image things are much easier than for us with CHDK but even that might eventually work out.
I could be wrong, but my impression is that only the lens identification is stored on Canon raw files, and the correction data is packaged in their software. That would explain why both Canon and 3rd parties issue software updates for every new camera. The formats also originated on ILC cameras where the lens characteristics might not be available to the firmware.

The CRW and CR2 formats have been reverse engineered quite a bit, e.g.  http://lclevy.free.fr/cr2/ https://sno.phy.queensu.ca/~phil/exiftool/canon_raw.html
Don't forget what the H stands for.

*

Offline koshy

  • *****
  • 946
IMO there are significant disadvantages to go going through a debayered "raw". Aside from time and storage, if any of the corrections aren't exactly as you want, you lose quality.
I said intermediate didn't I... This needs to be more verbose. Do not delete originals, do not store these files. They'd be written to a temporary folder in the course of opening the image. As for "if any of the corrections aren't exactly as you want" none of them allow for any user inetraction in Camera RAW for example. The stuff that gets applied at this stage is under the hood stuff as far as Adobe is concerned. In RawTherapee as they are already demosaiced the tools in the "Raw" tab will be disabled.

I could be wrong, but my impression is that only the lens identification is stored on Canon raw files, and the correction data is packaged in their software. That would explain why both Canon and 3rd parties issue software updates for every new camera. The formats also originated on ILC cameras where the lens characteristics might not be available to the firmware.
I'll read up on it in case of Canon but I note there are zero newer "Powershot" lcp files shipped by Adobe.
They are all here on Win: C:\ProgramData\Adobe\CameraRaw\LensProfiles\1.0 (Library on Mac)

The distinction is this: What is applied in camera is expected to be denoted in RAW format and will be applied without asking the user and without providing an opt-out. In newer versions of Camera RAW / Lightroom there will be a message about it (see screen grab) in older "silence". Anything that is not also applied in camera - as in DSLRs - comes as an optional lcp file.


*

Offline blackhole

  • *****
  • 684
  • A590IS 101b
    • Planetary astrophotography
I could be wrong, but my impression is that only the lens identification is stored on Canon raw files, and the correction data is packaged in their software. That would explain why both Canon and 3rd parties issue software updates for every new camera. The formats also originated on ILC cameras where the lens characteristics might not be available to the firmware.
That's right.

I'll read up on it in case of Canon but I note there are zero newer "Powershot" lcp files shipped by Adobe.
They are all here on Win: C:\ProgramData\Adobe\CameraRaw\LensProfiles\1.0 (Library on Mac)

The distinction is this: What is applied in camera is expected to be denoted in RAW format and will be applied without asking the user and without providing an opt-out. In newer versions of Camera RAW / Lightroom there will be a message about it (see screen grab) in older "silence". Anything that is not also applied in camera - as in DSLRs - comes as an optional lcp file.
Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html

*

Offline koshy

  • *****
  • 946
Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html
Yes, like I said no "CANON POWERSHOT G9 X MARK II" or anything recent.

*

Offline koshy

  • *****
  • 946
If we remove the information from the DNG:
Code: [Select]
exiftool.exe -OpcodeList3= -OpcodeList2= -OpcodeList1=  "CANON POWERSHOT G9 X MARK II CFA @1.4.dng"Lightroom, CameraRAW and DNG converter (upon conversion to Linear DNG) will no longer be able to use it.
=> Image stays distorted.

ADDITION:
I expect the opcodes to stem from the Canon Maker Notes. Sadly the pragmatic proof of using:
Code: [Select]
exiftool.exe -MakerNotes= "CANON POWERSHOT G9 X MARK II.CR2"Failed as DNG Converter won't read the CR2 thereafter.

The actual suspect stays untouched individually:
Code: [Select]
exiftool.exe  -VignettingCorrUnknown2= "CANON POWERSHOT G9 X MARK II.CR2"
Warning: Sorry, VignettingCorrUnknown2 is not writable
Nothing to do.

Code: [Select]
VignettingCorrUnknown2 (SubDirectory) -->
  | | | |     - Tag 0x4015 (968 bytes, undef[968]):
  | | | |         c7f4: 05 80 00 00 c8 03 00 00 00 00 00 00 02 00 00 00 [................]
  | | | |         c804: 01 00 00 00 00 00 00 00 00 00 00 00 0d 27 00 00 [.............'..]
  | | | |         c814: 10 27 00 00 00 00 00 00 60 09 00 00 60 15 00 00 [.'......`...`...]
  | | | |         c824: 40 0e 00 00 07 00 00 00 09 00 00 00 00 00 00 00 [@...............]
  | | | |         c834: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 [................]
  | | | |         c844: 00 04 00 00 80 00 00 00 01 00 00 00 06 00 00 00 [................]
  | | | |         c854: 07 00 00 00 09 00 00 00 11 00 00 00 00 00 00 00 [................]
  | | | |         c864: 10 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 [................]
  | | | |         c874: d8 0c 00 00 07 00 00 00 b0 19 00 00 3b 00 00 00 [............;...]
  | | | |         c884: 88 26 00 00 c9 00 00 00 61 33 00 00 e2 01 00 00 [.&......a3......]
  | | | |         c894: 39 40 00 00 be 03 00 00 11 4d 00 00 95 06 00 00 [9@.......M......]
  | | | |         c8a4: e9 59 00 00 a9 0a 00 00 c2 66 00 00 3c 10 00 00 [.Y.......f..<...]
  | | | |         c8b4: 9a 73 00 00 95 17 00 00 72 80 00 00 e5 20 00 00 [.s......r.... ..]
  | | | |         c8c4: 4a 8d 00 00 76 2c 00 00 23 9a 00 00 8e 3a 00 00 [J...v,..#....:..]
  | | | |         c8d4: fb a6 00 00 67 4b 00 00 d3 b3 00 00 30 5f 00 00 [....gK......0_..]
  | | | |         c8e4: ab c0 00 00 0c 76 00 00 84 cd 00 00 18 90 00 00 [.....v..........]
  | | | |         c8f4: 11 00 00 00 00 00 00 00 10 00 00 00 00 01 00 00 [................]
  | | | |         c904: 00 00 00 00 00 00 00 00 d8 0c 00 00 07 00 00 00 [................]
  | | | |         c914: b0 19 00 00 3b 00 00 00 88 26 00 00 ca 00 00 00 [....;....&......]
  | | | |         c924: 61 33 00 00 e6 01 00 00 39 40 00 00 c5 03 00 00 [a3......9@......]
  | | | |         c934: 11 4d 00 00 a3 06 00 00 e9 59 00 00 be 0a 00 00 [.M.......Y......]
  | | | |         c944: c2 66 00 00 5d 10 00 00 9a 73 00 00 c3 17 00 00 [.f..]....s......]
  | | | |         c954: 72 80 00 00 25 21 00 00 4a 8d 00 00 cb 2c 00 00 [r...%!..J....,..]
  | | | |         c964: 23 9a 00 00 fd 3a 00 00 fb a6 00 00 f3 4b 00 00 [#....:.......K..]
  | | | |         c974: d3 b3 00 00 de 5f 00 00 ab c0 00 00 e1 76 00 00 [....._.......v..]
  | | | |         c984: 84 cd 00 00 17 91 00 00 11 00 00 00 10 00 00 00 [................]
  | | | |         c994: 80 00 00 00 00 00 00 00 00 00 00 00 d8 0c 00 00 [................]
  | | | |         c9a4: 0b 00 00 00 b0 19 00 00 15 00 00 00 88 26 00 00 [.............&..]
  | | | |         c9b4: 1e 00 00 00 61 33 00 00 28 00 00 00 39 40 00 00 [....a3..(...9@..]
  | | | |         c9c4: 2c 00 00 00 11 4d 00 00 32 00 00 00 e9 59 00 00 [,....M..2....Y..]
  | | | |         c9d4: 36 00 00 00 c2 66 00 00 3f 00 00 00 9a 73 00 00 [6....f..?....s..]
  | | | |         c9e4: 3e 00 00 00 72 80 00 00 4a 00 00 00 4a 8d 00 00 [>...r...J...J...]
  | | | |         c9f4: 4a 00 00 00 23 9a 00 00 54 00 00 00 fb a6 00 00 [J...#...T.......]
  | | | |         ca04: 54 00 00 00 d3 b3 00 00 5f 00 00 00 ab c0 00 00 [T......._.......]
  | | | |         ca14: 67 00 00 00 84 cd 00 00 6a 00 00 00 10 00 00 00 [g.......j.......]
  | | | |         ca24: 80 00 00 00 00 00 00 00 00 00 00 00 d8 0c 00 00 [................]
  | | | |         ca34: 0b 00 00 00 b0 19 00 00 15 00 00 00 88 26 00 00 [.............&..]
  | | | |         ca44: 27 00 00 00 61 33 00 00 35 00 00 00 39 40 00 00 ['...a3..5...9@..]
  | | | |         ca54: 47 00 00 00 11 4d 00 00 51 00 00 00 e9 59 00 00 [G....M..Q....Y..]
  | | | |         ca64: 5d 00 00 00 c2 66 00 00 68 00 00 00 9a 73 00 00 []....f..h....s..]
  | | | |         ca74: 73 00 00 00 72 80 00 00 7d 00 00 00 4a 8d 00 00 [s...r...}...J...]
  | | | |         ca84: 7e 00 00 00 23 9a 00 00 85 00 00 00 fb a6 00 00 [~...#...........]
  | | | |         ca94: 85 00 00 00 d3 b3 00 00 86 00 00 00 ab c0 00 00 [................]
  | | | |         caa4: 7f 00 00 00 84 cd 00 00 7e 00 00 00 0f 00 00 00 [........~.......]
  | | | |         cab4: 00 20 00 00 00 00 00 00 00 20 00 00 07 01 00 00 [. ....... ......]
  | | | |         cac4: f6 1f 00 00 0e 02 00 00 d6 1f 00 00 15 03 00 00 [................]
  | | | |         cad4: b1 1f 00 00 1c 04 00 00 7e 1f 00 00 23 05 00 00 [........~...#...]
  | | | |         cae4: 23 1f 00 00 2a 06 00 00 88 1e 00 00 31 07 00 00 [#...*.......1...]
  | | | |         caf4: a6 1d 00 00 38 08 00 00 85 1c 00 00 3f 09 00 00 [....8.......?...]
  | | | |         cb04: 3f 1b 00 00 46 0a 00 00 ef 19 00 00 4d 0b 00 00 [?...F.......M...]
  | | | |         cb14: a0 18 00 00 54 0c 00 00 36 17 00 00 5b 0d 00 00 [....T...6...[...]
  | | | |         cb24: 42 15 00 00 41 0e 00 00 6e 12 00 00 21 00 00 00 [B...A...n...!...]
  | | | |         cb34: 80 00 08 00 94 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb44: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb54: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb64: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb74: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb84: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cb94: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cba4: a0 00 08 00 a0 00 08 00 a0 00 08 00 a0 00 08 00 [................]
  | | | |         cbb4: a0 00 08 00 00 00 00 00                         [........]
  | | | | + [BinaryData directory, 968 bytes]
  | | | | | VignettingCorrVersion = 5
  | | | | | - Tag 0x0000 (1 bytes, int8u[1]):
  | | | | |     c7f4: 05                                              [.]
  | | | | 45) VignettingCorr2 (SubDirectory) -->
  | | | |     - Tag 0x4016 (40 bytes, int32u[10] read as undef[40]):
  | | | |         cbbc: 28 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 [(...............]
  | | | |         cbcc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
  | | | |         cbdc: 01 00 00 00 00 00 00 00                         [........]
« Last Edit: 27 / June / 2018, 15:36:18 by koshy »

*

Offline blackhole

  • *****
  • 684
  • A590IS 101b
    • Planetary astrophotography
Here you can see what has been added with the last update, and what ever was supported in Adobe products.
https://helpx.adobe.com/camera-raw/kb/supported-lenses.html
Yes, like I said no "CANON POWERSHOT G9 X MARK II" or anything recent.
The CANON POWERSHOT G9 X MARK II is supported, see here:
https://helpx.adobe.com/camera-raw/kb/camera-raw-plug-supported-cameras.html
The PowerShot G1 X Mark III is also supported from CR 10.1 onwards.


*

Offline srsa_4c

  • ******
  • 4023
Code: [Select]
VignettingCorrUnknown2 (SubDirectory) -->
  | | | |     - Tag 0x4015 (968 bytes, undef[968]):
...
How are you getting this kind of listing?
Perhaps trying all this on a model that we have a fw dump from would make more sense.

*

Offline koshy

  • *****
  • 946
How are you getting this kind of listing?
Code: [Select]
exiftool -v4 FILENAME.RAW > list.txt
Perhaps trying all this on a model that we have a fw dump from would make more sense.
Right... I'll do that tonight.

 

Related Topics