Image data injection before JPEG generation for lcp profile generation - General Discussion and Assistance - CHDK Forum supplierdeeply
« previous next »
Pages: [1] 2 3 4   Go Down

Image data injection before JPEG generation for lcp profile generation

  • 30 Replies
  • 17738 Views
*

Offline koshy

  • *****
  • 1096
Image data injection before JPEG generation for lcp profile generation
« on: 21 / June / 2018, 19:12:09 »
Advertisements
I had this thought a while ago and I'll be brief about the key question: Is it at all possible to replace the image data captured by the sensor with "artificial data" prior to JPEG generation by the regular Canon FW, either pre or post interpolation.

The background to this is that at some point in 2009/2010 Canon shifted their approach to lens design. With more processing power being available lenses there started to favor the elimination of  astigmatism over that of field curvature simply because the latter can be fixed by an opposite amount of geometric distortion applied in image processing while the former cannot be remedied later on and while at it chose to make more widely angled compacts. The consequence is that up until 2009 RAW pixels equaled JPEG pixels. After that no more. Especially the 24mm wide angle equivalents are horrid without correction. I'll attach a D10 D20 comparison as the two cameras (arbitrary choice) are close enough and are models prior to after the design change.

To make RAW data useful again one would need suitable distortion data. Anything that writes raw natively embeds such in the RAW data. Well, that is beyond the feasible for CHDK only DNG for now. What to do depends on the focal length. It can be profiled but it is a mess.

The question I have thus is along the lines of: Could we feed a synthesized gridwork into the JPEG engine (which will distort it like it would distort the sensor data in creating a camera JPEG)? A JPEG of each focal length would be easy to make. These could then be used to analyze what the Canon FW does and enabling the same in the post processing chain. Thus getting to a point where RAW pixels can be made to match JPEG pixels once more. As it is raw data is much less useful out of the box on any modern camera that does not natively shoot raw as it used to be when CHDK was conceived.
« Last Edit: 21 / June / 2018, 19:14:04 by koshy »
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
*

Offline reyalp

  • ******
  • 14120
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #1 on: 21 / June / 2018, 23:02:38 »
Quote from: koshy on 21 / June / 2018, 19:12:09
I had this thought a while ago and I'll be brief about the key question: Is it at all possible to replace the image data captured by the sensor with "artificial data" prior to JPEG generation by the regular Canon FW, either pre or post interpolation.
Yes. "Raw develop" loads an entire raw file, replacing the data from the shot. Using https://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations you can also draw directly on the raw buffer using a lua script.
Quote
The question I have thus is along the lines of: Could we feed a synthesized gridwork into the JPEG engine (which will distort it like it would distort the sensor data in creating a camera JPEG)?
This is trivial to do with rawop.
Quote
A JPEG of each focal length would be easy to make.
This is somewhat annoying with modern cameras that have 100+ steps rather than a handful.
Quote
These could then be used to analyze what the Canon FW does and enabling the same in the post processing chain. Thus getting to a point where RAW pixels can be made to match JPEG pixels once more. As it is raw data is much less useful out of the box on any modern camera that does not natively shoot raw as it used to be when CHDK was conceived.
Yes. This was something I had in mind when I wrote the rawop stuff. The CHDK part is pretty trivial, but I don't know how to turn that into lens correction in a useful format. If someone wants to write a tool that does that, I'd be very happy to do the CHDK part.

The DNG spec has some distortion correction options, but I suspect many raw programs will ignore them anyway.
« Last Edit: 21 / June / 2018, 23:43:06 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline koshy

  • *****
  • 1096
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #2 on: 22 / June / 2018, 19:47:59 »
That all sounds more promising that I had feared.

Quote from: reyalp on 21 / June / 2018, 23:02:38
The DNG spec has some distortion correction options, but I suspect many raw programs will ignore them anyway.
Yes but availability will lead more to adapt and for all others any DNG with this can be converted to an intermediate "linear" DNG as Adobe calls it (which makes no sense) which is in a nut shell readily interpolated. That also takes care op bad pixel opcodes for software that does not support that.
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
*

Offline koshy

  • *****
  • 1096
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #3 on: 22 / June / 2018, 19:57:46 »
Quote from: reyalp on 21 / June / 2018, 23:02:38
I don't know how to turn that into lens correction in a useful format.
Adobe has done some work on this and it is in a peculiar state. I'll detail some of what I know about this and will attach a curious patent of theirs. The wording of the below may be a bit strange as I'll pull this from previous correspondence.

 I dug and was lucky enough to find a solid definition of all Adobe is doing in a patent of theirs.
"METHODS AND APPARATUS FOR CAMERA CALIBRATION BASED ON MULTI VIEW IMAGE GEOMETRY"

It is not as bad. the lcp file is redundant as each node stands for itself but it amounts to:
- finding the right lcp file to use (based on maker and lens information available as meta data)
- picking the right focal length/aperture combination entry from that file (aperture only if vignetting info is provided)

The details on geometric distortions in the PDF start in column 18 line 57

In the lcp files three parameters are provided for each focal length of the lens like:
Code: [Select]
stCamera:FocalLength="50"
[...]
stCamera:ScaleFactor="0.958189"
stCamera:RadialDistortParam1="0.187032"
stCamera:RadialDistortParam2="0.165785"
stCamera:RadialDistortParam3="0.40021"There also is the scale factor presumably to deal with differences in sensor size if such exist. Maybe also to deal with a difference in crop.

Then the details on vignetting in the PDF start in column 21 line 30

The vignetting of course depends on the aperture as well as on the focal length and makes for even more value sets in lcp like:
Code: [Select]
stCamera:FocalLength="50"
stCamera:FocusDistance="3"
stCamera:ApertureValue="8"
[...]
stCamera:VignetteModelParam1="-2.3844"
stCamera:VignetteModelParam2="4.041297"
stCamera:VignetteModelParam3="-6.086613"
Enough meta data is available to provide what would be needed to dig out the correct *.lcp file to use.

Normally the parameters of pincushion and other distortion are accurately determined as part of the process of designing a lens and so there is no need to estimate them after the fact.

In a perfect world the lens designer would just provide the data. However in a world where camera makers get the idea to cryptographically "secure" their while balancing meta data so that no other software besides their own raw converter can read it or maybe so that they can sell licenses that allow other software vendors with their raw image (Nikon did this) or go one step further and strongly encrypt the entire image (which Sigma did for Foveon images) it is altogether plausible that a software maker, namely Adobe needs to establish lens distortion parameters after the fact.
Actually those cameras that do provide this information readily in the raw data (Olympus / Panasonic / and I belive some Canon compacts that feature RAW natively) are not dealt with by Adobe's lcp files. They don't provide any. Adobe Camera Raw will display "Built in lens profile applied" and that will be all because as you say the maker ought to know best and if they are friendly enough to provide the data as meta data it gets used.

The main thrust of the Patent is in the development of such tables and then using the results, but the way I read the claims, using the results in itself is not covered unless you obtained the numbers according to the patent.  In other words, it is a very strange document in many ways. 

This will get more interesting if you learn what they did with it - because that obliterated the thrust of the patent.

It became the Adobe lens profile Creator which " is a free utility that enables the easy creation of lens profiles for use in the Photoshop family of products, such as Photoshop CC, the Camera Raw plug-in, and Lightroom. A lens profile describes the types of optical aberrations that exist in a particular lens and prescribes how to correct the lens distortions in an image captured from the same lens."

User Guide PDF: http://wwwimages.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/lensprofile_creator_userguide.pdf
Win: http://supportdownloads.adobe.com/detail.jsp?ftpID=5490
Mac: http://supportdownloads.adobe.com/detail.jsp?ftpID=5489

So, the results seem not to be covered by the patent, the equations technically can't be restricted if I get it right. Nobody needs to redistribute Adobe's *.lcp files since DNG converter installs them all. So that would leave the creation process and creation tool to Adobe but they give that away freely to all interested. The patent protects nothing but establishes Adobe's leading role in the field and maybe looks nice on the vita of the folks involved with this project. the software was released in 2012 and has not been altered thereafter.

I don't suppose there are other open source implementations?

I have not found anything command line. RawTherapee an open source GUI raw processing thing is said to support lcp.
Not that it would help us specifically but if we need to check on what they are doing with the lcp files at some point that is here:
http://rawtherapee.com/blog/screenshots
Source code: https://github.com/Beep6581/RawTherapee
« Last Edit: 22 / June / 2018, 20:22:54 by koshy »
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
*

Offline koshy

  • *****
  • 1096
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #4 on: 22 / June / 2018, 20:11:23 »
Quote from: reyalp on 21 / June / 2018, 23:02:38
The CHDK part is pretty trivial, but I don't know how to turn that into lens correction in a useful format. If someone wants to write a tool that does that, I'd be very happy to do the CHDK part.
So, to sum it up while the DNG embedded data would be even better imho, there is the lcp stuff from Adobe and a public creation tool exists as well as an open source GPL implementation on how to use the resulting files. Enough documentation to understand what it is all about, too. Could be worse.

Of course sending the gridwork their tool needs to the JPEG creation engine of the Canon FW will distort the gridwork rather than fix any distortion. The derived correction will thus be in the wrong direction. But either this can be remedied programatically for all resulting files in one go or at worst the resulting correction can be applied to the original gridwork again, to create imagery that will yield the right form in a second phase of processing. Won't be done in a day but might be feasible after all.
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
*

Offline waterwingz

  • ******
  • 12140
    • Waterwingz
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #5 on: 23 / June / 2018, 09:16:00 »
Quote from: koshy on 22 / June / 2018, 19:57:46
The patent protects nothing but establishes Adobe's leading role in the field and maybe looks nice on the vita of the folks involved with this project. the software was released in 2012 and has not been altered thereafter.
Actually, the patent provides two things to Adobe.

First of all it gives them control of the process even if the current software is “free".

Secondly, it prevents someone else from patenting it.

These might not seem like such big things by themselves but as part of a portfolio strategy they become quite valuable.
Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline Hardware_Hacker

  • ****
  • 322
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #6 on: 25 / June / 2018, 03:09:43 »
Quote from: koshy on 22 / June / 2018, 19:57:46
Quote from: reyalp on 21 / June / 2018, 23:02:38
I don't know how to turn that into lens correction in a useful format.
Adobe has done some work on this and it is in a peculiar state. I'll detail some of what I know about this and will attach a curious patent of theirs. The wording of the below may be a bit strange as I'll pull this from previous correspondence.

 I dug and was lucky enough to find a solid definition of all Adobe is doing in a patent of theirs.
"METHODS AND APPARATUS FOR CAMERA CALIBRATION BASED ON MULTI VIEW IMAGE GEOMETRY"

It is not as bad. the lcp file is redundant as each node stands for itself but it amounts to:
- finding the right lcp file to use (based on maker and lens information available as meta data)
- picking the right focal length/aperture combination entry from that file (aperture only if vignetting info is provided)

The details on geometric distortions in the PDF start in column 18 line 57

In the lcp files three parameters are provided for each focal length of the lens like:
Code: [Select]
stCamera:FocalLength="50"
[...]
stCamera:ScaleFactor="0.958189"
stCamera:RadialDistortParam1="0.187032"
stCamera:RadialDistortParam2="0.165785"
stCamera:RadialDistortParam3="0.40021"There also is the scale factor presumably to deal with differences in sensor size if such exist. Maybe also to deal with a difference in crop.

Then the details on vignetting in the PDF start in column 21 line 30

The vignetting of course depends on the aperture as well as on the focal length and makes for even more value sets in lcp like:
Code: [Select]
stCamera:FocalLength="50"
stCamera:FocusDistance="3"
stCamera:ApertureValue="8"
[...]
stCamera:VignetteModelParam1="-2.3844"
stCamera:VignetteModelParam2="4.041297"
stCamera:VignetteModelParam3="-6.086613"
Enough meta data is available to provide what would be needed to dig out the correct *.lcp file to use.

Normally the parameters of pincushion and other distortion are accurately determined as part of the process of designing a lens and so there is no need to estimate them after the fact.

I don't suppose there are other open source implementations?

My "wild guess" is Adobe's intermediate "linear" DNG and the "lcp profile generation" is generated by Adobe's PostScript.
i.e. By using “Direct PostScript”, and could posibly be done (initaly) by using GhostScript.

"...open source implementations?..."

By Using the “Direct GhostScript” and the Gonzo Utilities (by Don Lancaster), more info in a later post.

The attached file "D20_Dng_[MV-PW 2x 100].gif" shows image "De-Barrel" using "Pano Warp by MV's Plugins", twice, and some other 8BF plugins.

"...the defishing algorithms used in magic lantern..." might also be of some use.

H-H
« Last Edit: 25 / June / 2018, 03:24:46 by Hardware_Hacker »
Logged
*

Offline koshy

  • *****
  • 1096
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #7 on: 25 / June / 2018, 19:53:46 »
Quote from: Hardware_Hacker on 25 / June / 2018, 03:09:43
My "wild guess" is Adobe's intermediate "linear" DNG and the "lcp profile generation" is generated by Adobe's PostScript.
I'm not following your reasoning but if it helps:
I was referring to DNG Converter as shown in the attached screen shot. "DNG Backward Version" can be set to something lower than shown. That was in reply to:
Quote from: reyalp on 21 / June / 2018, 23:02:38
The DNG spec has some distortion correction options, but I suspect many raw programs will ignore them anyway.
With the point being that even if there are raw programs that don't support the DNG feature, DNGs can be converted to a state where they can be used with such programs. With the use of such DNG features becoming more wide spread software will adapt. Not using them because little but Adobe's stuff supports the feature gets this backwards.  ;)
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
*

Offline reyalp

  • ******
  • 14120
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #8 on: 25 / June / 2018, 22:59:38 »
Quote from: koshy on 25 / June / 2018, 19:53:46
With the point being that even if there are raw programs that don't support the DNG feature, DNGs can be converted to a state where they can be used with such programs.
I would be surprised if linear DNGs were better supported than the other features. In any case, I tested converting some native G7X CR2 images and they do not appear to have distortion applied.

Quote
With the use of such DNG features becoming more wide spread software will adapt. Not using them because little but Adobe's stuff supports the feature gets this backwards.
The experience with badpixel opcodes suggests that 3rd party developers are unlikely to make decisions based on which DNG features CHDK uses.

Open source projects might well accept code implementing advanced DNG features, but this would likely involve a lot of work.
Logged
Don't forget what the H stands for.
*

Offline koshy

  • *****
  • 1096
Re: Image data injection before JPEG generation for lcp profile generation
« Reply #9 on: 26 / June / 2018, 17:01:25 »
Quote from: reyalp on 25 / June / 2018, 22:59:38
I would be surprised if linear DNGs were better supported than the other features. In any case, I tested converting some native G7X CR2 images and they do not appear to have distortion applied.
Great you're in for a surprise. That was not an addition. It was part of the original standard.
It is the most trivial case, too (no demosaicing) and actually is supported anywhere that supports DNG.

PhotometricInterpretation
[...]
The following values are supported for the raw IFD, and are assumed to be the camera's native
color space:
• 32803 = CFA (Color Filter Array).
• 34892 = LinearRaw.
The CFA PhotometricInterpretation value is documented in the TIFF-EP specification. Its use
requires the use of the CFARepeatPatternDim and CFAPattern tags in the same IFD. The
origin of the repeating CFA pattern is the top-left corner of the ActiveArea rectangle.
The LinearRaw PhotometricInterpretation value is intended for use by cameras that do not use
color filter arrays, but instead capture all color components at each pixel. It can also be used
for CFA data that has already been de-mosaiced.
The LinearRaw value can be used in reduced resolution IFDs, even if the raw IFD uses the
CFA PhotometricInterpretation value.

Didn't check how easy the old 1.1 DNG spec is to be found these days. Attaching it from the archive.
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
Pages: [1] 2 3 4   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal