Can't use CRW files

uhoh

14

Re: Can't use CRW files

« Reply #10 on: 09 / March / 2017, 11:57:23 »

Advertisements

Quote from: reyalp on 09 / March / 2017, 02:14:38

If you can upload an example of one of the original CRW files somewhere (dropbox, google drive etc), we may be able to get a better idea of what's happening.

Note that rawconvert does not demosiac, it just takes the packed 12 bit data and converts it 8 or 16 bit formats that can be loaded into other imaging programs. If you load the resulting pgm in gimp or similar, you will generally a recognizable grayscale image with a checkerboard pattern from the CFA.

OK here's the pgm: https://www.dropbox.com/s/ec1thtgb15efyzl/CRW_2435.CRW.pgm?dl=0

And here's the jpeg:
https://www.dropbox.com/s/p9ydk6tkasau614/IMG_2435.JPG?dl=0

Re not demosaicing, that makes sense. Probably it's time to add that feature, if I can find a decent algorithm.

Logged

reyalp

14003

Re: Can't use CRW files

« Reply #11 on: 09 / March / 2017, 16:17:50 »

Quote from: uhoh on 09 / March / 2017, 11:57:23

OK here's the pgm: https://www.dropbox.com/s/ec1thtgb15efyzl/CRW_2435.CRW.pgm?dl=0

And here's the jpeg:
https://www.dropbox.com/s/p9ydk6tkasau614/IMG_2435.JPG?dl=0

I asked for the original CRW, but from the PGM I would say your CRW only contains random garbage from camera memory, not image data. This happens on some cameras where the raw data isn't available in certain shooting modes (often AUTO), or the raw buffer address is wrong. According the jpeg EXIF, this image was shot in AUTO, so that is likely to be the case.

If any of your CRWs are OK, you should be able to recognize the image in the PGM.

Quote

Re not demosaicing, that makes sense. Probably it's time to add that feature, if I can find a decent algorithm.

As noted in the source, rawconvert is intended as a debugging tool, not an imaging too. Getting good looking images out of raw requires a lot more than just demosiacing, especially for CRW framebuffer dump raw which have no calibration or other metadata.

You could feed the output of rawconvert to something that knows how to demosiac (dcraw, or pambayerp perhaps?) but again, getting nice results requires proper calibration.

DNG4PS2 gets around some of this by using the jpeg for metadata. If some of your raws contain valid images, setting up a VM that can run DNG4PS2 might be a good option. Or maybe the jpegs are good enough

Logged

Don't forget what the H stands for.

uhoh

14

Re: Can't use CRW files

« Reply #12 on: 09 / March / 2017, 18:53:19 »

Quote from: reyalp on 09 / March / 2017, 16:17:50

Quote from: uhoh on 09 / March / 2017, 11:57:23
OK here's the pgm: https://www.dropbox.com/s/ec1thtgb15efyzl/CRW_2435.CRW.pgm?dl=0

And here's the jpeg:
https://www.dropbox.com/s/p9ydk6tkasau614/IMG_2435.JPG?dl=0
I asked for the original CRW

It was in the original posting:
https://www.dropbox.com/s/93ew7b8sdc8zkmz/CRW_2435.CRW?dl=0

Anyway I think my best option here is to as you say accept the JPEGs or if possible re-shoot these, storing as DNGs.

« Last Edit: 09 / March / 2017, 18:55:55 by uhoh »

Logged

reyalp

14003

Re: Can't use CRW files

« Reply #13 on: 09 / March / 2017, 22:24:02 »

Quote from: uhoh on 09 / March / 2017, 18:53:19

It was in the original posting:
https://www.dropbox.com/s/93ew7b8sdc8zkmz/CRW_2435.CRW?dl=0

Oh, sorry, I missed that. I can confirm this file doesn't contain valid image data.

Quote

Anyway I think my best option here is to as you say accept the JPEGs or if possible re-shoot these, storing as DNGs.

Yes, and use P mode rather than AUTO.

If you can confirm that all the images you shot in AUTO are bad like the one you posted, I will update the port to disable raw in AUTO. Many cameras of this generation don't seem to have valid raw data in AUTO.

If only some of them are bad, or you also sometimes get the bad images in P mode, there may be some other bug in the port.

When you test, you should try both single shot and continuous shooting for 3 or more shots.

Logged

Don't forget what the H stands for.

uhoh

14

Re: Can't use CRW files

« Reply #14 on: 10 / March / 2017, 10:49:07 »

Quote from: reyalp on 09 / March / 2017, 22:24:02

If you can confirm that all the images you shot in AUTO are bad like the one you posted, I will update the port to disable raw in AUTO. Many cameras of this generation don't seem to have valid raw data in AUTO.

OS/X says the bad CRW' JPGs are:
Scene Capture Type: Standard
Exposure Mode: Auto exposure.

Earlier good DNGs' JPGs were
Scene Capture Type: Portrait
Exposure Mode: Auto exposure

I'll try the continuous shooting this weekend.

« Last Edit: 10 / March / 2017, 11:03:37 by uhoh »

Logged

reyalp

14003

Re: Can't use CRW files

« Reply #15 on: 10 / March / 2017, 13:18:19 »

Quote from: uhoh on 10 / March / 2017, 10:49:07

OS/X says the bad CRW' JPGs are:
Scene Capture Type: Standard
Exposure Mode: Auto exposure.

Yeah, sounds like this is one of the cameras that has trouble in auto mode. There were similar reports in the porting thread, but it wasn't really confirmed.

For most normal shooting, I'd suggest using P (Program AE)

Logged

Don't forget what the H stands for.

uhoh

14

Re: Can't use CRW files

« Reply #16 on: 12 / March / 2017, 17:37:00 »

Quote from: reyalp on 09 / March / 2017, 02:14:38

When you test, you should try both single shot and continuous shooting for 3 or more shots.

OK, DNGs still work fine. They work in both Portrait mode and in P mode.

All close-up DNGs have a fish-eye appearance whereas their equivalent JPGs do not.

Continuous mode does not initially support DNGs and just produced a series of JPGs, but then I enabled it in the settings and it works fine but it slow.

The High Speed Burst 4MB mode does not produce DNGs.

The monochrome mode produces color DNGs of course.

« Last Edit: 12 / March / 2017, 18:00:36 by uhoh »

Logged

reyalp

14003

Re: Can't use CRW files

« Reply #17 on: 12 / March / 2017, 19:43:27 »

Thanks for testing. I've updated the port to disable raw in auto mode, build 4767 and later.

Quote from: uhoh on 12 / March / 2017, 17:37:00

All close-up DNGs have a fish-eye appearance whereas their equivalent JPGs do not.

This is normal, the camera does jpeg distortion correction in software, the raw is what the sensor actually recorded. You can use a lens profile (there's one for ixus125/elph110 in https://chdk.setepontos.com/index.php?topic=9268.0) or correct the distortion in programs like raw therapee.

Quote

The High Speed Burst 4MB mode does not produce DNGs.

Does not save DNGs at all, or saves bad ones? The code for this port is set to disable raw in "low light", which might also catch this mode since they are implemented the same way. In any case, these reduced resolution modes are not expected to generate usable raw, so not saving in this mode would be correct.

Logged

Don't forget what the H stands for.

uhoh

14

Re: Can't use CRW files

« Reply #18 on: 13 / March / 2017, 08:36:27 »

Quote from: reyalp on 12 / March / 2017, 19:43:27

Does not save DNGs at all, or saves bad ones? The code for this port is set to disable raw in "low light", which might also catch this mode since they are implemented the same way. In any case, these reduced resolution modes are not expected to generate usable raw, so not saving in this mode would be correct.

Does not save.

Re not saving in low light, what's the reason?
I've used this camera to take astronomical photos and I plan to continue, but there have been problems with long exposures sometimes going all white.

Logged

reyalp

14003

Re: Can't use CRW files

« Reply #19 on: 13 / March / 2017, 17:14:00 »

Quote from: uhoh on 13 / March / 2017, 08:36:27

Re not saving in low light, what's the reason?

I meant the special "low light" or "ISO3200" modes, not just shooting in low light. These modes are half resolution like the high speed burst one you mentioned, and all work by binning the raw data to lower resolution before the main CPU sees it. CHDK currently isn't able to save useful raw in these cases, see https://chdk.setepontos.com/index.php?topic=10648.0

Quote

I've used this camera to take astronomical photos and I plan to continue, but there have been problems with long exposures sometimes going all white.

How long? It's not unusual to run into problems when exposures are many minutes long, but in this case, you will probably get better results taking multiple shots and stacking them. This can be done dedicated software like registax or deep sky stacker, or some general imaging software like gmic.

Logged

Don't forget what the H stands for.