Improving support for cameras with Canon raw - page 3 - General Discussion and Assistance - CHDK Forum  
« previous next »
Pages: 1 2 [3] 4   Go Down

Improving support for cameras with Canon raw

  • 31 Replies
  • 2342 Views
*

Offline reyalp

  • ******
  • 12731
Re: Improving support for cameras with Canon raw
« Reply #20 on: 14 / August / 2020, 01:50:05 »
Advertisements
Nice, thanks. I checked visually by building stubs and greping #define\s+MAX_CHUNKS_FOR_JPEG, didn't see any other mismatches.

Checked in.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 12731
Re: Improving support for cameras with Canon raw
« Reply #21 on: 19 / August / 2020, 21:51:02 »
Here's an updated patch with remoteshoot raw support

It requires updated chdkptp lua files, r939 or later from https://app.assembla.com/spaces/chdkptp/subversion/source/HEAD/trunk/lua

changes from the earlier version
* file type is now detected based on the name in the filewrite structure. We didn't use this before, so it could be wrong on some cameras, but it's only done on cameras with native raw so the impact should be limited.

chdkptp side
* now sets the raw/jpeg options as needed, so if you do -craw while the camera is set only to jpeg, it will switch to raw, and restore it after the shots are done.
* The -jpgdummy option is now -filedummy and creates cr2 and/or jpg files based on the selected formats (the old name is still accepted and does the same thing). I'm not sure if cams in raw only mode have the same crash that some do when the jpegs are mission.
* An unrelated bug that caused the -quick option to sometimes fail is fixed.

I've tested this on my cams and intend to check it in, but additional testing and feedback is welcome. I'd be particularly interested in cameras that use CAM_FILEWRITETASK_MULTIPASS without CAM_FILEWRITETASK_SEEKS (a495 a490 a3300 a3200 a800) since I don't have any those, and cameras with native raw.

It can be tested by running
Code: [Select]
chdkptp -e"exec require'camtests'.runbatch{shoot=true}"
Note the regular, non-remote shoot portion of the test may fail on cameras with native raw, if the camera isn't set to jpeg mode. I'll fix that after I get the other stuff checked in. fixed in chdkptp r940
« Last Edit: 28 / August / 2020, 22:53:08 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4285
Re: Improving support for cameras with Canon raw
« Reply #22 on: 21 / August / 2020, 10:18:13 »
Quote from: reyalp on 19 / August / 2020, 21:51:02
It can be tested by running
Code: [Select]
chdkptp -e"exec require'camtests'.runbatch{shoot=true}"
Well, I'm not having much luck.
G10, chdkptp r939:
Code: [Select]
___> exec require'camtests'.runbatch{shoot=true}
connect:start
using default device
c
connected: Canon PowerShot G10, max packet size 512
connect:ok
cam_info:start
platform:g10-102a version:1.5.0-5553 built:Aug 21 2020 15:02:46
CHDK core start:0x0eeecc size:141348 total free:964992
system heap start:0x111890 size:2493456 free:962128 free_block_max:960224
SD size:1926848 KB free:1803712 KB
cam_info:ok
list_connected:start
list
list_connected:ok
wait_status:start
wait_status:ok
exec_errors:start
exec_errors:ok
msgfuncs:start
msgfuncs:ok
rec:start
rec
rec:ok
rec_info:start
Rec info:
Propset:2 mode:P (2,still) flash_m:2 iso_m:800 focus_m:0 ND:2 zoom:0/14 rc:15 res:0 qual:0
Exposure info:
Tv:1/100.86 (639) Av:4.0 (384) minAv:2.9 (293) Sv:486 (699) SvM:800 (768) Bv:3 (363) NDEv:0 (0) SD:1612,!OK
rec_info:ok
remoteshoot:start
remoteshoot -jpg -filedummy camtest/
remoteshoot -seq=120 -craw -filedummy camtest/${imgpfx}_${imgfmt}_${shotseq}${ext}
WARNING: capture_get_data error I/O error
WARNING: error waiting for shot script nil
(...)
disconnect:start
dis
disconnect:ok
not_connected:start
not_connected:ok
passed 10
failed 5
The camera crashes right after the second shot (recreview appears on screen, LED lights up for half a second, then they go black).
A zero-byte jpeg remains on camera, one complete jpeg in camtest folder.

edit:
LED = card activity LED which blinks while a file is saved, it just lights up in this case, so it crashes early

edit2:
a3200
Code: [Select]
___> exec require'camtests'.runbatch{shoot=true}
connect:start
using default device
c
connected: Canon PowerShot A3200 IS, max packet size 512
connect:ok
cam_info:start
platform:a3200-100d version:1.5.0-5553 built:Aug 21 2020 16:10:12
CHDK core start:0x3ca4be0 size:138176 total free:592452
system heap start:0x16a898 size:1602888 free:368312 free_block_max:367328
exmem heap start:0x3cc67a0 size:386112 free:220852 free_block_max:187360
SD size:1926848 KB free:1782400 KB
cam_info:ok
list_connected:start
list
list_connected:ok
wait_status:start
wait_status:ok
exec_errors:start
exec_errors:ok
msgfuncs:start
msgfuncs:ok
rec:start
rec
rec:ok
rec_info:start
Rec info:
Propset:4 mode:P (2,still) flash_m:2 iso_m:200 focus_m:0 ND:1 zoom:0/12 rc:7 res:0 qual:1
Exposure info:
Tv:1/12.43 (349) Av:2.9 (293) minAv:2.9 (293) Sv:121 (507) SvM:200 (576) Bv:1 (147) NDEv:0 (0) SD:-1,OK
rec_info:ok
remoteshoot:start
remoteshoot -jpg -filedummy camtest/
remoteshoot:failed (...)/chdkptp-new2/trunk/lua/camtests.lua:605: init failed
stack traceback:
        (...)/chdkptp-new2/trunk/lua/camtests.lua:170: in function 'cliexec'
        (...)/chdkptp-new2/trunk/lua/camtests.lua:605: in function <(...)/chdkptp-new2/trunk/lua/camtests.lua:572>
        [C]: in function 'xpcall'
        (...)/chdkptp-new2/trunk/lua/camtests.lua:760: in function 'run'
        (...)/chdkptp-new2/trunk/lua/camtests.lua:819: in function 'runbatch'
        [string "require'camtests'.runbatch{shoot=true}"]:1: in main chunk
        [C]: in function 'xpcall'
        (...)/chdkptp-new2/trunk/lua/cli.lua:748: in function <(...)/chdkptp-new2/trunk/lua/cli.lua:739>
        (...tail calls...)
        [C]: in function 'xpcall'
        (...)/chdkptp-new2/trunk/lua/cli.lua:285: in function 'execute'
        (...)/chdkptp-new2/trunk/lua/cli.lua:400: in function <(...)/chdkptp-new2/trunk/lua/cli.lua:391>
        (...tail calls...)
        (...)/chdkptp-new2/trunk/lua/main.lua:295: in main chunk
        [C]: in function 'require'
        [string "require('main')"]:1: in main chunk
rsint:start
rsint:failed (...)/chdkptp-new2/trunk/lua/camtests.lua:681: init failed
stack traceback:
        (...)/chdkptp-new2/trunk/lua/camtests.lua:681: in function <(...)/chdkptp-new2/trunk/lua/camtests.lua:658>
        [C]: in function 'xpcall'
        (...)/chdkptp-new2/trunk/lua/camtests.lua:760: in function 'run'
        (...)/chdkptp-new2/trunk/lua/camtests.lua:820: in function 'runbatch'
        [string "require'camtests'.runbatch{shoot=true}"]:1: in main chunk
        [C]: in function 'xpcall'
        (...)/chdkptp-new2/trunk/lua/cli.lua:748: in function <(...)/chdkptp-new2/trunk/lua/cli.lua:739>
        (...tail calls...)
        [C]: in function 'xpcall'
        (...)/chdkptp-new2/trunk/lua/cli.lua:285: in function 'execute'
        (...)/chdkptp-new2/trunk/lua/cli.lua:400: in function <(...)/chdkptp-new2/trunk/lua/cli.lua:391>
        (...tail calls...)
        (...)/chdkptp-new2/trunk/lua/main.lua:295: in main chunk
        [C]: in function 'require'
        [string "require('main')"]:1: in main chunk
shoot:start
shoot -dng=1 -seq=300 -dl=camtest/IMG_${shotseq}.${imgfmt}
A/DCIM/279___08/CRW_0188.DNG->camtest/IMG_0300.DNG
A/DCIM/279___08/IMG_0188.JPG->camtest/IMG_0300.JPG
shoot:ok
play:start
play
play:ok
reconnect:start
reconnect
reconnect:ok
disconnect:start
dis
disconnect:ok
not_connected:start
not_connected:ok
passed 13
failed 2
rs fails (no shot is taken), regular shot succeeds, camera doesn't crash.
« Last Edit: 21 / August / 2020, 10:47:20 by srsa_4c »
Logged
*

Offline reyalp

  • ******
  • 12731
Re: Improving support for cameras with Canon raw
« Reply #23 on: 21 / August / 2020, 20:56:43 »
Quote from: srsa_4c on 21 / August / 2020, 10:18:13
Well, I'm not having much luck.
G10, chdkptp r939:
...
The camera crashes right after the second shot (recreview appears on screen, LED lights up for half a second, then they go black).
A zero-byte jpeg remains on camera, one complete jpeg in camtest folder.
Hmm. That's not good.

Can you try with FILEWRITE_DEBUG_LOG 1 ?

The romlog looks like it might be review related, what is the review setting?

Quote
a3200
Does the a3200 have continuous shooting enabled in the Canon menu? This was the generation of cam that had problems with cont, so init fails if it's on.
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4285
Re: Improving support for cameras with Canon raw
« Reply #24 on: 22 / August / 2020, 13:06:14 »
Quote from: reyalp on 21 / August / 2020, 20:56:43
Does the a3200 have continuous shooting enabled in the Canon menu? This was the generation of cam that had problems with cont, so init fails if it's on.
Thanks for the reminder, that made the test succeed.
Code: [Select]
___> exec require'camtests'.runbatch{shoot=true}
connect:start
using default device
c
connected: Canon PowerShot A3200 IS, max packet size 512
connect:ok
cam_info:start
platform:a3200-100d version:1.5.0-5553 built:Aug 21 2020 16:10:12
CHDK core start:0x3ca4be0 size:138176 total free:592452
system heap start:0x16a898 size:1602888 free:368312 free_block_max:367328
exmem heap start:0x3cc67a0 size:386112 free:220852 free_block_max:187360
SD size:1926848 KB free:1803776 KB
cam_info:ok
list_connected:start
list
list_connected:ok
wait_status:start
wait_status:ok
exec_errors:start
exec_errors:ok
msgfuncs:start
msgfuncs:ok
rec:start
rec
rec:ok
rec_info:start
Rec info:
Propset:4 mode:P (2,still) flash_m:2 iso_m:200 focus_m:0 ND:1 zoom:0/12 rc:7 res:0 qual:1
Exposure info:
Tv:1/4.90 (220) Av:2.9 (293) minAv:2.9 (293) Sv:121 (507) SvM:200 (576) Bv:0 (-31) NDEv:0 (0) SD:-1,OK
rec_info:ok
remoteshoot:start
remoteshoot -jpg -filedummy camtest/
remoteshoot -craw -filedummy camtest/
remoteshoot -seq=100 -dng -jpg -filedummy camtest/${imgpfx}_${imgfmt}_${shotseq}${ext}
remoteshoot -raw -dnghdr -filedummy camtest/${imgfmt}_${shotseq}${ext}
remoteshoot -quick=3 -jpg -filedummy camtest/${imgfmt}_${shotseq}${ext}
remoteshoot -quick=3 -int=5 -jpg -filedummy camtest/${imgfmt}_${shotseq}${ext}
cont mode not set, skipping remoteshoot cont test
remoteshoot:ok
rsint:start
cont mode not set, skipping rsint cont test
rsint:ok
shoot:start
shoot -dng=1 -seq=300 -dl=camtest/IMG_${shotseq}.${imgfmt}
A/DCIM/279___08/CRW_0200.DNG->camtest/IMG_0300.DNG
A/DCIM/279___08/IMG_0200.JPG->camtest/IMG_0300.JPG
shoot:ok
play:start
play
play:ok
reconnect:start
reconnect
reconnect:ok
disconnect:start
dis
disconnect:ok
not_connected:start
not_connected:ok
passed 15
failed 0
Quote
Can you try with FILEWRITE_DEBUG_LOG 1 ?
I made a build with that, but no new lines appeared in the romlog. The crash happened with review set to 2 seconds in Canon menu. Setting review to 'off' made the test succeed.
Code: [Select]
___> exec require'camtests'.runbatch{shoot=true}
connect:start
using default device
c
connected: Canon PowerShot G10, max packet size 512
connect:ok
cam_info:start
platform:g10-102a version:1.5.0-5553 built:Aug 22 2020 14:28:27
CHDK core start:0x0eeecc size:141572 total free:964752
system heap start:0x111970 size:2493232 free:961888 free_block_max:959992
SD size:1926848 KB free:1778336 KB
cam_info:ok
list_connected:start
list
list_connected:ok
wait_status:start
wait_status:ok
exec_errors:start
exec_errors:ok
msgfuncs:start
msgfuncs:ok
rec:start
rec
rec:ok
rec_info:start
Rec info:
Propset:2 mode:P (2,still) flash_m:2 iso_m:0 focus_m:0 ND:2 zoom:0/14 rc:15 res:0 qual:0
Exposure info:
Tv:1/8.00 (288) Av:2.9 (293) minAv:2.9 (293) Sv:126 (513) SvM:100 (480) Bv:0 (68) NDEv:0 (0) SD:774,OK
rec_info:ok
remoteshoot:start
remoteshoot -jpg -filedummy camtest/
remoteshoot -seq=120 -craw -filedummy camtest/${imgpfx}_${imgfmt}_${shotseq}${ext}
remoteshoot -craw -jpg -filedummy camtest/${imgpfx}_${imgfmt}_${shotseq}${ext}
remoteshoot -seq=100 -dng -jpg -filedummy camtest/${imgpfx}_${imgfmt}_${shotseq}${ext}
remoteshoot -raw -dnghdr -filedummy camtest/${imgfmt}_${shotseq}${ext}
remoteshoot -quick=3 -jpg -filedummy camtest/${imgfmt}_${shotseq}${ext}
remoteshoot -quick=3 -int=5 -jpg -filedummy camtest/${imgfmt}_${shotseq}${ext}
remoteshoot -quick=3 -craw -filedummy camtest/${imgfmt}_${shotseq}${ext}
remoteshoot -quick=3 -craw -jpg -filedummy camtest/${imgfmt}_${shotseq}${ext}
cont mode not set, skipping remoteshoot cont test
remoteshoot:ok
rsint:start
cont mode not set, skipping rsint cont test
rsint:ok
shoot:start
shoot -dng=1 -seq=300 -dl=camtest/IMG_${shotseq}.${imgfmt}
A/DCIM/280CANON/CRW_0021.DNG->camtest/IMG_0300.DNG
A/DCIM/280CANON/IMG_0021.JPG->camtest/IMG_0300.JPG
shoot:ok
play:start
play
play:ok
reconnect:start
reconnect
reconnect:ok
disconnect:start
dis
disconnect:ok
not_connected:start
not_connected:ok
passed 15
failed 0
If the test is able to crash the camera, it would be useful to know how.

edit:
A simple rs -craw is enough to trigger the crash.
« Last Edit: 22 / August / 2020, 17:55:46 by srsa_4c »
Logged
*

Offline reyalp

  • ******
  • 12731
Re: Improving support for cameras with Canon raw
« Reply #25 on: 22 / August / 2020, 19:50:11 »
Quote from: srsa_4c on 22 / August / 2020, 13:06:14
Thanks for the reminder, that made the test succeed.
Great. That behavior from chdkptp isn't very friendly, but it's only a few cams...
Quote
Can you try with FILEWRITE_DEBUG_LOG 1 ?
I made a build with that, but no new lines appeared in the romlog. The crash happened with review set to 2 seconds in Canon menu. Setting review to 'off' made the test succeed.
[/quote]
Hm, assuming you're using the code from 5547 or later, it really should output something, but I guess at 2 seconds it might have all scrolled out of the log.

Quote
If the test is able to crash the camera, it would be useful to know how.
Agreed.  Review is usually not desirable with remote shoot, but it would definitely be better not to crash.
Quote
edit:
A simple rs -craw is enough to trigger the crash.
I assume rs -jpg does not?

What format was set in the Canon UI? I wonder if the crash could be related to forcing the propcases, so if it was on jpeg, you might try setting to raw only in the Canon UI.

I wasn't able to reproduce it on G7 X or a couple of my jpeg only cams with review enabled, but that's not too surprising.
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4285
Re: Improving support for cameras with Canon raw
« Reply #26 on: 23 / August / 2020, 17:41:35 »
Quote from: reyalp on 22 / August / 2020, 19:50:11
What format was set in the Canon UI? I wonder if the crash could be related to forcing the propcases, so if it was on jpeg, you might try setting to raw only in the Canon UI.
I did some testing and found that the crash has nothing to do with remote shooting.
- there are no problems when review is off
- when review is on and the Canon UI is set to JPG, forcing craw with set_canon_image_format() results in crash after the next photo
- other combinations (jpg->raw+jpg, craw->jpg) do not seem to cause trouble
Could be a propset 2 thing or specific to G10.
Too bad that the review setting is only exposed as a param...
Logged
*

Offline reyalp

  • ******
  • 12731
Re: Improving support for cameras with Canon raw
« Reply #27 on: 23 / August / 2020, 18:53:34 »
Quote from: srsa_4c on 23 / August / 2020, 17:41:35
I did some testing and found that the crash has nothing to do with remote shooting.
- there are no problems when review is off
- when review is on and the Canon UI is set to JPG, forcing craw with set_canon_image_format() results in crash after the next photo
- other combinations (jpg->raw+jpg, craw->jpg) do not seem to cause trouble
Could be a propset 2 thing or specific to G10.
Thanks, good to know. My guess would be propset 2 and 3 which have raw as a resolution and "raw with jpeg" as a menu option.

@koshy
Can you try on g11:
In the Canon menu, set Review to 2s
In func menu, set image size to L (not RAW)
In chdkptp
rs -craw

Quote
Too bad that the review setting is only exposed as a param...
Yeah, disabling review (by default at least) in remoteshoot would be nice generally, since it impacts shooting rate or even hangs if it's hold mode. The review setting is a prop (and settable) on later cams, at least from propset 5.

edit:
G11 100k build attached. You'll also need the latest lua files for chdkptp (no need to rebuild the executable)
I could make remoteshoot use the raw+jpeg setting for propset 2/3 cams. It would be a bit slower because the jpeg would still be encoded and discarded instead of being skipped.
« Last Edit: 23 / August / 2020, 19:28:10 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline koshy

  • *****
  • 1051
Re: Improving support for cameras with Canon raw
« Reply #28 on: 23 / August / 2020, 19:34:15 »
Quote from: reyalp on 23 / August / 2020, 18:53:34
edit: G11 100k build attached.
Ah, hence the
Quote
> rs -craw
ERROR: unsupported format
with the latest trunk  :o
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
*

Offline koshy

  • *****
  • 1051
Re: Improving support for cameras with Canon raw
« Reply #29 on: 23 / August / 2020, 19:42:50 »
Quote from: reyalp on 23 / August / 2020, 18:53:34
edit: G11 100k build attached.
That gives this:
Code: [Select]
> rs -craw
WARNING: capture_get_data error I/O error
WARNING: error waiting for shot script nil
ERROR: I/O error uninit I/O errorand the cam crashes, enclosing ROMLOG
Logged
Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)
Pages: 1 2 [3] 4   Go Up
« previous next »
 

Related Topics