CHDK Forum
Using CHDK => General Help and Assistance on using CHDK stable releases => Topic started by: Razor512 on 09 / January / 2015, 22:24:56
-
Hi, I recently updated to version 1.3, and did not backup the card, or save the original 1.2 download.
While 1.3 has some great features that I like, I have once in a while had trouble many scripts, and saving raw files. (some scripts will simply cause no raw file to be saved, or save corrupted raw files, for example, attempting to do an HDR bracket, or time lapse, will have raw files simply not be saved, or a number of 38KB raw files. (attached a sample)
anyway, at least for now, I would like to downgrade to the 1.2 builds, but cannot find the latest 1.2 build for the canon powershot SX230 HS (firmware 1.01B)
-
The final 1.2.0 builds are conserved here : https://app.box.com/s/p7ih4eqbcj8r9u09iwcj
You will need the common file in _common_files.zip (https://app.box.com/s/p7ih4eqbcj8r9u09iwcj/1/2836533057/24093399657/1) plus the sx230hs-101b-1.2.0-3770_BETA.zip (https://app.box.com/s/p7ih4eqbcj8r9u09iwcj/2/2836533057/24093347889/1) file.
FWIW - I do not believe your issues are in any way related to the update from 1.2.0 to 1.3.0. If it is, this will be a surprise to all the CHDK developers and we would like to know a lot more about what you are seeing.
As step one, please confirm that reverting to 1.2.0 solves all your problems.
TIA
-
I recorded a video of the issue, but basically what happens is if multiple images are taken either with the shutter button held down, or a script set to take multiple images, only 1 or 2 raw files will be saved.
If multiple images are taken in a row in pretty much single shot mode, then the raw files save fine every time. (I do not have the option enabled that only saves a single raw file in a series)
https://www.youtube.com/watch?v=PigJRE7iIWE&feature=youtu.be (https://www.youtube.com/watch?v=PigJRE7iIWE&feature=youtu.be)
With version 1.2, the issue goes away.
Video performing the same test with CHDK 1.2
https://www.youtube.com/watch?v=-sbbfsWGKlQ&feature=youtu.be (https://www.youtube.com/watch?v=-sbbfsWGKlQ&feature=youtu.be)
-
I recorded a video of the issue, but basically what happens is if multiple images are taken either with the shutter button held down, or a script set to take multiple images, only 1 or 2 raw files will be saved.
If multiple images are taken in a row in pretty much single shot mode, then the raw files save fine every time. (I do not have the option enabled that only saves a single raw file in a series)
https://www.youtube.com/watch?v=PigJRE7iIWE&feature=youtu.be (https://www.youtube.com/watch?v=PigJRE7iIWE&feature=youtu.be)
With version 1.2, the issue goes away.
Video performing the same test with CHDK 1.2
https://www.youtube.com/watch?v=-sbbfsWGKlQ&feature=youtu.be (https://www.youtube.com/watch?v=-sbbfsWGKlQ&feature=youtu.be)
I can't think of any changes that would cause this - there don't appear to be any SX230 specific changes between the 2 versions you are using (except removal of unused GPS code).
When you updated to 1.3 did you replace all the CHDK files (including modules) or just the diskboot.bin file?
What happens if you turn off DNG and just save CHDK RAW files?
Have you tried resetting all your CHDK settings (delete the *.CFG in the CHDK directory; but keep a copy so you can restore them if needed)?
Can you post the script you are using please.
Phil.
-
I will try the latest nightly of 1.3
my install process is always erasing the card, then extracting the zip file, then copying everything except the text files in the root folder to the SD card. The first thing I do is then disable the CHDK boot logo, make the card bootable, disable the temperature info and other random OSD elements, then create the badpixel file with the camera set to 1600ISO which creates a 40KB badpixel file. I then enable raw and set the DNG to 1.1
The script that I am using is attached in this reply (awesome script, it works quickly, and allows for many exposures to be taken with smaller step sizes so that HDR images have less noise).
Edit: I just tried the nightly build sx230hs-101b-1.3.0-3903-full.zip same issue with the settings from the video. (the card was formatted first)
with DNG disabled, the issue still happens.
-
This port had unresolved issues with the file counter
http://chdk.setepontos.com/index.php?topic=11527.msg113065#msg113065 (http://chdk.setepontos.com/index.php?topic=11527.msg113065#msg113065)
-
cool, will use that (I think I lost that script some time ago, and ended up grabbing one from the main wiki)
For the raw files not saving, it happens even when not using a script, and simply using the continuous shooting, this it is not a script issue.
Between version 1.2, and 1.3, version 1.3 has different raw options, as it allows you to even change the crop size of the raw file. (changing the crop size does not have an impact on the issue).
Other than that, CHDK has been the best thing to happen to canon point and shoots. (pretty much all of the images were taken with its raw function https://www.flickr.com/photos/razor512/ (https://www.flickr.com/photos/razor512/)
-
For the raw files not saving, it happens even when not using a script, and simply using the continuous shooting, this it is not a script issue.
The link I posted explains why this happens. I posted a second test build to further diagnose the problem (in May of 2014) but never heard back, so I made no changes in the code.
-
Sorry for not getting back, I just enabled email notifications so that I will be able to better keep track of these threads.
For that post that you linked to, I ran the script and CHDK release and posted the results in the linked thread.
http://chdk.setepontos.com/index.php?topic=11527.msg119819#msg119819 (http://chdk.setepontos.com/index.php?topic=11527.msg119819#msg119819)
-
Sorry for not getting back, I just enabled email notifications so that I will be able to better keep track of these threads.
for that post that you linked to, I ran the script and CHDK release
...
This is with the script set to take 13 images
Cross posted here : http://chdk.setepontos.com/index.php?topic=11527.msg119821#msg119821 (http://chdk.setepontos.com/index.php?topic=11527.msg119821#msg119821)
-
Sorry for not getting back, I just enabled email notifications so that I will be able to better keep track of these threads.
for that post that you linked to, I ran the script and CHDK release
with the debug enabled, simply taking a raw file after turning the camera on (DNG 1.1)
MEM: 0x53369555
FP: 500
MP: 500
Thanks for testing. Based on this I have increased the pause in CHDK 1.3 and 1.4 to 320 ms.
It would be helpful if you can take a 5+ individual shots, with raw enabled, not using continuous shooting, and note the FP value after each shot.
If normal shooting hits the 320 ms timeout every time, we may need to look for other options.
-
Sorry for the delay,
I did 5 non burst shots though i am not sure if these results are reliable as the value is normally 450-460, but after less than a second, it just changes to 500
Shot 1: FP: 500
Shot 3: FP: 500
Shot 4: FP: 500
Shot 5: FP: 500
Shot 6: FP: 500
though there is something weird with the debug info, after taking an image, for a split second, the FP: value will show a different value other than 500, then go back to 500
Most commonly, FP would be at 450-460 for a split second before jumping to 500. I recorded a video of the behavior.
https://www.youtube.com/watch?v=YA4DwN-ENBg&feature=youtu.be (https://www.youtube.com/watch?v=YA4DwN-ENBg&feature=youtu.be)
The SD card used in the camera, is the PNY Elite Performance 64GB (read speeds are around 88MB/s and write speeds top out at about 60MB/s when used with a USB 3.0 card reader.
I wonder why the FP is lower with the script than with the manual single shot image?
-
Sorry for the delay,
I did 5 non burst shots though i am not sure if these results are reliable as the value is normally 450-460, but after less than a second, it just changes to 500
I think what you are seeing is the value updating while the wait is happening, with the ISO just causing the UI to be refreshed to actually see it. The value is cleared when the raw hook is reached, and really shouldn't change outside of the actual wait.
The SD card used in the camera, is the PNY Elite Performance 64GB (read speeds are around 88MB/s and write speeds top out at about 60MB/s when used with a USB 3.0 card reader.
I wonder why the FP is lower with the script than with the manual single shot image?
This isn't about card speed. CHDK inserts code to capture the raw (the "raw hook") somewhere in some poorly understood Canon code. On some cameras, the Canon firmware hasn't always incremented the file counter by the time the code reaches this point. The "pause for file counter" setting makes the raw hook wait, either until the counter changes, or some timeout. The timeout is needed because the counter might be correct already. In that test build, the timeout is 500, so if FP is 500, it means the counter was already right.
I think what you are seeing is that in normal single shooting, the counter is right pretty much every time. So the code waits 500 ms (during which you can see the FP value change) and then continues. The wait is only consistently needed when shooting fast.
IIRC this matches results I got from someone else in IRC when the problem first came up for this camera.
We could make the pause longer in continuous mode, but that' wouldn't help scripts like this that just holds down shoot_half and clicks shoot full.
The current 1.3 (and 1.4) autobuilds will wait 320 ms, which was longer than any of your script results showed. This should fix the missing raw problem, but may slow down regular shooting, or the first shot in a burst.
Please let me know if you see the missing raw problem with 1.3.
I'll need to think more about other possible solutions or testing.
-
In testing the sx230hs-101b-1.3.0-3910-full build, the raw files still fails to save.
With this build and doing a bunch of continuous shots, it managed to create 2 raw files, though they were really small.
In doing a 4 image burst, it did 2 38KB files
-
In testing the sx230hs-101b-1.3.0-3910-full build, the raw files still fails to save.
Can you confirm that the test build with the FP debugging correctly saves raw every time?
With this build and doing a bunch of continuous shots, it managed to create 2 raw files, though they were really small.
In doing a 4 image burst, it did 2 38KB files
This is very strange. I don't think file counter issue by itself shouldn't be able to cause anything like this. Will need to investigate more.
Were these the only raw files saved, or were some correct?
-
With the 1.2 build where I copied the FP info, all of the raw files saved at their 18.2MB size (no corruption)
For the 1.3 build using the burst shooting (Shutter release button held down), none of the raw files saved properly (only 2 were saved, and they were 38KB in size), but doing non burst shooting, allowed for every image taken, to properly save a raw file
-
With the 1.2 build where I copied the FP info, all of the raw files saved at their 18.2MB size (no corruption)
For the 1.3 build using the burst shooting (Shutter release button held down), none of the raw files saved properly (only 2 were saved, and they were 38KB in size), but doing non burst shooting, allowed for every image taken, to properly save a raw file
Just to be sure, you tested each build with exactly the same method (e.g. continuous mode or script) and settings?
-
Yep, I used continuous mode for both after creating the bad pixel file and enabling raw (DNG 1.1).
The raw issue still happens even if not using DNG.
-
The 38KB seems to be a complete DNG header and thumbnail without any image data :blink:
This is pretty odd, since the file is only opened once, and there aren't really any branches that would skip writing the data after the thumb is written...
-
The 38KB seems to be a complete DNG header and thumbnail without any image data :blink:
This is pretty odd, since the file is only opened once, and there aren't really any branches that would skip writing the data after the thumb is written...
write() tries to take the fileio semaphore too and fails if it cannot (default timeout is 3 seconds). The same semaphore is used by filewritetask. Perhaps it is 'taken' for a longer time in continuous mode.
edit:
I don't see this problem on my ixus115 (r47 just like the sx230) in continuous mode.
-
write() tries to take the fileio semaphore too and fails if it cannot (default timeout is 3 seconds). The same semaphore is used by filewritetask. Perhaps it is 'taken' for a longer time in continuous mode.
Good idea.
I just skimmed though the porting thread (http://chdk.setepontos.com/index.php?topic=6397.650 (http://chdk.setepontos.com/index.php?topic=6397.650)) and was reminded of a bunch problems and loose ends in this port.
It looks like nafraf did a code gen conversion that was never tested or added, so that seems like a good place to start.
There was also the UI memory stuff, which could let as get away from using exmem.
I'll make a test build with code gen and something to check if the write failure was semaphore timeout.
-
Here's a test build including narfrafs codegen re-write and some additional debugging.
For the codegen re-write, please check that it boots, runs, and generally functions as well as current 1.3 builds do.
For the raw issues:
Please try the fasthdr.lua from the previous test, and also try single shots like before.
The misc debug values now has 4 numbers
1) SE: will be non-zero if the semaphore timeout was hit as srsa suggested. Please report if this becomes non-zero when the small dngs are created, or dngs are completely missing.
2) FP: the last wait for the file counter. This now only displays the final value, so it shouldn't change during a single shot
3) MP: The maximum value of FP seen since reboot
4) TC: The number of times the timeout has expired since last reboot. Based on your previous results, this should go up by when you are shooting slow single shots.
Finally, please test the script CHDK/SCRIPTS/TEST/vidtest.lua and post the log.
There will be a line of number across the top of these screen. You can ignore them.
-
(Same issue, burst shooting, and shooting with the script, dos not save the raw file reliably. in a burst shot it was able to save 1 raw file successfully, but 2 others were 38KB, and none of the others were saved)
I have done a few single shot images, and some burst shooting, and ran the script.
The debug into seems a little different, SE seems to stay at 0
For manual single shot images, FP and MP are both at 320. TC seems to just count up
After doing some burst shooting, raw files did not save properly (38KB files), and the debug results were
SE: 9
FP: 50
MP: 320
TC: 3
------------------------------------------------------------------------------------------------
The script did not create a log file. After running it, displays
:77: A/fasthdr.log: error
*** TERMINATED ***
SE: 16
FP: 10
MP: 320
TC: 1
-------------------------
When it failed to make a log file, I went and enabled native Lua calls, and disabled the write protect n the card, then booted using the firmware update option within the playback menu, and it still had the same error.
I recorded a video of the test (long boring video)
https://www.youtube.com/watch?v=4eu203AwR9Y&feature=youtu.be (https://www.youtube.com/watch?v=4eu203AwR9Y&feature=youtu.be)
-
Thanks, that confirms srsa's diagnosis that the corrupt raw is related to the fileio semaphore. This is probably also responsible for the log error.
The single shot results confirm that the counter is normally up to date when shooting slowly or the first shot in a burst.
You don't really need to make videos of these things, if something is unclear I'll ask.
Can you please do this one:
Finally, please test the script CHDK/SCRIPTS/TEST/vidtest.lua and post the log.
If this passes, we can probably enable an option that will avoid the semaphore error. Just run the script with the default settings and post the log.
-
Ran the vid test
-
Thanks. I'll update the port with that result in a bit.
In the meantime, here's a build with the semaphore timeout increased to 10 sec. Please try the the fasthdr script with this.
-
With the build with new build (10 second), the raw issue still happens with raw files either not being saved, or only 38KB files being saved, though it did add a very long delay between images beign taken in burst mode as well as with the HDR script.
(this issue happens with both the script and shooting in continuous mode).
The log file also failed to save (same issue as last time).
-
With the build with new build (10 second), the raw issue still happens with raw files either not being saved, or only 38KB files being saved, though it did add a very long delay between images beign taken in burst mode as well as with the HDR script.
Very strange, this is starting to sound more like a race condition...
Attached is a new test build. With this build, there is an option in the Miscellaneous menu "Enable Unsafe IO". Try fasthdr with this checked.
-
With unsafe IO enabled, the script was able to successfully save all DNG files (script was set to take 13 images, and it created 13 18.2MB DNG files (all readable)
burst mode also successfully saved each raw file properly
The script was still unable to create a log file, as it had the same error as before
I also took this chance to test the exif delay (went as high as 250ms), and the first 2 raw files still get the same exif info that do not match the exif of the jpegs (in terms of shutter speed), though it is easily worked around by simply increasing the number of images taken and just ignoring the first image, so it is not much of an issue.
Edit: here is a download of the jpegs and dng files saved by the camera while running the script :)
https://www.dropbox.com/s/1p0r95u80gpcnlk/13exp%20bracket.zip?dl=0 (https://www.dropbox.com/s/1p0r95u80gpcnlk/13exp%20bracket.zip?dl=0) (152MB)
None of the raw files match the jpegs but other than the first one, the rest do not impact the images, as HDR processing software does not freak out about them, and they are all different (thus no 2 images with the exact same exif info but visually different exposures).
-
With unsafe IO enabled, the script was able to successfully save all DNG files (script was set to take 13 images, and it created 13 18.2MB DNG files (all readable)
burst mode also successfully saved each raw file properly
The script was still unable to create a log file, as it had the same error as before
This is expected, the log file uses script file IO functions which can't use the "unsafe" option.
Note that with unsafe IO enabled, you will probably get occasional crashes. This isn't really a solution, but it at least narrows down the problem. The reason 1.2 didn't suffer this problem is that all IO was done the "unsafe" way, which worked most of the time but crashes occasionally.
I think the next step is to compare capt seq hook code on this camera to other similar generation cameras that don't suffer the problem. msl reported that the sx220 does not have the same problem, although it has some other issues.
I also took this chance to test the exif delay (went as high as 250ms), and the first 2 raw files still get the same exif info that do not match the exif of the jpegs (in terms of shutter speed), though it is easily worked around by simply increasing the number of images taken and just ignoring the first image, so it is not much of an issue.
I don't remember much about the exif problem, I'll have to go back and re-read the old threads.
-
With unsafe IO enabled, the script was able to successfully save all DNG files (script was set to take 13 images, and it created 13 18.2MB DNG files (all readable)
burst mode also successfully saved each raw file properly
The script was still unable to create a log file, as it had the same error as before
This is expected, the log file uses script file IO functions which can't use the "unsafe" option.
Note that with unsafe IO enabled, you will probably get occasional crashes. This isn't really a solution, but it at least narrows down the problem. The reason 1.2 didn't suffer this problem is that all IO was done the "unsafe" way, which worked most of the time but crashes occasionally.
I think the next step is to compare capt seq hook code on this camera to other similar generation cameras that don't suffer the problem. msl reported that the sx220 does not have the same problem, although it has some other issues.
Doesn't this mean that 'is_video_recording()' must be returning true?
Otherwise the semaphore timeout would now be set to 0 (infinite) and the camera would hang trying to write the DNG file data.
Phil.
-
Doesn't this mean that 'is_video_recording()' must be returning true?
Otherwise the semaphore timeout would now be set to 0 (infinite) and the camera would hang trying to write the DNG file data.
In the stock code it would be, but for the test build I overrode the timeout for the non-video case to 200ms. This of course risks the FsIo assert in other places.
-
I think the next step is to compare capt seq hook code on this camera to other similar generation cameras that don't suffer the problem. msl reported that the sx220 does not have the same problem, although it has some other issues.
I spent some time looking at this. Aside from addresses, the canon capt_seq code for sx220 looks identical. The CHDK hook locations are also the same except the nr/darkframe and remote hooks being in a slightly different position. Those shouldn't matter for this problem, especially if those features aren't in use.
Other r47 cameras have the raw hook in a similar location too.
Regarding the sx220, msl said
The fast shooter works very well. With the default settings I get 5 JPGs and 5 DNGs with the same numbers in the file names.
In continuous mode the camera crashes after the first picture with the activated RAW/DNG option. This picture is saved as DNG but not as JPG. Then freezes the camera. A ROMLOG is not available for this crash.
It's probably more of a problem of the memory.
Freezing without a romlog could potentially be a semaphore issue.
edit:
It would be useful to know if the dark frame control and remote hook work on both the sx230 and sx220.
edit:
Thinking about this more... the fact the DNG header is sometimes saved suggests a timing issue, where something prior to the raw hook allows a task that does file IO to run, and it sometimes only gets ahold of the semaphore after the DNG header is written. The DNG code sleeps after writing the header to allow the reversing task to start up.
This in turn suggests that moving the raw hook earlier might avoid the issue, but I didn't see any really obvious places in the preceding function. There are multiple code paths between the remote hook (which if it works must be before the raw is created) and the end of the function.
One option might be to add debug code that just checks the semaphore status at various points.
It's puzzling that sx220 doesn't suffer the same issue, but if it's dependent on timing card speed and shooting settings could be a factor.
-
Here's a new test build. This won't fix the problem, but may provide clues toward a solution.
Attached is a new variant of the fast hdr script. Running it should create two log files in the root of the SD card: fhdrdbg.log and fhdrurt.log
Please run the script with this build, and post both log files. You must have lua native function calls enabled in the misc menu.
-
Sorry for the late reply.
I set it to take 13 exposures, and have attached both log files
-
Sorry for the late reply.
I set it to take 13 exposures, and have attached both log files
Thanks. From a quick look, that should provide some useful data, I'll have to spend some more time with it to see what the next steps are.
If you can do it again with just 3 shots, that would be helpful. The uart log is limited size, so seeing the full sequence would be better.
-
I deleted the log files then ran it again with 3 exposures.
In doing 3 exposures, it saved the 1st and 3rd raw file properly, but did not save any raw file for the 2nd image.
-
Sorry for double post, when running it again, I forgot to delete all of the old images so the image count would start from 1 again
With no images saved in the camera, and no log files, from a fresh boot, I ran the script and attached the log files
This time it only managed to save 1 raw file (the raw file for the first image only. (the 3 jpegs were saved properly)
-
Thanks, that's good data.
From your last log I can see that the canon firmware starts writing the jpeg from the first shot while the raw hook is waiting for the image counter to increment in the second shot. This blocks the raw from getting saved.
Can you check what the review mode is set to in the canon firmware? Please post what it was here, and if it was set to anything other than "off", try with it off.
-
I have the image review disabled since it adds too much of a delay where it spends a few seconds displaying each image after it is taken. Should I enable it? (it would suck to have to keep it enabled (I would prefer to keep it off)
-
Should I enable it? (it would suck to have to keep it enabled (I would prefer to keep it off)
No need.