issues with open and fopen while recording video

srsa_4c

4451

Re: issues with open and fopen while recording video

« Reply #40 on: 26 / November / 2014, 21:12:54 »

Advertisements

Quote from: philmoz on 26 / November / 2014, 20:44:48

Are you saying that CAM_IS_VIDEO_REC_WORKS + 'allow unsafe IO' doesn't work (e.g. can't start a script while recording video)?

No, I'm saying the opposite. I was able to start scripts while recording was in progress.
"Scripts fail to open files while recording" means that scripts like llibtst.lua can't open files (and therefore stop with an error) when started during video recording.

Since this isn't the first time I'm being misunderstood, may I ask which part of my post caused the confusion?

Logged

philmoz

3450

Re: issues with open and fopen while recording video

« Reply #41 on: 26 / November / 2014, 21:24:59 »

Quote from: srsa_4c on 26 / November / 2014, 21:12:54

Quote from: philmoz on 26 / November / 2014, 20:44:48
Are you saying that CAM_IS_VIDEO_REC_WORKS + 'allow unsafe IO' doesn't work (e.g. can't start a script while recording video)?

No, I'm saying the opposite. I was able to start scripts while recording was in progress.
"Scripts fail to open files while recording" means that scripts like llibtst.lua can't open files (and therefore stop with an error) when started during video recording.

Since this isn't the first time I'm being misunderstood, may I ask which part of my post caused the confusion?

Thanks for clarifying. The issue with file IO in Lua is because it use fopen not open, 'allow unsafe IO' won't help here because fopen uses the semaphore internally, so we can't allow it to continue or it will lockup.

I read the original post as though you were seeing the same behaviour all the time, regardless of how you set CAM_IS_VIDEO_REC_WORKS and 'allow unsafe IO' - my bad

So it's probably safe to push this to SVN and work on adding CAM_IS_VIDEO_REC_WORKS where possible.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14117

Re: issues with open and fopen while recording video

« Reply #42 on: 29 / November / 2014, 22:44:13 »

Quote from: philmoz on 26 / November / 2014, 21:24:59

So it's probably safe to push this to SVN and work on adding CAM_IS_VIDEO_REC_WORKS where possible.

I agree. I've tried the latest patch on my cams and it seems OK.

Logged

Don't forget what the H stands for.

msl

1280
A720 IS, SX220 HS 1.01a

Re: issues with open and fopen while recording video

« Reply #43 on: 30 / November / 2014, 05:30:33 »

Rev. 3760, 3761, 3762

Can someone explain the backgrounds for non programmers. I know the general problems with the video recording.

Why is this new function only enabled for some cameras? What about the other cameras? What needs to be tested?

Sorry, but I think, it is not the best way, to introduce new features without any general description - especially new menu functions.

"Enable Unsafe IO?" is understandable for programmers and technically interested people, but not for all photographers.

We should make CHDK again understandable for all! Otherwise CHDK will more and more only a tool for nerds.

msl

Logged

CHDK-DE: CHDK-DE links

philmoz

3450

Re: issues with open and fopen while recording video

« Reply #44 on: 30 / November / 2014, 05:52:28 »

Quote from: msl on 30 / November / 2014, 05:30:33

Rev. 3760, 3761, 3762

Can someone explain the backgrounds for non programmers. I know the general problems with the video recording.

Why is this new function only enabled for some cameras? What about the other cameras? What needs to be tested?

Sorry, but I think, it is not the best way, to introduce new features without any general description - especially new menu functions.

"Enable Unsafe IO?" is understandable for programmers and technically interested people, but not for all photographers.

We should make CHDK again understandable for all! Otherwise CHDK will more and more only a tool for nerds.

msl

In summary...

This is a work-in-progress to try and find a reasonable way to solve the lockup during video recording issue.

The problem has existed for a long time; but was made worse after the change to use the Canon file IO semaphore to protect the CHDK IO functions. Video recording takes the semaphore and doesn't release it until recording stops. Any attempt to do IO in CHDK would stall trying to take the same semaphore - as would calling the fopen firmware function (e.g. from Lua script).

The 'fix' adds a timeout so the attempt to take the semaphore can fail without getting the semaphore. In this case the IO operation will fail; but CHDK will not lock up.

Since the problem only occurs during video recording, then ideally the fix should only be active at the same time. Otherwise a slightly longer IO operation (e.g. RAW save) could trigger the CHDK semaphore check to fail which would be an undesirable change in behaviour.

The problem is that the 'is_video_recording()' function does not work reliably on all cameras - hence the new #define to identify cameras with a working 'is_video_recording()' function. Without this the code falls back to a long timeout (3 secs). Enabling it for other cameras requires testing if 'is_video_recording()' works correctly.

The 'allow unsafe IO' option was something I added to let the open/close/read/write code continue even if the semaphore check fails. This is similar to what CHDK used to do with file IO. It won't help with IO in Lua scripts (which uses fopen); but will, for example, allow a script to be started during video recording. If this proves unstable then it may get removed.

So this is currently still in the 'nerd' tools realm and may still change

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14117

Re: issues with open and fopen while recording video

« Reply #45 on: 30 / November / 2014, 17:08:42 »

@msl
I agree this isn't ideal, but I don't have a better solution, and this is a big improvement on the previous behavior.

Having CHDK lock up completely if you tried to do IO (use alt mode) in 1.3 was an unacceptable step backward from 1.2. This change gets us from "extremely frustrating + possible data loss" to "annoying"

I'm not fond of the "Enable unsafe IO" option either, but again, I don't have a better solution. Enabling it by default would risk random crashes, but for some people the ability to do file IO while recording might warrant that, and it was possible in 1.2

CHDK is a hack, and is always going to have some weird hacky bits.

Quote

Why is this new function only enabled for some cameras?

The basic workaround is enabled for all cameras, but without knowing if is_video_recording works, the timeout must be annoyingly long.

Quote

What about the other cameras? What needs to be tested?

See this post and the following one http://chdk.setepontos.com/index.php?topic=9986.msg113394#msg113394

We should identify the cameras where is_video_recording is broken. This means going through the possible capture modes and checking if it's on without a video actually being recorded. movie_digest mode appears to be a likely culprit.

You should be able to tell if it's wrong by seeing whether the bitrate display appears, but I will try to make a test script.

Logged

Don't forget what the H stands for.

reyalp

14117

Re: issues with open and fopen while recording video

« Reply #46 on: 01 / December / 2014, 00:13:07 »

Here's a test script. However, I don't have any of the misbehaving cameras, so I don't know if it actually works.

This requires the get_video_recording script function, which I added in trunk 3766.

The test has two parts:
1) mode test
This switches through all the available modes (like setmode.lua) and check whether get_video_recording is true. If I understood srsa's comment in http://chdk.setepontos.com/index.php?topic=9986.msg113419#msg113419 video_recording should be true in movie digest mode, but I'm not clear if just switching to the mode is enough, or you need to shoot once?

2) record test
Try to start recording a video, and verify that video_recording becomes true. Then try to stop, and verify that it goes to false.

For both tests, the value of movie_status is logged.

The results are logged recorded in A/vidtest.log

The script has some options. The defaults should be OK on most cameras.

video start: This controls how the script tries to start video. 'auto' tries uses VIDEO_STD with shoot_full if it is a valid mode, and otherwise will try to use the VIDEO button without changing mode. 'shoot' forces the first behavior and 'vid_btn' forces the second. 'skip' to skip this test.

mode change delay: The time it waits after mode switches. IIRC some cameras need a delay in setmode.lua, if it's too short you will probably see failures in the modes test.

modes test: uncheck this to skip the modes test.

vidtest.lua (4.19 kB - downloaded 20 times.)

Logged

Don't forget what the H stands for.

msl

1280
A720 IS, SX220 HS 1.01a

Re: issues with open and fopen while recording video

« Reply #47 on: 01 / December / 2014, 07:16:11 »

Thanks for the extensive explanations. And sorry for the 'nerd'. I was a little bit frustrated.

And here are some test results:

A720
The test script (default parameters) runs without problem under the recent trunk version - see log file.

SX220
This camera makes some problems. With the default script parameters and the recent trunk version the camera crashes after starting video recording. The mode test was ok.

Code: [Select]

ASSERT!! FocusLensController.c Line 708
Occured Time  2014:12:01 12:00:32
Task ID: 17301544
Task name: CaptSeqTask
SP: 0x00390F74
StackDump:
...

The changing of the parameter 'video start' to 'video button' helped. A video was stored. But the script says: "open log failed". Then I compiled a CHDK version with the option: #define CAM_IS_VID_REC_WORKS. Success - a log file was written.

msl

vidtest_a720.log (1.65 kB - downloaded 14 times.)

vidtest_sx220.log (2.63 kB - downloaded 14 times.)

Logged

CHDK-DE: CHDK-DE links

reyalp

14117

Re: issues with open and fopen while recording video

« Reply #48 on: 01 / December / 2014, 22:19:09 »

Quote from: msl on 01 / December / 2014, 07:16:11

Thanks for the extensive explanations. And sorry for the 'nerd'. I was a little bit frustrated.

I'm glad to have contributors who care enough to be frustrated

Quote

With the default script parameters and the recent trunk version the camera crashes after starting video recording.

The changing of the parameter 'video start' to 'video button' helped. A video was stored.

This is quite strange. The normal mode switches to VIDEO_STD and then presses shoot_full. I'm not sure why that would cause a focus controller assert.

Quote

But the script says: "open log failed". Then I compiled a CHDK version with the option: #define CAM_IS_VID_REC_WORKS. Success - a log file was written.

This is also strange. Open log failed would probably mean the semaphore was still in use. Maybe it stays that way some time after movie_status changes, but CAM_IS_VID_REC_WORKS should make a shorter timeout

I'll have to think about this more.

Anyway, the log helpful. Since it appears movie_status it updates just by switching mode, the mode test should catch cameras like this if they have MOVIE_DIGEST set up incorrectly. On SX220, it looks like it's correct.

Logged

Don't forget what the H stands for.

msl

1280
A720 IS, SX220 HS 1.01a

Re: issues with open and fopen while recording video

« Reply #49 on: 02 / December / 2014, 08:43:48 »

Quote from: reyalp on 01 / December / 2014, 22:19:09

This is quite strange. The normal mode switches to VIDEO_STD and then presses shoot_full. I'm not sure why that would cause a focus controller assert.

Ok, the crash was my bad. I had activated the sd override for other tests. That explains the focus controller assert. Without the override the camera does not crash.

'press ("shoot_full")' always makes a photo, also in video mode. That is the problem with the auto option. I think, the logic for the auto detection isn't quite right. A better solution might be:

Code: [Select]

-- line 87:
if capmode.valid('VIDEO_STD') and get_video_button() == 0 then

Btw., the SX220 is a little diva. CHDK & MOVIE_DIGEST were long time a problem. The camera crashed when this mode was selected. Only since the rework in rev. 3614 (thx rudi) MOVIE_DIGEST works fine.

msl

Logged

CHDK-DE: CHDK-DE links