issues with open and fopen while recording video - page 4 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 2 3 [4] 5 6 7   Go Down

issues with open and fopen while recording video

  • 69 Replies
  • 24862 Views
*

Offline reyalp

  • ******
  • 14080
Re: issues with open() and fopen() while recording video
« Reply #30 on: 04 / November / 2014, 23:24:15 »
Advertisements
Quote from: philmoz on 04 / November / 2014, 06:05:11
What about a user selectable option to allow 'potentially unsafe' IO operations to continue?

If the option is enabled, then instead of failing when TakeSemaphore times out we just revert to the old functionality and call the underlying firmware function without the semaphore protection.

There would be minimal risk of triggering the FsIoNotify function since the firmware is already inside an IO loop (for the video recording), it should not be opening and closing other files.

This would only apply for open/close/write/opendir/closedir, doing it in fopen would trigger a lockup.

I had a quick play with this and it seems to work - the main difference to the 1.2 functionality is that things are slower (due to the 1/2 second timeout in TakeSemaphore). 1/2 second is probably too conservative , I tried it at 200ms and it still seems to work.
Danger of having it too short is it would timeout in the startup fstuff or during jpeg saving and you'd get the FsIoNotify crash. I don't have a good feeling for what a "safe" time would be. I guess it would be possible to record how long is spent waiting for the semaphore.

I had similar thoughts about making it just fall back to the old code if the semaphore timed out, though I hadn't considered making it a user option.

If we found a reliable way to detect video recording, we could just do it in that case, which would get rid of the timeout delay. I guess we could do this now for some cameras.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: issues with open() and fopen() while recording video
« Reply #31 on: 05 / November / 2014, 03:04:19 »
Quote from: reyalp on 04 / November / 2014, 23:24:15
Danger of having it too short is it would timeout in the startup fstuff or during jpeg saving and you'd get the FsIoNotify crash. I don't have a good feeling for what a "safe" time would be. I guess it would be possible to record how long is spent waiting for the semaphore.

I recorded the maximum delay for successful taking of the semaphore, and created a task to try taking the semaphore every 10ms.

This is a worst case scenario, simulating constant IO attempts.

With image and movie files on the SD card the worst startup delay in TakeSemaphore I've seen is 250ms (starting in playback mode on G12).

Unfortunately saving Canon RAW or CHDK RAW/DNG also blocks the semaphore - up to 1.3 seconds on my G12.

What a mess :(

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14080
Re: issues with open() and fopen() while recording video
« Reply #32 on: 06 / November / 2014, 01:29:26 »
Quote from: philmoz on 05 / November / 2014, 03:04:19
Unfortunately saving Canon RAW or CHDK RAW/DNG also blocks the semaphore - up to 1.3 seconds on my G12.
Makes sense since it's held for the entire write() call. This also means it could be a lot longer on some cams, especially if using slow cards.

If we leave script on fopen (borked no matter what) this may not be a huge issue, trying to manipulate the menu while a raw is saving shouldn't be a common case. For CHDK raw, the saving happens in spytask, so you really can't do much.

Script console logging could potentially run into it, since it uses open/write and runs from kbd task.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: issues with open() and fopen() while recording video
« Reply #33 on: 06 / November / 2014, 02:33:23 »
Next idea:

- Add a new (sigh) #define CAM_IS_VIDEO_RECORDING_WORKS, set this on cameras where is_video_recording is ok.
- Set TakeSemaphore timeout value as:
    - CAM_IS_VIDEO_RECORDING_WORKS && is_video_recording() --> 200ms (prevents lockups)
    - CAM_IS_VIDEO_RECORDING_WORKS && !is_video_recording() --> 0 (could be a camera setting)
    - !CAM_IS_VIDEO_RECORDING_WORKS --> 3000ms (prevents lockups, should not fail on RAW/DNG save, CHDK UI may be slow in video)

The user setting to allow 'unsafe' IO in video would only be available if CAM_IS_VIDEO_RECORDING_WORKS is defined, and the TakeSemaphore call timed out when is_video_recording() returned true.

Edit: patch attached.

Phil.
« Last Edit: 06 / November / 2014, 03:58:50 by philmoz »
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14080
issues with open and fopen while recording video
« Reply #34 on: 23 / November / 2014, 20:33:22 »
edit:
Seems this can be replied to if you don't have open() and fopen() in the subject  :blink:

edit:
srsa from pm
Quote
I have only done little testing (without changing platform_camera.h) on one of my cameras - the movie recording lock-up is obviously gone. Since several file operations are affected, how should "real" testing look like? Could it be scripted somehow? I guess one ingredient would be a card with only a small free space left (to reliably stop the test movie recording)...
The v5 patch has a copy&paste bug in unsafe_io_warning().

I tested some on D10 and elph130. It seems to work as expected.

is_video_recording appears to work on both cameras. I tested both with and without the define.

What do we need to be sure is_video_recording works? In my testing, I added movie_status and is_video_recording result to misc OSD values, and verified that is_video_recording was only 1 when recording video and the "busy" stage after you stop recording.

As far as testing goes, I'm not sure. I think the main scenarios are
1) Semaphore or related functions are wrong: In this case, I'd expect an immediate failure. All dryos cams were already using the semaphore for the FsIoNotify crash fix.
2) Semaphore times out in normal operation, chdk fails to handle error return correctly. It would be hard to check every scenario, but realistically the most likely thing people are going to do is adjust settings in the alt menu or run a script. Since the current trunk behavior is wait forever, we should probably have noticed other situations where CHDK waits for a long time.
« Last Edit: 23 / November / 2014, 22:24:35 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: issues with open and fopen while recording video
« Reply #35 on: 24 / November / 2014, 05:59:14 »
Updated patch:
- fix copy/paste error in gui.c
- make capture of file IO stats conditionally compiled into build (add OPT_FILEIO_STATS=1 to localbuildconf.inc).

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: issues with open and fopen while recording video
« Reply #36 on: 24 / November / 2014, 16:41:37 »
Did not test the new patch yet, but I have some text left from my unsent post.

Quote from: srsa_4c on 03 / November / 2014, 19:32:16
I could perhaps add the file access semaphore init function to the sigfinder's list and use a script (I do have such a script) to mass-disassemble its content for visual inspection.
Added. Here's the disassembly of the function (for all supported Vx fw). My script is not able to parse stubs_entry.S yet, so the disassembly only shows 2 named functions.

While experimenting with scripts in movie mode I've noticed that the following scenario doesn't work:
- start cam
- load script
- start movie recording
- run script (will not start, module loading error instead)
... because the interpreter (Lua) will only be loaded when the script is actually started. I'm not sure if this is considered to be a problem.

@reyalp
Thanks for finding out the cause.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: issues with open and fopen while recording video
« Reply #37 on: 24 / November / 2014, 16:51:32 »
Quote from: srsa_4c on 24 / November / 2014, 16:41:37
While experimenting with scripts in movie mode I've noticed that the following scenario doesn't work:
- start cam
- load script
- start movie recording
- run script (will not start, module loading error instead)
... because the interpreter (Lua) will only be loaded when the script is actually started. I'm not sure if this is considered to be a problem.

With the latest patch, setting the 'allow unsafe IO' flag should let the module load:
- if the camera has a working 'is_video_recording' function,
- and CAM_IS_VID_REC_WORKS is defined.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: issues with open and fopen while recording video
« Reply #38 on: 26 / November / 2014, 16:53:09 »
Quote from: philmoz on 24 / November / 2014, 16:51:32
With the latest patch, setting the 'allow unsafe IO' flag should let the module load:
- if the camera has a working 'is_video_recording' function,
- and CAM_IS_VID_REC_WORKS is defined.
Confirmed working on the sx100 (DryOS r23) and s80 (Vx). No lockups on these cameras with or without 'CAM_IS_VID_REC_WORKS', with or without 'allow unsafe IO' (tested while recording video). Except for, of course, event procedures that try to write to files. Scripts fail to open files while recording.

edit:
Same result on a460 (late Vx).
« Last Edit: 26 / November / 2014, 17:59:42 by srsa_4c »
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: issues with open and fopen while recording video
« Reply #39 on: 26 / November / 2014, 20:44:48 »
Quote from: srsa_4c on 26 / November / 2014, 16:53:09
Quote from: philmoz on 24 / November / 2014, 16:51:32
With the latest patch, setting the 'allow unsafe IO' flag should let the module load:
- if the camera has a working 'is_video_recording' function,
- and CAM_IS_VID_REC_WORKS is defined.
Confirmed working on the sx100 (DryOS r23) and s80 (Vx). No lockups on these cameras with or without 'CAM_IS_VID_REC_WORKS', with or without 'allow unsafe IO' (tested while recording video). Except for, of course, event procedures that try to write to files. Scripts fail to open files while recording.

edit:
Same result on a460 (late Vx).

Are you saying that CAM_IS_VIDEO_REC_WORKS + 'allow unsafe IO' doesn't work (e.g. can't start a script while recording video)?

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
Pages: 1 2 3 [4] 5 6 7   Go Up
« previous next »
 

Related Topics