CHDKPTP - PC Remote Control Performance Analysis - page 19 - RAW Shooting and Processing

waterwingz

12138

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #180 on: 13 / September / 2012, 01:07:50 »

Advertisements

Quote from: reyalp on 13 / September / 2012, 00:57:56

It could perhaps be related to this lovely thing http://chdk.setepontos.com/index.php?topic=8273.0

Please .. just shoot me now ?

Logged

Ported : A1200 SD940 G10 Powershot N G16

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #181 on: 13 / September / 2012, 01:32:05 »

Yeah I read the entire thread. CHN power supplies can be a real problem I can assure you. You have put a dummy load and scope them down and polish up the electronics, before even plugging one into your precious hardware. I have had a case where switcher spikes exceeded the input rating of the camera. I have had cases where PS noise caused disturbingly visible noise on the display. Buyer beware // terrible stuff out there.

Whether power supplies were cause the problem for Alarik or not, I don't know, but, he should have used batteries to rule in/out.

As for our case, I hope it is not so lovely. The S90 has a very different directory scheme than the SX110, thus I am pretty sure now that was the source of the last error. Hence I am modifying dcimdl, my first programming task with this endeavor. That's where we begin the uphill climb. Let us see what happens.

Logged

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #182 on: 13 / September / 2012, 01:37:31 »

QUESTION .. If I have DNG enabled, which one gets saved first, JPG or DNG?

Logged

reyalp

14079

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #183 on: 13 / September / 2012, 02:11:43 »

Quote from: SticK on 13 / September / 2012, 01:37:31

QUESTION .. If I have DNG enabled, which one gets saved first, JPG or DNG?

DNG. All CHDK raw processing happens before the Canon jpeg process starts.

Quote

As for our case, I hope it is not so lovely. The S90 has a very different directory scheme than the SX110, thus I am pretty sure now that was the source of the last error.

As far as I know, format_exec_error is the only thing that would display the "no match" string, and it should only be called with an "error" type script message from the camera. It's called in exactly two places in con_methods:exec. An error type message is generated camera side Lua encounters a compile or runtime error. In general, there would be a normal lua message, something like " :80: attempt to call global 'foo' (a nil value)"

Logged

Don't forget what the H stands for.

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #184 on: 13 / September / 2012, 02:46:34 »

Hmmmm? When I enable DNG, do you create a directory /117CANON if not existent, or if it exists do you deposit the DNG in it?

Logged

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #185 on: 13 / September / 2012, 11:22:35 »

Re: http://chdk.setepontos.com/index.php?topic=8613.msg90735#msg90735

S90: setting: Canon CR2 10 MB & JPEG 2 MB, dark subt ON. Beginning grass-roots tests.

I have dcimdl delay set to 200 ms in order to force errors more frequently to a handle on what's happening. I got the (first ever here) FsIoNotify.c error (during listdir in my case) which I think you got above with your delay under 1s. This could related to the S90 being similar to yours (electronics/FW). It happens much more frequently if I alternate myshoots at 1/8s, 8s, 1/8s, 8s etc. Next is to increase delay to 1s and then 2s and see what happens.

Attached. Is this what you got to be sure?

In the meantime, let us discuss this:

The HellFire question ..... why does this code work "perfectly" (random FsIoNotify notwithstanding) ?? With no CHDK files enabled, according to your function return time description, it should be making a mess of my files|directories and downloads ... and yet ... I have successfully downloaded up to 8 1/8s myshoots (16 Canon CR2&JPG files over slow USB 1.0 total of 12MB, each requiring about 4s save after the shot has been processed ... hmmm), and each time deleting directories clean. This (very desirable) behavior does not jive with your description: in other words, the function calls in dcimdl appear to hold until completed, even observing CHDK activity flicker responses on the camera LCD. ?? ?? ?? Strange.

Code: [Select]

function dcimdl(destdir,del,pretend) 
    if pretend then
        pretend = '-pretend '
    else
        pretend = ''
    end
    local dcim,err=con:listdir('A/DCIM')
    if not dcim then
        printf('error listing directory %s\n',tostring(err))
    	return
    end
    for i,dname in ipairs(dcim) do
        if dname ~= 'CANONMSC' then
            cli:print_status(cli:execute('mdl '..pretend..' -fmatch=%.[JDC][PNR][G2W]$ DCIM/'..dname..' '..destdir))
        end
    end
	if del then
		cli:print_status(cli:execute('rm '..pretend..' -skip_topdirs DCIM'))
	end
end

function myshoot(tv, av, sv, destdir)
  cli:execute('shoot -tv='..tv..' -av='..av..' -sv='..sv..'')
  sys.sleep(200)
  dcimdl(destdir, true, false)
end

My (chip-in) take at this stage? There is a (perhaps lovely?) random event that interferes with any of the calls in dcimdl. It does not interfere with shoot. If this problem were related to unexpected directory states as you had proposed, then failure should be consistent and repeatable. Very strange. Please enlighten me.

ROMLOG_CC S90 Canon CR2 and JPG FAIL after 8 myshoots.LOG (4.13 kB - downloaded 32 times.)

Logged

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #186 on: 13 / September / 2012, 12:37:12 »

S90 TEST Canon CR2 & JPG. I kept 200ms dcimdl delay to force more frequent errors and randomly shot 1/8s ,1/4s, 4, and 8s to introduce more file size variability (see attached). This time I hit a max of 32 successful downloads (64 files) this time before FAIL.

The last two FAILs were FsIoNotify at listdir. This one is MotionVector at listdir. Note that despite my effort to increase FAIL frequency with 200ms and random shutter speeds, it took an hour to get this one.

Could we try this? ...... not guaranteed, but please send me a version of listdir peppered with debug code you feel might be good to narrow down the trap.

NEXT TEST : 2s dcimdl delay, otherwise same settings and methods.

[59] S90 200ms dcimdl listdir FAIL after 32 good myshoots [Canon CR2&JPG].PNG (33.82 kB, 701x504 - viewed 77 times.)

ROMLOG_59 S90.LOG (3.38 kB - downloaded 34 times.)

Logged

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #187 on: 13 / September / 2012, 14:35:33 »

S90 TEST Canon CR2 & JPG, 2s dcimdl delay

Result: 100 myshoots all 1/8s (200 files downloaded). NO FAIL, test terminated by me.

This looks like it agrees with your camera tests with DNG despite the fact you don't have Canon RAW.

NEXT TEST: I will run this n=2000 times, which is about 1-1/2 hrs.

Code: [Select]

function loop_myshoots(n)
	for i=1,n do 
		myshoot(1/8,8,200,"C:\\CANON_S90\\")
		sys.sleep(1000)
	end
end

When you have the time .... can you please give me the guidelines for error code retruns from dcimdl so that I can break from this loop on error ref:my code 2 posts ago? For now I will just let it go and hope there is no error.

Logged

reyalp

14079

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #188 on: 13 / September / 2012, 16:51:16 »

Quote from: SticK on 13 / September / 2012, 02:46:34

Hmmmm? When I enable DNG, do you create a directory /117CANON if not existent, or if it exists do you deposit the DNG in it?

As explained earlier by Phil, the S90 does not correctly implement the folder Canon firmware folder naming scheme for this camera. I'm not sure exactly what it will do in this case, but it's a fair bet it will be <some number>CANON. If you don't check "raw in dir with jpeg" then it might be 100CANON consistently.

Quote

When you have the time .... can you please give me the guidelines for error code retruns from dcimdl so that I can break from this loop on error ref:my code 2 posts ago? For now I will just let it go and hope there is no error.

Don't understand. What specific error are you asking about ? dcimdl doesn't have any "error code returns" ?

If you are talking about the "ERROR 56" I already explained that isn't an intentional error code. If you want to know exactly where in chdkptp side code it was detected, you can replace the

Code: [Select]

		print('no match '..errmsg)

in lua/chdku.lua format_exec_error with

Code: [Select]

		error('no match '..errmsg)

This will give you a stack trace and abort the function call. This is unlikely to provide much insight beyond which command filed though, because it's a camera side error being reported.

If you are talking about the FSIoNotify.c assert, I believe that is completely explained.

If you are talking about some other error, I have no idea what it is.

Quote

The HellFire question ..... why does this code work "perfectly" (random FsIoNotify notwithstanding) ?? With no CHDK files enabled, according to your function return time description, it should be making a mess of my files|directories and downloads ...

Huh ? I really have no idea what you are talking about here. What would "make a mess of" the downloads ? What "function return time description"

If you could reference the specific post (by link or Reply #nnn) maybe it would be clearer.

Quote

and yet ... I have successfully downloaded up to 8 1/8s myshoots (16 Canon CR2&JPG files over slow USB 1.0 total of 12MB, each requiring about 4s save after the shot has been processed ... hmmm), and each time deleting directories clean. This (very desirable) behavior does not jive with your description: in other words, the function calls in dcimdl appear to hold until completed, even observing CHDK activity flicker responses on the camera LCD. ?? ?? ?? Strange.

Again, I can't make any sense of this. I told you the functions in dcimdl don't return until they are complete. I don't understand what description this is supposed to not jive with

Logged

Don't forget what the H stands for.

SticK

779

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #189 on: 14 / September / 2012, 01:17:15 »

S90 TEST: 2s dcimdl delay, JPG+CanonCR2, 800 myshoots 1/8s PASSED, test terminated by me.

This is very encouraging but statistically not guaranteed. That is, unless the FAIL mechanism and the time (arbitrary 2s at present) and reason for the dcimdl delay are fully understood, I consider this unreliable (a real hack). And since CHDK is already a hack, we want to avoid hack-on-hack as much as possible.

Discussion

Quote "the S90 does not correctly implement the folder Canon firmware folder naming scheme for this camera. "

That shouldn't be a problem any more .. the solution I am in the process of testing is supposed to be generic. When I've got it understood, worked out and tuned, it should work on the SX110 & D10 w/o mods.

Quote "The commands in dcimdl will only return when they are actually complete."

Yes, I see that works OK.

Quote "What would "make a mess of" the downloads ? What "function return time description""

Here is your answer and what I was referring to >>> Quote "Shoot is a special case, because that's the camera firmware opening and writing files in ways that we aren't in direct control of."

Hence hypothesis .... I am myshooting simultaneous 2.5M JPGs and 10M CanonCR2s, which I measured to take ~4 seconds to write after image acquisition, much *less* than the 2s dcimdl and now the 1s dcimdl delays. If shoot did not wait for Canon write complete, the result would be a disastrous mess. That's my drift ..... because 2s < 4s and 1s << 4s (no FAIL under 100 myshoots), shoot is in fact waiting for completion to a large extent (welcome news). The CHDK behavior response I see in the camera LCD LL corner confirms the test results. What I am experiencing here ....... is shoot *only*needs a tail-end delay so listdir does not error out. From what I am seeing, that tail-end delay is completely independent of image file size and what image crunching Canon FW does (barrel distortion on JPEG, dark subt, bad pixel removal, and save to SD). Therefore the delay can be determined and for the S90 might be small (likely less than 1 second), and, I speculate it can be fixed as one value for all of the latest (Digic 3+) cameras.

Therefore next objective: find the minimum dcimdl delay that is reliable. It is known that 200 ms dcimdl causes errors in listdir under 50 myshoots. 2s is good to 800+ myshoots. By binary subdivision, determine the cut-off delay where errors no longer occur under 50 myshoots (ie PASS to 100 myshoots). Then, increase delay by 20% and test 1000 myshoots (the all-statistically-conclusive test, 10GB overnight). If it does not fail, then the delay value can be considered reliable and the solution not a hack. The same test will be done on the SX110, and the delay may need to be tuned. The standardized delay then will be the longer of the two.

Test: 1s dcimdl delay, 100 myshoots PASSED, test terminated by me.

NEXT TEST 400ms dcimdl delay, 100 myshoots

Logged