CHDKPTP - PC Remote Control Performance Analysis - page 18 - RAW Shooting and Processing - CHDK Forum
« previous next »
Pages: 1 ... 13 14 15 16 17 [18] 19 20 21 22 23 ... 47   Go Down

CHDKPTP - PC Remote Control Performance Analysis

  • 465 Replies
  • 139868 Views
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #170 on: 12 / September / 2012, 00:20:11 »
Advertisements
Yeah nice, this is good progress overall // welcome news indeed.  With DNGs it's very long and boring over bench setup USB 1.0 unfortunately .. a couple of hours for the two silly tests. 

I increased the dcimdl entry delay from 200 to 400 ms and it failed at 31 successful myshoots.  Same problem FolderCreate0. 

Quote "Shoot waits for most of the shooting process to complete. In particular, it shouldn't return until CHDK raw is complete, if it's enabled. "

That's what I sensed from observation and see it thoroughly with DNG.  What you told me the 1st time seems contrary.  Also on the S90 the Canon CR2 will be important to handle properly.  And for your behavior description of listdir and mdl?  Could you point me to where you describe those?  I can't find it.

Quote "In this case, you might try only removing the files from within the folders instead of removing the folders themselves."

That was your initial code.  But we moved away from that concept because of the S90 and its peculiar naming scheme if I recall.  Having to do it this will leave empty subdirectories ... do you foresee any conflicts there?  edit: If possible I'd rather try to make dcimdl work in the current configuration.

Quote "since firmware meminfo obviously works on this camera"

It is more important that it work on the S90.  Can you please send me an S90 build with these goodies inside?

=========

If you have 10 minutes ... can you try enabling JPG and DNG on yours and doing a hundred?

Here is what I am using right now .......

Code: [Select]
function dcimdl(destdir,del,pretend)

--local status,mi=con:execwait('return get_meminfo()');
--print("enter dcimdl:",status,serialize(mi))

--sys.sleep(4000)

    if pretend then
        pretend = '-pretend '
    else
        pretend = ''
    end

    local dcim,err=con:listdir('A/DCIM')

--local status,mi=con:execwait('return get_meminfo()');
--print("after listdir:",status,serialize(mi))
--sys.sleep(400)

    if not dcim then
        printf('error listing directory %s\n',tostring(err))
    return
    end
    for i,dname in ipairs(dcim) do
        if dname ~= 'CANONMSC' then
            cli:print_status(cli:execute('mdl '..pretend..' -fmatch=%.[JDC][PNR][G2W]$ DCIM/'..dname..' '..destdir))

--local status,mi=con:execwait('return get_meminfo()');
--print("after mdl:",status,serialize(mi))
--sys.sleep(1000)

            if del then
                --cli:print_status(cli:execute('rm '..pretend..' -nodirs -fmatch=%.[JDC][PNR][G2W]$ DCIM/'..dname))
                cli:print_status(cli:execute('rm '..pretend..' -skip_topdirs DCIM'))

--local status,mi=con:execwait('return get_meminfo()');
--print("after rm:",status,serialize(mi))

end
        end
    end
end

function myshoot(tv, av, sv, destdir)
  cli:execute('shoot -tv='..tv..' -av='..av..' -sv='..sv..'')
  sys.sleep(400)
  dcimdl(destdir, true, false)
end




« Last Edit: 12 / September / 2012, 00:22:42 by SticK »
Logged
*

Offline reyalp

  • ******
  • 14128
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #171 on: 12 / September / 2012, 02:03:27 »
I was able to reproduce the ASSERT!! ReadFDir.c Line 96 error on d10 in a build without cam meminfo, in under 50 shots.

I have not yet been able to reproduce the FolderCreate0 one, but I can't use a post shoot sleep of less than about a second without getting the FsIoNotify one.

Quote from: SticK on 12 / September / 2012, 00:20:11
Yeah nice, this is good progress overall // welcome news indeed.  With DNGs it's very long and boring over bench setup USB 1.0 unfortunately .. a couple of hours for the two silly tests. 
A USB 2.0 computer would seem like a good investment ;)


Quote
Quote "Shoot waits for most of the shooting process to complete. In particular, it shouldn't return until CHDK raw is complete, if it's enabled. "

That's what I sensed from observation and see it thoroughly with DNG.  What you told me the 1st time seems contrary.
Well, it's complicated. CHDK will wait for the CHDK raw code to finish, but that doesn't mean it will have zero impact on whatever follows. So enabling / disabling raw could affect the delay required, but not for the whole several seconds that it takes to save a raw.

Quote
Also on the S90 the Canon CR2 will be important to handle properly.
I have no idea how that interacts with shoot, since I've never had a camera that does Canon native raw.

Quote
  And for your behavior description of listdir and mdl?  Could you point me to where you describe those?  I can't find it.
I linked the post, don't know what more I can do...

Quote
That was your initial code.  But we moved away from that concept because of the S90 and its peculiar naming scheme if I recall.
I don't think so. Since we are using listdir to get the dcim directory names, the camera naming scheme doesn't really matter.

All you have to do to preserve the directories is switch -skip_topdirs to -nodirs in the rm command.  Note that this will also leave the CANONMSC directory, but still delete the file. In general, I'd suggest doing the minimum number of unexpected things to the canon firmware. Deleting files and directories out from under it is not something the designers would have anticipated.

BTW, the rm command is still in the wrong place, as I pointed out here: http://chdk.setepontos.com/index.php?topic=8613.msg90669#msg90669
If you ever ended up with more than one directory, it would remove *all* the files from all the directories after you downloaded the contents of the *first* directory.

Quote
Having to do it this will leave empty subdirectories ... do you foresee any conflicts there?  edit: If possible I'd rather try to make dcimdl work in the current configuration.
*shrug* no idea. I just observed that the task allegedly responsible for the crash deals looks like it has something to do with folders, and we are doing something the Canon firmware wouldn't expect to folders, so it might be worth trying a different way.

Quote
It is more important that it work on the S90.  Can you please send me an S90 build with these goodies inside?
Attached is a build for s90 with cam_meminfo enabled. You can check the meminfo functionality by just doing
=return get_meminfo()
If it returns values and doesn't crash, then it's almost certainly OK.

...
well the forum went down when I first tried to post this, lets try again

Quote
If you have 10 minutes ... can you try enabling JPG and DNG on yours and doing a hundred?
100 shots with jpeg and DNG completed successfully (1.96 gb on a 1gb card, nice ;)). Note this is with a 2 second sleep (because as mentioned above, I get the FsIoNotify error if it's too short), and the live view window not active.
Logged
Don't forget what the H stands for.
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #172 on: 12 / September / 2012, 10:57:28 »
Quote "I was able to reproduce the ASSERT!! ReadFDir.c Line 96 error on d10 in a build without cam meminfo, in under 50 shots."

This is a tough problem.  To be able to reproduce it at arm's length and get statistical consistency with my cameras  is already half-way to solidifying the solution.  Implementing cam meminfo and seeing similar results so far (D10 & SX110) attests to the quality of the solution.  That sounds like even more good news to me.

Quote "A USB 2.0 computer would seem like a good investment"

True.  The instrument controller is USB 2.0 with about a dozen ports.  I cannot use that one for development because it is in use.  The bench laptop is a 10-yr-old 500 MB WinXp Compaq .. an old dependable workhorse.  I did venture out to FutureShop for the back-to-school specials and was disappointed in flimsy junk they're making these days.

Quote "CHDK will wait for the CHDK raw code to finish, but that doesn't mean it will have zero impact on whatever follows"

OK I think I see .. that means if the camera is set to emit only a big Canon JPEG (say 9 MB) and/or a Canon CR2 (9MB) that take a few seconds to save, shoot returns long before the saves are complete, unless CHDK RAW is enabled and Canon raw disabled ... right, more-or-less?  If this entire assembly passes bench tests and the S90 ends up in the instrument, then I will use Canon RAW because it is compressed and transfers faster.  So you can envision a scenario where one shot is 1 MB (JPEG only small format) and the next is almost 20MB, a real possibility.  Thus I would have to make sleep cover the worst case, which could easily be 7 seconds or more, when a 1 MB write is almost instantaneous by comparison.  While imaging I can easily do a few hundred shots in a row so you can see the impact.  Plus ... I don't know yet what effects there are from long exposure times & dark subtracts // yet to be explored, especially on the S90.   This leads me to where I am sure you know by now where I'm heading: if my understanding is right, is there something in your bag of tricks that will make shoot sensitive to the end-of-Canon-save? 

Quote "I have no idea how that interacts with shoot, since I've never had a camera that does Canon native raw."

This is where I can chip in ... will the first line of attack once your ZIP is up on the S90.

Quote "Note that this will also leave the CANONMSC directory"

Yes ... that's it !  Now I recall .. the problem was dangling database handles that made a mess with PLAY mode, recall 3 of 3 with no files , the BIG ?, basically a confused Canon PUP state.  Removing CANONMSC each time results in a clean start -- I can give you assurance of that now.  So yes, we need to do remove the dirs somehow, without getting the file create0 error.

Quote "Deleting files and directories out from under it is not something the designers would have anticipated."

Exactly .. but .. starting with an empty card (or perhaps just DCIM) would have been anticipated hence that I feel is the only real stable starting point supporting complete tree deletion.  I think that when Canon detects the presence of CANONMSC without a CTG file it behaves abnormally.

Quote "All you have to do to preserve the directories is switch -skip_topdirs to -nodirs in the rm command.  Note that this will also leave the CANONMSC directory, but still delete the file"

I will try it again but I am sure to run into the PLAY instability which was eliminated by removing all the directories ... Catch-22?  --- leave the directories and get PLAY problem and but no FileCreat0 (you expect // untested), or remove subdirs get no PLAY problem but have FileCreate0 (present state of affairs).  Hmmm.  Would the lesser of two evils please stand up?

Quote "If you ever ended up with more than one directory, it would remove *all* the files from all the directories after you downloaded the contents of the *first* directory. "

Based on our discussion, this is still in limbo without me doing the tests all over again for a more certain determination of instability.

Quote "and we are doing something the Canon firmware wouldn't expect to folders, so it might be worth trying a different way."

Yeah .. to get out the catch-22.  I will attempt to quantify the behavior better and try a couple of extras that are floating around in my head.

Quote "Attached is a build for s90 with cam_meminfo enabled. You can check the meminfo functionality by just doing =return get_meminfo() If it returns values and doesn't crash, then it's almost certainly OK."

I am *very much* looking forward to this with all fingers crossed!

Quote "1.96 gb on a 1gb card, nice"

Remote camera operation is very powerful and indispensable for many applications I think.  That fact that you're capturing the full resolution (S90) of camera LCD frame buffer is nothing short of amazing.  How this will pan out for me personally depends on the interaction effects in the instrumentation scenario.  You can see how important it is to *completely* bench test and characterize performance.  You can imagine ... if hadn't noticed the random failure and transferred camera to the instrumentation computer prematurely, I would have blamed interaction first and ended up barking up the wrong tree for who-knows-how-long. or worse, giving up on the endeavor altogether.

Quote "Note this is with a 2 second sleep"

Yes, I have to get a handle on the JPG+DNG failure on the SX110, first absolutely, before doing anything else.  But I will test you ZIP on the S90 with meminfo ASAP ... that's just as important to report back.

Nice work.
« Last Edit: 12 / September / 2012, 10:59:35 by SticK »
Logged
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #173 on: 12 / September / 2012, 12:15:51 »
SX110 2s dcimdl delay.  DCIM subdirs delete after each download.  Alas, FAIL after 45 myshoots.

Same FolderCreate0 exception.

Next cam meminfo ZIP on S90 - basic functional test.

Logged
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #174 on: 12 / September / 2012, 12:56:07 »
S90 CAM MEMINFO - good news - it works // one shot test only.

Also the S90 has 300K more free block than SX110 // have a look attached.

Next ---- back to SX110 and try out not-delete-directories ... see if we can get around the FolderCreate0 exception in the huge cargo JPG+RAW.
Logged
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #175 on: 12 / September / 2012, 13:04:33 »
S90 - GREMLIN ... I've noticed this before with previous builds .. when the S90 is being operated by CHDKPTP, I can't access Canon buttons on the camera itself, with the exception of <Alt> and CHDK menus. 

For example, if I press <SET> in REC mode on the camera there is no response when tethered.  If I press <Set> on CHDKPTP however, camera responds and works properly.

On the SX110, it's no problem ... all buttons work as though it were untethered.
Logged
*

Offline reyalp

  • ******
  • 14128
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #176 on: 12 / September / 2012, 13:07:35 »
Quote from: SticK on 12 / September / 2012, 10:57:28
This leads me to where I am sure you know by now where I'm heading: if my understanding is right, is there something in your bag of tricks that will make shoot sensitive to the end-of-Canon-save? 
If we knew how to do that, then don't you think we'd already be doing it ???
Quote
This is where I can chip in ... will the first line of attack once your ZIP is up on the S90.
This is not the same as actually being able to debug it.
Quote
Quote "Note that this will also leave the CANONMSC directory"

Yes ... that's it !  Now I recall .. the problem was dangling database handles that made a mess with PLAY mode, recall 3 of 3 with no files , the BIG ?, basically a confused Canon PUP state.  Removing CANONMSC each time results in a clean start -- I can give you assurance of that now.  So yes, we need to do remove the dirs somehow, without getting the file create0 error.
Huh ? In my experience, removing the files gives you the ?? bad file icon / ghost files, and they go away on the next reboot. I haven't seen any evidence this actually causes any problem. It might, it might not, the answer will only be found by testing.

The problem you had before was IIRC due to saving CHDK raws, which don't correctly use the camera directory naming scheme.
Quote
Quote "Deleting files and directories out from under it is not something the designers would have anticipated."

Exactly .. but .. starting with an empty card (or perhaps just DCIM) would have been anticipated hence that I feel is the only real stable starting point supporting complete tree deletion.  I think that when Canon detects the presence of CANONMSC without a CTG file it behaves abnormally.
There's really no point guessing. Careful, controlled testing is the only thing that matters.
Quote
Quote "All you have to do to preserve the directories is switch -skip_topdirs to -nodirs in the rm command.  Note that this will also leave the CANONMSC directory, but still delete the file"

I will try it again but I am sure to run into the PLAY instability which was eliminated by removing all the directories ... Catch-22?
Speculating is a wast of time, only testing will give you the truth.
Quote
Quote "If you ever ended up with more than one directory, it would remove *all* the files from all the directories after you downloaded the contents of the *first* directory. "

Based on our discussion, this is still in limbo without me doing the tests all over again for a more certain determination of instability.
You seem to be confused. I'm pointing out a simple logic error, unrelated to any of the above issues. The code you are using is wrong.

Here's what your code is doing
Code: [Select]
for each directory in dcim
   download files in directory
   remove all files in all directories under dcim
Do you see the problem ???

My original code looked like this
Code: [Select]
for each directory in dcim
   download files in directory
   remove all files in directory
Logged
Don't forget what the H stands for.
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #177 on: 12 / September / 2012, 15:44:29 »
Quote "If we knew how to do that, then don't you think we'd already be doing it."

Just making sure .... do you have a team there to focus their attention on this?  :)

Quote "This is not the same as actually being able to debug it."

Sure ... that's why I said "chip in" ..... somewhat more than help but a lot less than debug.

Quote "It might, it might not, the answer will only be found by testing. "

Yes I will do all that, systematically, one step at a time.

Quote "removing the files gives you the ?? bad file icon / ghost files, and they go away on the next reboot"

I think in my case it was a hanging CTG file that caused the weird behavior in PLAY mode.  That's why I am willing to revisit your suggestions.  I will post the results when I get there.

Quote "Speculating is a wast of time"

In science speculation is often frowned upon, but, I am one of the few who encourages it.  So long as it is in reasonable context, speculation allows one's mind to explore freely and stumble upon ideas that could provide bases for hypotheses and potentially lead to discoveries that otherwise would never happen.  In effect, it is stepping, often taking a leap, outside the box.  While I would never replace testing when possible, I have been often criticized for my way of thinking so what you're saying is not new to me.

Quote "There's really no point guessing. Careful, controlled testing is the only thing that matters."

That was in part a guess yes, but also and in part empirical data ... so I feel there is good basis for what I said.  As I mentioned, I will revisit all this.  You and I are doing what I would call a "global iteration of top-level features converging on correct general functionality," by attacking specific detailed problems.  Do you like the feel you're beginning to get of remote control?  Personally, I use it (RemoteCapture) any time I get the opportunity.

Quote "You seem to be confused."

Maybe // maybe not.  :)

===============

A miracle?  Listen to this ... I ordered an AC adapter for the S90 from CHN a week ago Monday, and it arrived today.  Ten days is the fastest I received anything from there.  But as is usual, not one of my transactions with CHN resulted in a perfect item | not-a-one.  The adapters for the SX110 were very noisy and would shut down the camera when I open RemoteCapture.  It took two days of electrical modifications to get them to work.  So in effect the entire sum I paid was for the little camera connector.  And this one ... a South American A/C plug!  So now I have to go shopping ... and that's just the beginning. 

So if you don't get new data from me today, at least you'll know why.

Logged
*

Offline SticK

  • *****
  • 779
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #178 on: 12 / September / 2012, 23:57:33 »
The S90 is now AC powered so we can do more extensive tests at ease. I am repeating tests with dcimdl as I last posted, small JPG only.  Here is an ERROR 56 ??  I did not have memory sniff ON, but I think it comes from rm. JPG was transferred.  The system did not crash, but /117___09 was left hanging empty.

In dcimdl, could you describe the error and the possible situations that may have led to it?
Logged
*

Offline reyalp

  • ******
  • 14128
Re: CHDKPTP - PC Remote Control Performance Analysis
« Reply #179 on: 13 / September / 2012, 00:57:56 »
Quote from: SticK on 12 / September / 2012, 23:57:33
In dcimdl, could you describe the error and the possible situations that may have led to it?
Nope. Don't know what it is.

The "no match" in the cmd window means chdku.lua format_exec_error looked for a line number in the error message and didn't find one. 56 is the entire error message.

It could perhaps be related to this lovely thing http://chdk.setepontos.com/index.php?topic=8273.0
Logged
Don't forget what the H stands for.
Pages: 1 ... 13 14 15 16 17 [18] 19 20 21 22 23 ... 47   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal