Erase CRW/CR2/DNG files without corresponding JPG

[GrAnd]

I bet the next working address is 0x0200

Look at the MPOPUP_MASK 0x00FF and MPOPUP_CANCEL 0x0100...
So, there is no gap anymore for adding extra menu items. At least next time these constants should be changed a bit.

[rocstar]

I tried the build for the SD850 and it did not work for me.  I formatted the card several times, and try autoboot as well as manual CHDK boot.  The blue LED turns on, and the screen goes blank and locks up hard.  Requires battery pull.

[wontolla]

@ PhyrePhox and rockstar
Thanks for testing and sorry for the late response. I have many things to do lately. You know, the same old story: my sister's racoon ate my card reader and my aunt broke her ankle trying to catch it.

@ PhyrePhox
The camera just dies? Darn, this is a huge step backwards; it already worked in the A560! May be it doesn't work with a lot of files (160 in your case). When I did my tests I always tried with a few only. Please try this new build both with few and lots of files if you can.

@rocstar
Sorry about the problems caused. I don't understand, it works flawlessly in my S3. I hope you dare to try this again, may be first try to erase RAWs in file browser list only, to see if that works, that's a simple loop. Because erase in one folder and erase in DCIM folder takes 2 and 3 nested loops respectively.

It seems I will have to "go back to formula" grrrrrrrrrrr!!!


New builds for people to try (based on Allbest's 48). Please post here if it works or not in your camera. Remember these builds are for testing purposes. Do not use in the field or with valuable pictures. For details on how it works see Reply #31.

allbest-a460-100d-50.zip - 0.22MB
allbest-a5xx.zip - 1.32MB
allbest-a6xx.zip - 1.18MB
allbest-a7xx.zip - 0.52MB
allbest-g7.zip - 0.89MB
allbest-ixus55x.zip - 0.43MB
allbest-ixus65_sd630-100a-50.zip - 0.22MB
allbest-ixus70x.zip - 0.65MB
allbest-ixus700x.zip - 0.44MB
allbest-ixus8xx.zip - 0.73MB
allbest-ixus950_sd850-100c-50.zip - 0.22MB
allbest-s2isx.zip - 0.66MB
allbest-s3is-100a-50.zip - 0.22MB updated 31 March
   S3 diskboot.bin ps.fir In case you can't unzip previous file


Who the heck has been porting CHDK for IXUS? It's getting harder to upload builds for all cameras!

[rocstar]

Wontolla,
I tried the a 550, and it worked great!
The SD850 still has the same problem.  Just booting the firmware locks up the camera hard and I have to pull the battery to get it back.  The blue print LED turns on steady.  That's the only thing it does.

[obitus]

Hello,

I downloaded the file for the S3IS - but I cannot unpack the zip file.
The a5xx.zip works though, I just tested.
Is it possible that the S3IS file is corrupted?

I'd really like to test the release - thanks for the great work!

obitus

[wontolla]

@rockstar
Thanks for testing. Well, now we have success with S3 and A550.

Just booting the firmware locks up the camera hard

So, the problem is when loading the Purge RAW build? When turning on the camera?
Or is it when trying to erase some RAW files in File Browser?
Have you ever build CHDK in your computer? To find the problem in the SD850, one way to go is you make your own build using my diff file.

@obitus
Weird, I just downloaded S3 and I can unpack it. Anyways, I uploaded a fresh zip file and also the "fir" and "bin" files just in case. Thanks for testing.

[obitus]

Hello wontolla,

I was now able to download and extract the S3 file.
I have 1 JPG and 1 CRW left with their original names, img_0004.jpg and img_0004.crw
I went to the 100CANON folder, selected "Purge RAW" - and the S3 locked and reloaded, the files still being there.
No matter what is in the directory, my S3 always reloads.
Do you need more info about my S3, settings or anything?

[Steveo]

I have an A570IS.  This is a superb feature and I now use it constantly.  I would like to echo obitus' comments above.. but it's weird...

First tried it all options/levels to try it worked, and the easiest was to highlight DCIM folder and purge all.  However, after a day of doing this, it stopped working.  Now as described by obitus, when I try it, the camera locks up (screen image is updated with what the lens sees, but no menu/buttons work.  After couple of seconds the camera goes blank, with lens out if out etc.  A power up restores camera, but the RAW's are still there.

It seems to work if I do the procedure on the 100CANON folder (or whatever the last folder is called before the images).  But it is annoying that it did work and now it's locking the camera up.

Does the code log all images deleted or have a limited buffer or something?  It seems weird it worked perfect for a set amount of time - i'd say I deleted 30-50 before it stopped working, but that is a rough guess if it helps!

Steveo

[wontolla]

Thanks guys for your input.

I made several tests last night. Given the weird behaviour I thought the problem was when the folder had too many files and the memory of the camera couldn't cope with it. So I tested with 200 photos and erasing 100 random JPG files. And it worked like a charm! It took like one minute but no hanging.
But then I did the same 200 photos but erasing only one JPG and the camera frizzed.

I will try to change the searching method (may be several partial searches instead of one big loop), I still think there is a problem with the camera's memory but unfortunately I don't understand how things work at low level so may be another developer has a clue. May be there is a way to know state of the memory, if it is full etc.

I'll let you know when I have a new build.
Any clues developers?

This is the Purge RAW function code, three main parts:
1.- is executed when erasing everything in DCIM folder
2.- is executed when a Canon folder is selected
3.- is executed when you call the function in the file list

Code: C

1. //-------------------------------------------------------------------
2. static void fselect_purge_cb(unsigned int btn) {
3.  
4.     DIR             *d,  *d2,  *d3,  *d4;
5.     struct dirent   *de, *de2, *de3, *de4;
6.     struct fitem    *ptr, *ptr2;
7.     char            sub_dir[20], sub_dir_search[20];
8.     char            selected_item[256];
9.     int             i, found=0;
10.  
11.     if (btn==MBOX_BTN_YES) {
12.         //If selected folder is DCIM (this is to purge all RAW files in any Canon folder)
13.         if (selected->name[0] == 'D' && selected->name[1] == 'C' && selected->name[2] == 'I' && selected->name[3] == 'M') {
14.             sprintf(current_dir+strlen(current_dir), "/%s", selected->name);
15.             d=opendir(current_dir);
16.             while ((de=readdir(d)) != NULL) {//Loop to find all Canon folders
17.                 if (de->name[0] != '.' && de->name[1] != '.') {//If item is not UpDir
18.                     sprintf(sub_dir, "%s/%s", current_dir, de->name);
19.                     d2=opendir(sub_dir);
20.                     while ((de2=readdir(d2)) != NULL) {//Loop to find all the RAW files inside a Canon folder
21.                         if (de2->name[0] == 'C' || de2->name[9] == 'C') {//If file is RAW (Either CRW/CR2 prefix or file extension)
22.                             d3=opendir(current_dir);
23.                             while ((de3=readdir(d3)) != NULL) {//Loop to find all Canon folders
24.                                 if (de3->name[0] != '.' && de3->name[1] != '.') {//If item is not UpDir
25.                                     sprintf(sub_dir_search, "%s/%s", current_dir, de3->name);
26.                                     d4=opendir(sub_dir_search);
27.                                     while ((de4=readdir(d4)) != NULL) {//Loop to find a corresponding JPG file inside a Canon folder
28.                                         if (de2->name[4] == de4->name[4] && de2->name[5] == de4->name[5] &&//If the four digits of the Canon number are the same
29.                                             de2->name[6] == de4->name[6] && de2->name[7] == de4->name[7] &&
30.                                             de4->name[9] == 'J' && !(de4->name[0] == 'C' || de4->name[9] == 'C' || de4->name[0] == 0xE5)) {//If file is JPG, is not CRW/CR2 and is not a deleted item
31.                                             started();
32.                                             found=1;//A JPG file with the same Canon number was found
33.                                         }                                
34.                                     }
35.                                     closedir(d4);                
36.                                 }  
37.                             }
38.                             closedir(d3);
39.                             //If no JPG found, delete RAW file
40.                             if (found == 0) {
41.                                 sprintf(selected_item, "%s/%s", sub_dir, de2->name);
42.                                 remove(selected_item);
43.                                 finished();
44.                             }
45.                             else {
46.                                 found=0;
47.                                 finished();
48.                             }                            
49.                         }
50.                     }
51.                     closedir(d2);
52.                 }
53.             }
54.             closedir(d);
55.             i=strlen(current_dir);
56.             while (current_dir[--i] != '/');
57.             current_dir[i]=0;
58.         }
59.         //If item is a Canon folder (this is to purge all RAW files inside a single Canon folder)
60.         else if (selected->name[3] == 'C') {
61.             sprintf(current_dir+strlen(current_dir), "/%s", selected->name);
62.             d=opendir(current_dir);
63.             while ((de=readdir(d)) != NULL) {//Loop to find all the RAW files inside the Canon folder
64.                 if (de->name[0] == 'C' || de->name[9] == 'C') {//If file is RAW (Either CRW/CR2 prefix or file extension)
65.                     d2=opendir(current_dir);
66.                     while ((de2=readdir(d2)) != NULL) {//Loop to find a corresponding JPG file inside the Canon folder
67.                         if (de->name[4] == de2->name[4] && de->name[5] == de2->name[5] &&//If the four digits of the Canon number are the same
68.                             de->name[6] == de2->name[6] && de->name[7] == de2->name[7] &&
69.                             de2->name[9] == 'J' && !(de2->name[0] == 'C' || de2->name[9] == 'C' || de2->name[0] == 0xE5)) {//If file is JPG and is not CRW/CR2 and is not a deleted item
70.                             started();
71.                             found=1;//A JPG file with the same Canon number was found
72.                         }                                
73.                     }
74.                     closedir(d2);
75.                     //If no JPG found, delete RAW file                
76.                     if (found == 0) {
77.                         sprintf(selected_item, "%s/%s", current_dir, de->name);
78.                         remove(selected_item);
79.                         finished();
80.                     }
81.                     else {
82.                         found=0;
83.                         finished();
84.                     }
85.                 }
86.             }
87.             closedir(d);
88.             i=strlen(current_dir);
89.             while (current_dir[--i] != '/');
90.             current_dir[i]=0;
91.         }
92.         else {
93.             //If inside a Canon folder (files list)
94.             for (ptr=head; ptr; ptr=ptr->next) {//Loop to find all the RAW files in the list
95.                 if ((ptr->name[0] == 'C' || ptr->name[9] == 'C') && !(ptr->marked)) {//If file is RAW (Either CRW/CR2 prefix or file extension) and is not marked
96.                     for (ptr2=head; ptr2; ptr2=ptr2->next) {//Loop to find a corresponding JPG file in the list
97.                         if (ptr->name[4] == ptr2->name[4] && ptr->name[5] == ptr2->name[5] &&//If the four digits of the Canon number are the same
98.                             ptr->name[6] == ptr2->name[6] && ptr->name[7] == ptr2->name[7] &&
99.                             ptr2->name[9] == 'J' && !(ptr2->name[0] == 'C' || ptr2->name[9] == 'C')) {//If file is JPG and is not CRW/CR2
100.                             started();
101.                             found=1;
102.                         }
103.                     }
104.                     //If no JPG found, delete RAW file          
105.                     if (found == 0) {
106.                         sprintf(selected_file, "%s/%s", current_dir, ptr->name);
107.                         remove(selected_file);
108.                         finished();
109.                     }
110.                     else {
111.                         found=0;
112.                         finished();
113.                     }
114.                 }
115.             }
116.         }
117.         gui_fselect_read_dir(current_dir);
118.     }
119.     gui_fselect_redraw = 2;
120. }

[GrAnd]

Just a minor speedup suggestion: to insert "break;" just after each "found=1;".

Code: C

1.                         sprintf(selected_file, "%s/%s", current_dir, ptr->name);

I suspect here should be "selected_item", not "selected_file"

Code: C

1.                 if (de->name[0] == 'C' || de->name[9] == 'C') {//If file is RAW (Either CRW/CR2 prefix or file extension)

What about a deleted file with .CR* extension?

Code: C

1.                             de2->name[9] == 'J' && !(de2->name[0] == 'C' || de2->name[9] == 'C' || de2->name[0] == 0xE5)) {//If file is JPG and is not CRW/CR2 and is not a deleted item

Minor: Extra unnecessary check: "de2->name[9] == 'J'"  -> so it can'be "de2->name[9] == 'C'"

[wontolla]

Thank you for the sugestions GrAnd.

Just a minor speedup suggestion: to insert "break;" just after each "found=1;".

Minor? That could reduce the effort/time to a half! In the average case. No sense in keep searching if it's already found, right?
My teacher told me to avoid using "goto" or "break" statements. May be he wasn't right after all.

I suspect here should be "selected_item", not "selected_file"

Since this is the case when the list is already generated by the File Browser, the names of the files are already in the structure, so I can use "selected_file". In fact, this is the first part of the code I wrote, inspired in the "fselect_delete_file_cb" function.

What about a deleted file with .CR* extension?

What do you mean? If it is already deleted, we don't need to search for its JPG partner.

Minor: Extra unnecessary check: "de2->name[9] == 'J'"  -> so it can'be "de2->name[9] == 'C'"

True, I'll remove it for the sake of simplicity.

GrAnd, any way to know if the amount of memory is not enough for this application? Or what could be the cause of camera hanging sometimes and sometimes it simply works!

Thanks!

[GrAnd]

My teacher told me to avoid using "goto" or "break" statements. May be he wasn't right after all.

Regarding "goto" - yes, it could be weird. But "break" is very useful sometimes. Especial in case if you have a lot of conditions in the cycle. Of course, the appropriate condition can be placed in "while" statement, e.g.

Code: C

1.         while (!found && (de4=readdir(d4)) != NULL)

What about a deleted file with .CR* extension?

What do you mean? If it is already deleted, we don't need to search for its JPG partner.

I mean in that line(s) a deleted file with CRW/CR2 extension can be caught.

GrAnd, any way to know if the amount of memory is not enough for this application? Or what could be the cause of camera hanging sometimes and sometimes it simply works!

But you do not use a dynamic memory (no malloc()/free() functions I see).

[stevodude]

so is this in the cvs? or not ready yet. but can I make a build test with raw deletion for my s5is...
sorry for the dummy Q.
thats the only other thing that would be handy, as I have to download all images, then go through all crw/jpg to find the crw files without matching jpg's and delete before running my normal photo process...

[PhyrePhoX]

not in svn yet. you can manually merge the changes and compile on your own though

[wontolla]

It's been weeks since I worked on this. Sorry for the delay.

To avoid confusion, I moved the links and the explanation of this feature to the first post of this thread. Any other link is outdated.

Regarding the "hanging" problems some of you reported: I've been able to reproduce them only twice, so I can't still find the cause. GrAnd mentioned somewhere else a watchdog timeout, maybe it has something to do here. I'll continue experimenting, but I guess we'll have to get rid of the "Delete all RAW files in DCIM folder" function.

Have fun!