adding dual partition support for >2012 camera's and exFAT for all

Mlapse

583
S95 S110

« on: 17 / June / 2022, 08:12:40 »

Advertisements

we've already had this discussion in another thread, https://chdk.setepontos.com/index.php?topic=13772.40
but in a seperate request wanted to point out why I think this is a step forward for CHDK driven cams.

the possibility was build by Philmoz and tested by me on the s110 and g1x.

i would like to have tested it on a <2011 cam, but my last one is still occupied for a while.

what we have now:

<2011
FAT16 as boot & image/files partition
if larger than 4GB:
FAT16 as boot partition
FAT32 as image/files partition

>2012
FAT16 as boot & image partition
if larger than 4GB:
FAT32 as boot & image/files partition

this works good enough for most and since Reyalp has changed the boot window from 4 to 10 seconds it will work up to 256GB cards.

what I'm suggesting is this change:

<2011
FAT16 as boot & image partition
if larger than 4GB:
FAT16 as boot partition
FAT32 as image/files partition
option if larger than 32GB:
FAT16 as boot partition
exFAT as image/files partition

>2012
FAT16 as boot & image/files partition
if larger than 4GB:
FAT32 as boot & image/files partition
option if larger than 32GB:
FAT16 as boot partition
exFAT as image/files partition

If it all works with FAT32, why the change?
boot takes longer than needed on >32GB FAT32 partitions.
saving images takes longer than needed on >32GB FAT32 partitions.
starting the cam is less and less responsive with large and larger FAT32 cards. plus a risk of timeout/missing modules

other advantages:
you can now use the default formatting tools of your os.
performance of the camera is improved.
there is little need to create a seperate pre and after 2011 install guide
that part of the builds can become more uniform.

disadvantages:
it has to be build

I would love to see the dual partition feature included in all builds with the possibility to use exFAT as a second partition.
Even if most of the cams will never be in my hands.

For those who don't like new or just have a natural 'glass is half empty' disposition. consider the efficiency, progress and performance. and remember, you can still keep on using FAT32, my proposal does not exclude what already is.

i've tested reboot time with SUIx and 2 identical sd cards of 64GB, something most of you will have lying around...or you can reformat the one card you have
making it easyer to test and check my finds.

however, this is the smallest SDXC card and gives only limited gains.
resolution in the SUIx logs is 1 second, so that's not really helpfull either.

64GB red EVO plus/S110-102b - reboot test
25 images before reboot, 5 runs:

FAT16/exFAT FAT32
1 sec        3 sec
1 sec        3 sec
2 sec        3 sec
1 sec        3 sec

with bigger cards, the FAT16/exFAT time between shutdown/power on at reboot seems to stay under 2 seconds while the FAT32 time for a reboot grows with card size. (256GB, FAT32, average reboot: 9 seconds)

If these facts don't convince..or you think it's too much work.
in that case my feature request to Philmoz is, could you make a test version for the S110-103a as well, because i love what you've done and want it on all MY cams

« Last Edit: 17 / June / 2022, 12:27:39 by Mlapse »

Logged

frustration is a key ingredient in progress

philmoz

3450

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #1 on: 17 / June / 2022, 18:36:18 »

@mlapse

The changes for the G1X and S110 have been checked into the trunk (1.6) version so should be available on the auto build server for these cameras.

As to the rest - something like this is not likely to happen for the reasons listed below. Basically it's a huge amount of work and testing.

The changes are required in the CHDK startup code which is unique to each camera and firmware version. There are currently over 320 of these.

Some like the G1X and S110 are easier to change as the CHDK startup code is generated with a custom tool so making changes is simpler and less error prone. For the others the changes have to be done manually and this requires extracting, copying and modifying code from the camera firmware disassembly - it is a tedious process and, being manual, can be error prone.

Testing then becomes an issue as we would need to find people with all the affected cameras and get them to test the changes.

The exFAT file system was introduced in 2006 and it appears it was not adopted by Canon until 2009 / 2010 - this makes sense given their 2-3 year development cycle for new cameras. So many CHDK supported cameras would not benefit at all since the firmware does not support exFAT.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14079

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #2 on: 17 / June / 2022, 21:57:59 »

Quote from: Mlapse on 17 / June / 2022, 08:12:40

you can now use the default formatting tools of your os.

To autoboot with exFAT, you still have to make a multi-partition card, which historically has been a major stumbling block for many people and causes other inconveniences. So while giving people the option of multi-partition with exFAT is good, I'd argue single should still be the top recommendation.

It would be good to mention the performance issues with large cards somewhere and note that exFAT multi-partition support can be requested on the forum.

Anyway, as Phil said, adding multi-partition to all the applicable ports be a major project, and isn't very safe to do without having someone test each affected platform.

It would be interesting to know if digic 6 and later cams see the same performance problems with large FAT32 cards. They have faster SD interfaces and CPUs, so it may be less of an issue.

Logged

Don't forget what the H stands for.

Mlapse

583
S95 S110

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #3 on: 18 / June / 2022, 00:34:29 »

Philmoz, thank you for checking in the g1x and s110 with dual partition support, makes it possible for me to evolve to chdk 1.7 when it is there.
and i had no clue about how much work it would entail.
was more thinking in the line of copy/paste to all with that custom tool would be easier than individual changes, but apparently i was wrong

and yes, 320 is a bit much to test...i tend to forget it's not the models, but all the revisions that make up that number.
even if only half of those are beyond 2009 and would support exFAT it is overwhelming.
so i withdraw my request

I'm sorry i asked, that's what you get with non-programmers, they always assume it's a breeze

i will think about where and how to mention the performance issues.
something in the line of:
FAT32 formatted large card has missing modules at boot or your cam feels slow or sluggish now you put in a large FAT32 card=>
try and see if it won't happen with exFAT and firmware update=>
If it doesn't, card size is too large for FAT32 in combination with your camera=>
you can request an update for your model/revision so your cam will support dual partition with exFAT support.

Reyalp, as for trouble with dual partition, i've had my share complete with wrecked windows due to 3rd party partition swap tool (before win supported multi partitions) and last year win10 losing multi partition capability due to installing a disk creator for raspberry. i even had a dedicated linux machine for a while just to copy dual partition cards.
so i totally agree with you that single partition is best. that's why i thought of it as a possible option.
for me however, it is worth all the trouble because of the gains.

although i don't have one anymore, i would think the issue persists on a digic 6 or higher cam. it is not the card that gets slow, it's the canon hard/software that gets into trouble.
the speedyer cpu or sd interface may make the numbers smaller, but they would still be there....it is the growing in boot-time with fat32 card size compared to exfat where this does not appear to happen that makes me come to this belief.

as soon as i've picked up a G-series digic6/7 cam i will request a build for it and test it.

now my fall back plan for next years project is working better than my regular project

i'm going back to the chdkptp drawing board. still have a lot of work to do there.

« Last Edit: 18 / June / 2022, 13:27:42 by Mlapse »

Logged

frustration is a key ingredient in progress

Mlapse

583
S95 S110

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #4 on: 11 / September / 2022, 13:02:25 »

i have tested the dual setup with exfat on the s110 and g1x for a while now and i have found no downside.....apart from the dual partition

@philmoz
if you would be willing to add one more cam to the exfat dual boot list, i could go for something close to the s110, like the s100. i have one of those as well.
but i have a m3-121a that would really appreciate a go at this.
i can imagine this being a port that is just too much work to change. so i won't be that disappointed if it doesn't work out

« Last Edit: 11 / September / 2022, 13:22:37 by Mlapse »

Logged

frustration is a key ingredient in progress

philmoz

3450

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #5 on: 11 / September / 2022, 19:17:36 »

Quote from: Mlapse on 11 / September / 2022, 13:02:25

if you would be willing to add one more cam to the exfat dual boot list, i could go for something close to the s110, like the s100. i have one of those as well.
but i have a m3-121a that would really appreciate a go at this.

@mlapse
Neither the S100 or M3 ports support dual partition cards.
I'm not sure any of the Digic6 or 7 ports have dual partition support.
I don't know why this was not added to the S100 originally.

Does the M3 suffer from the performance problem on large FAT32 cards?

If it has the same problem, then it should be possible to add dual partition support to the M3 - the firmware code looks the same as other cameras in this area.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

philmoz

3450

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #6 on: 11 / September / 2022, 22:03:00 »

@mlapse

I've added dual partition support (with exFat) to the S100.

I don't know which firmware version you have so I uploaded all of them to - https://drive.google.com/drive/folders/144HYAJ4wcR_om64CqDAaUiIUct1wdt5K?usp=sharing

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

Mlapse

583
S95 S110

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #7 on: 12 / September / 2022, 06:29:10 »

that's quick, i've checked the 102a version out and it is running at this moment.
that is the only s100 version i have, so i can only test this one.
it looks like it performs similar to the s110, quicker response.
for me it's a keeper

i've tested the m3 with fat32 vs exfat and booting the cam or saving an image is noticeable slower.
this is without chdk, so i have no data how long saving a shot actually takes, but booting is, like with the other cams, countable in seconds extra.

« Last Edit: 12 / September / 2022, 06:36:25 by Mlapse »

Logged

frustration is a key ingredient in progress

philmoz

3450

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #8 on: 13 / September / 2022, 05:14:19 »

Here's a test version for the M3 1.21a.

Edit: remove bad build.

« Last Edit: 13 / September / 2022, 08:10:55 by philmoz »

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

Mlapse

583
S95 S110

Re: adding dual partition support for >2012 camera's and exFAT for all

« Reply #9 on: 13 / September / 2022, 07:51:12 »

the single partition chdk version that was on there is 1.6.0-6121-ALPHA.

alas, this 6131 build you posted did not boot on my 121a cam with a dual partition fat/exfat card.

normally on the m3 when you close the lid, you see a short flash of a green led behind the on/off button.
when i close the lid with this build, that led is on for 1-2 seconds and pressing the on/off button has no result.

i've copied the files twice, just to be sure i did not make a mistake. same result.
with the s100 build...in a s100

the card boots normally.

fyi. if i put the s100 build in the m3 the led does not light up when closing the lid.
it also does not light up if you do not have a locked card in your cam.

« Last Edit: 13 / September / 2022, 08:15:44 by Mlapse »

Logged

frustration is a key ingredient in progress