Another confirmed case of firmware bit rot - page 2 - General Discussion and Assistance - CHDK Forum supplierdeeply
« previous next »
Pages: 1 [2] 3   Go Down

Another confirmed case of firmware bit rot

  • 22 Replies
  • 784 Views
*

Offline reyalp

  • ******
  • 12586
Re: Another confirmed case of firmware bit rot
« Reply #10 on: 22 / June / 2020, 02:33:03 »
Advertisements
Here's a patch that adds support for batch generating the crc files and including them in build zips. I'm not totally happy with this, I think compiling in the checksum information in (ala bin_compat) is probably a better approach.

batch-rebuild-firmware-crc generates files named <pid in hex><sub>.crc in the platform directory with the ports actual source. So if d10 100b is a copy of 100a, both 31bc100a.crc and 31bc100b.crc are created in the 100a directory. Similarly, sx270hs 102c uses the source in the sx280 102c directory, so it will contain both 325f102c.crc  (sx280) and 3260102c.crc (sx270). This simplifies getting them into the final zip.
The intention is that all of these files would be checked in, and the build command would be run manually when a port is added.
Missing dumps are printed, but not treated as an error, since copied ports aren't all in the archive.

The firsub rule is updated to copy any .crc files from the platform directory to CHDK/DATA

The crc files are deleted by the top level clean, and also CHDK/Makefile clean (which isn't called in batch builds)

Dumps for copied builds I didn't find in my collection or the archive:
Code: [Select]
a2000 100a
a610 100d
d10 100b
g11 100h
ixus1000_sd4500 100b
ixus220_elph300hs 101d
ixus70_sd1000 101a
ixus75_sd750 101b
ixus800_sd700 101a
ixus850_sd800 100d
s2is 100i
sx500is 100e
sx280hs 102d
sx275hs 102b
sx275hs 102c
sx270hs 102b
g5x 110a
This is less than I expected, and I haven't yet tried to track down the rest on the forum.

For the copied builds I do have dumps for, the checksums for the main ROM code do differ, as expected. RAM data and code are often the same.
Logged
Don't forget what the H stands for.
*

Offline Caefix

  • ***
  • 213
  • Sorry, busy deleting test shots...
Re: Another confirmed case of firmware bit rot
« Reply #11 on: 22 / June / 2020, 13:37:18 »
Quote from: Caefix on 08 / June / 2020, 15:14:50
Quote from: Ant on 08 / June / 2020, 14:48:31
Quote from: srsa_4c on 08 / June / 2020, 13:22:59
official firmware update file.
If link   :( then try
https://www.touslesdrivers.com/index.php?v_page=12&v_code=2
Just an idea for that few OFUs.
There´s somewhere a "if new_FW > is_FW then FW_update()"
Could that somehow be changed to "if new_FW >= is_FW then FW_update()"
or a "set_is_FW=0.9" to activate them for affected cams?
Logged
All lifetime is a loan from eternity.
*

Offline srsa_4c

  • ******
  • 4228
Re: Another confirmed case of firmware bit rot
« Reply #12 on: 27 / June / 2020, 17:43:57 »
Quote from: reyalp on 22 / June / 2020, 02:33:03
Here's a patch that adds support for batch generating the crc files and including them in build zips.
Sorry for trying too late, but this isn't quite working for me. Neither the single, nor the batch one. I define PRIMARY_ROOT (and every other option) in localbuildconf.inc, but make errors out with
ERROR: zero byte dump /(...)/trunk/platform/s80/sub/100g/PRIMARY.BIN
in other words, it complains about an in-tree PRIMARY.BIN which isn't there.

Quote
Dumps for copied builds I didn't find in my collection or the archive:
sx280 102d: https://filebin.net/jfgb729aiclz2gqn

sx270/275/280 102b main fw can be extracted from the official update, but as far as I know, they are identical - camera name and capability flags are in camera specific higher part of the ROM.
Logged
*

Offline reyalp

  • ******
  • 12586
Re: Another confirmed case of firmware bit rot
« Reply #13 on: 27 / June / 2020, 22:00:29 »
Quote from: srsa_4c on 27 / June / 2020, 17:43:57
Quote
Here's a patch that adds support for batch generating the crc files and including them in build zips.
Sorry for trying too late, but this isn't quite working for me. Neither the single, nor the batch one. I define PRIMARY_ROOT (and every other option) in localbuildconf.inc, but make errors out with
ERROR: zero byte dump /(...)/trunk/platform/s80/sub/100g/PRIMARY.BIN
in other words, it complains about an in-tree PRIMARY.BIN which isn't there.
Thanks for trying, looks like this was my mistake: I normally set PRIMARY_ROOT on the command line (make PRIMARY_ROOT=...) which is passed on in the environment. Makefile variables are not by default.
You should be able fix this by adding -d $(PRIMARY_ROOT) to the line that invokes the python script.

Anyway, I'm currently working on a different approach that handles copied subs differently (in the python script rather than makefile) and generates header files.

Quote
sx270/275/280 102b main fw can be extracted from the official update, but as far as I know, they are identical - camera name and capability flags are in camera specific higher part of the ROM.
Thanks. I do the have SX270 102c dump, and it does indeed end up with identical checksums to the corresponding sx280hs firmware. I wasn't sure if we could rely on that though.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 12586
Re: Another confirmed case of firmware bit rot
« Reply #14 on: 29 / June / 2020, 01:36:19 »
Here's an updated patch.

Checksums are now built into the core. rebuild-firmware-crc uses make-fw-crc.py to generate a header file platform/<cam>/sub/<fw>/firmware_crc_data.h
This contains structures with checksums for all the subs supported by that port. make-fw-crc uses the camera_list.csv to figure out copied subs, and the header is always created in the sub with the real port. Copied platforms like sx270hs aren't specifically supported, but they end up building the the real port directory (i.e. sx280) so will work as long as the checked regions of the firmware are identical.

The header doesn't currently include or check the PID, so if we ever had build that could run on multiple platforms and needed different checksums for a given firmware version, changes would be needed.

The intent is the headers would be checked in, and only updated manually when a new port is added or checks are updated. The file must be present, but you could make a dummy with sub_count 0. I started adding support for something similar to bin_compat_fix.h, but I don't think any current ports need it.

The checksum is checked on boot, based on conf.check_firmware_crc. This happens immediately before the main spytask loop, so the splash screen is not shown while the check is running. If an error is detect, you get a popup instead of the splash (example attached).

The patch adds two items to the debug menu for testing:
"Checksum Canon firmware" runs the check and always display results
"Checksum at boot lets" you control the CFG value.

I like having the checksum data in diskboot.bin since it avoids having a bunch of obscurely named files to copy around, and means the module doesn't have to deal with parsing and error conditions, but it adds up to a few hundred bytes on top of the core code needed to support this. I could be convinced to switch back to files.

It doesn't currently allow dumping the firmware, but I plan to add that.

edit:
If you want to test the failure case, you can edit the 3rd column of the firmware_crc_block_t for your firmware version.
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4228
Re: Another confirmed case of firmware bit rot
« Reply #15 on: 29 / June / 2020, 17:49:04 »
Quote from: reyalp on 29 / June / 2020, 01:36:19
If an error is detect, you get a popup instead of the splash (example attached).
Just a minor suggestion, I think it might be more appropriate to use "warning" rather than "error". A small corruption does not necessarily cause recognizable problems.

Quote
I like having the checksum data in diskboot.bin since it avoids having a bunch of obscurely named files to copy around, and means the module doesn't have to deal with parsing and error conditions, but it adds up to a few hundred bytes on top of the core code needed to support this.
There are a few ARAM ports where core size is close to the amount of ARAM. This extra will probably still fit, so falling back to separate file is not necessarily needed.
Another possible way to avoid adding extra files would be piggy-backing extra data on diskboot files (that's something I was considering in the past). I can't say for sure if that method would work on fir/fi2 files though.
Logged
*

Offline reyalp

  • ******
  • 12586
Re: Another confirmed case of firmware bit rot
« Reply #16 on: 30 / June / 2020, 02:25:12 »
Quote from: srsa_4c on 29 / June / 2020, 17:49:04
Just a minor suggestion, I think it might be more appropriate to use "warning" rather than "error". A small corruption does not necessarily cause recognizable problems.
Agreed.

Quote
There are a few ARAM ports where core size is close to the amount of ARAM. This extra will probably still fit, so falling back to separate file is not necessarily needed.
Yeah, ARAM cameras should have a build error if the size goes over, but I think we've reduced diskboot size quite a bit in 1.5 compared to 1.4.

Quote
Another possible way to avoid adding extra files would be piggy-backing extra data on diskboot files (that's something I was considering in the past).
I've had thoughts along these lines too, but never pursued it.

New version attached:
* Now prompts to click OK to dump the firmware. Can also cancel.
* Moved the check to the simple module "run" function. firmware_crc_desc is now exported, but avoids all the autoload glue in core/modules.c which makes the total size smaller.
* Removed the "Checksum Canon firmware" menu item, and the ability to use an external file. * Running the module from tools does the same as the boot time check.
* Changed some wording: warning instead of error, ROM instead of firmware.

I'm inclined to keep the "Checksum at boot" item that lets you set the cfg value.

I think this is basically complete, aside from tidying up a few things like lang strings and doing a full batch build.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 12586
Re: Another confirmed case of firmware bit rot
« Reply #17 on: 01 / July / 2020, 02:08:35 »
I discovered that just throwing up dialogs while not in alt mode causes some problems (canon firmware also sees key presses, and there are drawing issues), so here's an updated version that implements a GUI handler. It shows a progress bar while running the CRC and dump.

Another thing I wondered about is if hogging the CPU with the checksum would cause problems starting in record mode. It doesn't seem to on my cameras (both newer cameras that use power long press, and A540 which has a physical switch), the switch to rec happens before the checksum stuff runs.
Logged
Don't forget what the H stands for.
*

Offline Caefix

  • ***
  • 213
  • Sorry, busy deleting test shots...
Re: Another confirmed case of firmware bit rot
« Reply #18 on: 01 / July / 2020, 10:50:38 »
Quote from: reyalp on 01 / July / 2020, 02:08:35
the switch to rec happens before the checksum stuff runs.
CRC stuff in playmode (only (once a day)) would not desturb me...
Logged
All lifetime is a loan from eternity.
*

Offline reyalp

  • ******
  • 12586
Re: Another confirmed case of firmware bit rot
« Reply #19 on: 01 / July / 2020, 12:57:06 »
Quote from: Caefix on 01 / July / 2020, 10:50:38
CRC stuff in playmode (only (once a day)) would not desturb me...
It's intended to only run on the first startup of a fresh install or config reset, although you can currently opt to have it run every boot. The issue is that the user might try to boot into rec mode on the first boot.

We could try to skip the check if the camera is in rec, but I'm not sure the play/rec state is updated by the time it runs.

In any case, it seems to work OK (on my cameras, anyway). The dialog can get erased by canon firmware activity.
Logged
Don't forget what the H stands for.
Pages: 1 [2] 3   Go Up
« previous next »
 

Related Topics