CHIMP - Canon Hack Installation and Management Platform - page 4 - General Discussion and Assistance - CHDK Forum supplierdeeply

CHIMP - Canon Hack Installation and Management Platform

  • 134 Replies
  • 31370 Views
Re: CHDK for non-hackers
« Reply #30 on: 23 / April / 2017, 09:28:07 »
Advertisements
Is this discussion is also happening in other forums?  Would you please provide links if so?

No other forums, just the ChdkUtility wiki, which isn't supposed to be a platform for discussions.
Author of CHIMP, Canon Hack Installation and Management Platform

*

Offline reyalp

  • ******
  • 12072
Re: CHDK for non-hackers
« Reply #31 on: 23 / April / 2017, 15:50:46 »
I'm OK with CHDKMETA (or _HDKMETA perhaps?), although I'm not sure DSLR hack guys would be very happy about it.
HACKMETA?

In any case, unless you have SDM and DSLR developers on board, worrying too much about interoperability seems premature.
Quote
You're out of date :) There are currently proposals for three files, namely:

SOFTWARE.JSN to indicate the installed software
CAMERA.JSN to indicate the detected camera (cannot be reliably produced during CHDK build)
MODEL.JSN to indicate the selected camera (applies to the N/N Facebook case)

There is one additional proposal that's too high level to discuss ATM.
I saw those, but I'm unclear on the purpose and not yet convinced of the need. Do these things solve a significant problem, or do they just seem like a good idea in the abstract?

I do appreciated the time and thought you are putting into this, but it strikes me as overkill for a install utility. We've gone ~9 years without any of this stuff, and I don't see it addressing any of the major problems people have installing.
Quote
If it's generated by a client software, that has to be really bad to either mess it up programmatically or cause the user to do that at a later stage
Except that users will continue to use old versions of software that doesn't know about it, or manually copy files.

Quote
I can see why, although I can also relate to the notion of keeping a memory card as tidy as possible (did I mention OCD already?). I just found out about the MISC directory, which seems to be created by newer DSLRs, and I find it even more compelling (since it's top-level).
My reasoning is that how Canon uses these directories is unspecified, and may change at any time, so we should avoid them unless there is a compelling reason to do otherwise.

Quote
I believe metadata belongs with the binary, so if present, it must be on the primary partition (at least).
CHDK is split though: DISKBOOT.BIN must go on the boot partition, and the modules must go on the data partition. Yet the version should apply to both.

Quote
Compared to that, the idea of an unencrypted metadata block sounds great, although I'm not sure why it should be favored over the JSON proposal, other than the latter's being significantly harder to implement (although I am biased).
The main benefit is that the information is directly attached to the binary, so the question about where to put it goes away, and you can rely on it always being present. But again, I'm not set on this over the other proposal.
Don't forget what the H stands for.

Re: CHDK for non-hackers
« Reply #32 on: 23 / April / 2017, 16:57:39 »
HACKMETA?

Whichever you like more.

Quote
I saw those, but I'm unclear on the purpose and not yet convinced of the need. Do these things solve a significant problem, or do they just seem like a good idea in the abstract?

When a user runs the utility with a card containing a previously installed build, the metadata allows to determine:

1. whether an update is required and
2. the platform and revision for downloading one

You asked for a proposal, so I went ahead and submitted one (or a few :)).

Quote
CHDK is split though: DISKBOOT.BIN must go on the boot partition, and the modules must go on the data partition. Yet the version should apply to both.

It does, but I don't see why the utility should check both partitions.

Quote
The main benefit is that the information is directly attached to the binary, so the question about where to put it goes away, and you can rely on it always being present.

I stand corrected. Once the metadata are part of the binary, separate JSON file(s) would be unnecessary. As for the format, I'm leaning towards BSON.

P.S. Assuming the binary is different in every build, there is a way of making sure the separate metadata file is up to date by adding a hash of the binary. If it's OK with you, I'll update the JSON proposal accordingly.
« Last Edit: 24 / April / 2017, 03:21:19 by dmitrys »
Author of CHIMP, Canon Hack Installation and Management Platform

Re: CHDK for non-hackers
« Reply #33 on: 24 / April / 2017, 08:32:36 »
Looking back at my last post, I seem to have made a fool of myself on numerous occasions.

I saw those, but I'm unclear on the purpose and not yet convinced of the need. Do these things solve a significant problem, or do they just seem like a good idea in the abstract?

I believe I had misunderstood the question. Let me try again.

Since metadata are currently only generated by CHDK Utility, they are useful in the update scenario. Namely,

1. SOFTWARE.JSN is used to determine the currently installed version and the target platform and revision,
2. CAMERA.JSN is used to present the user with the camera details without having to detect those from EXIF metadata, and
3. MODEL.JSN is used to skip the model selection step (which only applies to the N/N Facebook case).

Quote from: reyalp
CHDK is split though: DISKBOOT.BIN must go on the boot partition, and the modules must go on the data partition. Yet the version should apply to both.

You were right, of course. I completely overlooked the case of a card with the partitions switched on pre-Redstone 2 Windows.

The metadata should be present on both partitions, and if we want to validate it, so should DISKBOOT.BIN. Otherwise, the partitions would need to be switched, DISKBOOT.BIN read, and the partitions switched back. That wouldn't be very pretty.

Here's the updated code:
Quote
        private void CopySecondaryFiles(string srcPath, string destPath)
        {
            CopyFiles(srcPath, destPath, new[] { "DISKBOOT.BIN" });
            CopyAllDirectories(srcPath, destPath);
        }

My apologies and thanks for bearing with me.
« Last Edit: 24 / April / 2017, 08:36:39 by dmitrys »
Author of CHIMP, Canon Hack Installation and Management Platform


Re: CHDK for non-hackers
« Reply #34 on: 24 / April / 2017, 08:46:39 »
Here's another borderline insane idea.

I'm operating on the assumption of the build info being present (in an easily retrievable form) in the unencrypted DISKBOOT.BIN.

Something like the following (in binary form, of course) could then be appended to the encrypted DISKBOOT.BIN:

Quote
{
  "encoding": "dancing",
  "data": "046107253"
}

That way, the build info would only be kept once, with no dependencies on metadata format(s).

I haven't looked at the FIR/FI2 encoding yet, but I believe it could benefit from a similar treatment.
Author of CHIMP, Canon Hack Installation and Management Platform

*

Offline reyalp

  • ******
  • 12072
Re: CHDK for non-hackers
« Reply #35 on: 24 / April / 2017, 13:21:12 »
Looking back at my last post, I seem to have made a fool of myself on numerous occasions.
Not at all, like I said, I really appreciate the time and thought you are putting into this :)

Quote
Since metadata are currently only generated by CHDK Utility, they are useful in the update scenario. Namely,

1. SOFTWARE.JSN is used to determine the currently installed version and the target platform and revision,
2. CAMERA.JSN is used to present the user with the camera details without having to detect those from EXIF metadata, and
3. MODEL.JSN is used to skip the model selection step (which only applies to the N/N Facebook case).
Is the idea that install utilities would maintain the second two, or are these also expected in the CHDK zip?

Regarding #2, I assume you would give the user a way of preparing a card for a different camera than what is already installed?

Regarding #3, the SOFTWARE.JSN distinguishes N and N_facebook already (by "camera"), so is it really needed? I'm also not clear how the example on https://github.com/CHDKUtil/ChdkUtility/wiki/Camera-model-metadata-proposal relates to dealing with the N/N_Facebook distinction.

Quote from: reyalp
The metadata should be present on both partitions, and if we want to validate it, so should DISKBOOT.BIN. Otherwise, the partitions would need to be switched, DISKBOOT.BIN read, and the partitions switched back. That wouldn't be very pretty.
Multi-partition is basically not pretty, especially with windows versions that don't support it :x

Having it on both increase the risk of it being out of sync. But like I said, it's ugly no matter how you slice it.

Quote
Something like the following (in binary form, of course) could then be appended to the encrypted DISKBOOT.BIN:
This is an interesting idea. We have up to avoided distributing the FI2 keys due to DMCA concerns of the people who made the tool, but OTOH the dancingbits keys are included in the CHDK source. My feeling is if we are going to add any un-encrypted data, it might as well be the whole version information, it's probably only a few tens of bytes.
Don't forget what the H stands for.

Re: CHDK for non-hackers
« Reply #36 on: 24 / April / 2017, 17:08:09 »
Is the idea that install utilities would maintain the second two, or are these also expected in the CHDK zip?

As I said, #2 cannot be reliably provided in the CHDK zip. #3 could be, but I don't see why it should be.

Quote
Regarding #2, I assume you would give the user a way of preparing a card for a different camera than what is already installed?

Should I? I might be adding an Uninstall option at some point, so they could do an uninstall, then restart and install something different.

Quote
Regarding #3, the SOFTWARE.JSN distinguishes N and N_facebook already (by "camera"), so is it really needed? I'm also not clear how the example on https://github.com/CHDKUtil/ChdkUtility/wiki/Camera-model-metadata-proposal relates to dealing with the N/N_Facebook distinction.

Its only use is for displaying correct and complete camera model (not "n_facebook") to the user, so it should always be present.

The example is somewhat devious, since the utility automatically resolves the IXUS 132/135 ambiguity.

Quote
Multi-partition is basically not pretty, especially with windows versions that don't support it :x

Just to make sure, is it required for cards smaller than (or equal to) 4GB?

Quote
Having it on both increase the risk of it being out of sync.

I think the integrity problem would be solved by adding a hash of DISKBOOT.BIN to SOFTWARE.JSN.

Quote
We have up to avoided distributing the FI2 keys due to DMCA concerns of the people who made the tool

Wasn't reverse-engineering explicitly exempt from DMCA?

Quote
My feeling is if we are going to add any un-encrypted data, it might as well be the whole version information, it's probably only a few tens of bytes.

It's just that I spent some time deciphering the dancingbits code (did someone do an April Fools commit and forget to revert?), and I'd hate to see all that effort go down the drain :P

Edit: Added show-off screenshots.
« Last Edit: 24 / April / 2017, 17:21:02 by dmitrys »
Author of CHIMP, Canon Hack Installation and Management Platform

Re: CHDK for non-hackers
« Reply #37 on: 25 / April / 2017, 12:46:47 »
I updated the proposal as follows:

1. Clarified some optional/empty fields.
2. Added a link to the actual implementation.
3. Add a hash property.

I'm considering removing the compiler data, since it serves no purpose in update/uninstall scenarios. What do you think?

The hash validation is already implemented, so if the software metadata is out of sync, detection is degraded to assume none is present. Little protection is provided against intentional tampering (e.g. there is no validation of the file name actually being DISKBOOT.BIN), but it seems reasonably resistant to accidental damage.

I'm getting close to releasing an early preview (probably sans source), so can we agree on a name for the metadata directory?

Edit 2017-05-02: Added file name validation.
« Last Edit: 02 / May / 2017, 13:51:46 by dmitrys »
Author of CHIMP, Canon Hack Installation and Management Platform


*

Offline reyalp

  • ******
  • 12072
Re: CHDK for non-hackers
« Reply #38 on: 26 / April / 2017, 17:47:10 »
I'm getting close to releasing an early preview (probably sans source), so can we agree on a name for the metadata directory?
I vote for _HDKMETA or HACKMETA. Feel free to pick one, if it gets added to the build scripts we will use whichever one.

Quote
I'm considering removing the compiler data, since it serves no purpose in update/uninstall scenarios. What do you think?
We have sometimes had compiler specific bugs, but that probably doesn't matter to the installer. OTOH, it should be easy to include from the build, and it may be useful to be able to see by looking in the zip without loading it on a camera.
Don't forget what the H stands for.

Re: CHDK for non-hackers
« Reply #39 on: 26 / April / 2017, 18:10:46 »
_HDKMETA it is then. I'll update the source shortly.

Quote from: reayalp
We have sometimes had compiler specific bugs, but that probably doesn't matter to the installer. OTOH, it should be easy to include from the build, and it may be useful to be able to see by looking in the zip without loading it on a camera.

No problem, it's just that I'm currently unable to deduce that from the zip, so right now it's null.
Author of CHIMP, Canon Hack Installation and Management Platform

 

Related Topics