Source code re-org for modules?

philmoz

3450

Source code re-org for modules?

« on: 30 / November / 2012, 21:34:19 »

Advertisements

The modules in the trunk version are now 100% platform independent across all cameras and firmware versions.
(I built them all and did a binary compare of all the module files.)

I'm currently testing a change to the batch build that will build the module files only once - hopefully will speed it up a bit.

I think a re-org of the source code is in order - e.g.:
- move the modules directory out of core to the top level
- move all the module code files out of core into the modules directory
- move the games directory out of core to be a sub-directory of modules.

Any comments, questions or suggestions?

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

waterwingz

12139

Re: Source code re-org for modules?

« Reply #1 on: 30 / November / 2012, 21:43:03 »

Quote from: philmoz on 30 / November / 2012, 21:34:19

The modules in the trunk version are now 100% platform independent across all cameras and firmware versions.
...
Any comments, questions or suggestions?

Yea - I have a comment. Thanks for doing this.

My only question is whether there is anything that needs to be done to ensure it stays this way ? Or did they end up that way on their own so will likely stay that way ?

Logged

Ported : A1200 SD940 G10 Powershot N G16

philmoz

3450

Re: Source code re-org for modules?

« Reply #2 on: 30 / November / 2012, 21:55:38 »

Quote from: waterwingz on 30 / November / 2012, 21:43:03

Quote from: philmoz on 30 / November / 2012, 21:34:19
The modules in the trunk version are now 100% platform independent across all cameras and firmware versions.
...
Any comments, questions or suggestions?
Yea - I have a comment. Thanks for doing this.

My only question is whether there is anything that needs to be done to ensure it stays this way ? Or did they end up that way on their own so will likely stay that way ?

It was one of the original goals; but not so simple to achieve.

Moving forward it will take a bit of diligence to ensure it's not broken by updates; although there might be some things that can be done in the build to help prevent it.
For example in the modified makefile I'm testing it bypasses loading the propset?.h file. So any attempt to use a propset value will generate a compile error.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14082

Re: Source code re-org for modules?

« Reply #3 on: 01 / December / 2012, 01:13:38 »

Quote from: philmoz on 30 / November / 2012, 21:55:38

For example in the modified makefile I'm testing it bypasses loading the propset?.h file. So any attempt to use a propset value will generate a compile error.

The "right" way to do this would be to make a set of module safe includes, and not include anything else in module code. Currently, the modules include things like platform.h and stdlib.h which have platform dependent ifdefs in them.

I'm somewhat wary of having the autobuild build modules only once without that sort of protection, it seems like this could lead to subtle errors which might go unnoticed for a long time.

The other possible concern is merging outstanding branches. The remote capture stuff is mostly in the core, but it does touch dng a bit. I suspect that trying to merge the "move" will not have the desired effect, but it can probably be resolved manually without a huge amount of pain. I don't know about tsvstars ui branch, it didn't start as a proper branch in the first place, and I don't know what if anything will go back to the trunk.

Logged

Don't forget what the H stands for.

philmoz

3450

Re: Source code re-org for modules?

« Reply #4 on: 01 / December / 2012, 01:55:08 »

Quote from: reyalp on 01 / December / 2012, 01:13:38

Quote from: philmoz on 30 / November / 2012, 21:55:38
For example in the modified makefile I'm testing it bypasses loading the propset?.h file. So any attempt to use a propset value will generate a compile error.
The "right" way to do this would be to make a set of module safe includes, and not include anything else in module code. Currently, the modules include things like platform.h and stdlib.h which have platform dependent ifdefs in them.

I'm somewhat wary of having the autobuild build modules only once without that sort of protection, it seems like this could lead to subtle errors which might go unnoticed for a long time.

The other possible concern is merging outstanding branches. The remote capture stuff is mostly in the core, but it does touch dng a bit. I suspect that trying to merge the "move" will not have the desired effect, but it can probably be resolved manually without a huge amount of pain. I don't know about tsvstars ui branch, it didn't start as a proper branch in the first place, and I don't know what if anything will go back to the trunk.

Good points; but making it completely foolproof will be hard.
I'll have a look at the current include files and see what can be done relatively easily.

The code location changes make sense to me even if the 'build-once' module change is left out - not sure about merging though. Need to think more about that.

I've created a test branch to play with in case anyones interested - it currently reduces the full build time from 40 minutes to 28 minutes on my system.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14082

Re: Source code re-org for modules?

« Reply #5 on: 01 / December / 2012, 02:14:16 »

Just to be clear, I'm not objecting to this going into the trunk. I don't think we need the module stuff to be completely foolproof.

Another impact on related to merging would be porting stuff between the release and trunk branches. Again, not a showstopper, just something to keep in mind.

Thanks for working on this, it's definitely a worthwhile improvement, both in terms of organization and batch build speed.

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: Source code re-org for modules?

« Reply #6 on: 01 / December / 2012, 15:53:22 »

Quote from: philmoz on 30 / November / 2012, 21:34:19

Any comments, questions or suggestions?

Not related directly, but it would be nice if new modules could be added in a way that doesn't require modifying CHDK source. Maybe one subdir for a module somewhere in the source tree.

Logged

philmoz

3450

Re: Source code re-org for modules?

« Reply #7 on: 03 / December / 2012, 03:45:26 »

Quote from: srsa_4c on 01 / December / 2012, 15:53:22

Quote from: philmoz on 30 / November / 2012, 21:34:19
Any comments, questions or suggestions?
Not related directly, but it would be nice if new modules could be added in a way that doesn't require modifying CHDK source. Maybe one subdir for a module somewhere in the source tree.

Not sure what you mean here.

I'm trying to consolidate all the module code into one directory.

As far as modifying CHDK for a new module you have to at least:
- add it to the makefile
- add it to a menu if it has a UI you want to invoke
- add code to ubasic, lua or other core routines to call the module
- potentially update the loader to handle load/unload of the module

I doubt it would be possible to get around most of this.

There was an experiment in dynamically building a module menu at startup; but it was not stable on all cameras and would also cause problems with the current 'user' menu implementation.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

srsa_4c

4451

Re: Source code re-org for modules?

« Reply #8 on: 03 / December / 2012, 11:54:49 »

Quote from: philmoz on 03 / December / 2012, 03:45:26

Not sure what you mean here.

I'm trying to consolidate all the module code into one directory.

As far as modifying CHDK for a new module you have to at least:
- add it to the makefile
- add it to a menu if it has a UI you want to invoke
- add code to ubasic, lua or other core routines to call the module
- potentially update the loader to handle load/unload of the module

I doubt it would be possible to get around most of this.

There was an experiment in dynamically building a module menu at startup; but it was not stable on all cameras and would also cause problems with the current 'user' menu implementation.

To work on a new module, one has to edit core/modules/Makefile. If the makefile is altered (due to a commit made by someone else), there's a chance for conflicts, and the makefile parts related to the new module have to be re-adjusted. I know that the same happens in case of non-module related development, and that it's a relatively minor annoyance.
There are modules which can be run directly from the file browser (games, benchmark, utilities), and do not necessarily need changes in CHDK code. As there's no SDK for building modules, a simple "put the module's files in a subdir of /modules" approach may make things a bit easier (especially in case of non-official modules).

The above is only my opinion, I'm not asking you to do this.

Logged

philmoz

3450

Re: Source code re-org for modules?

« Reply #9 on: 07 / December / 2012, 18:40:50 »

The latest version in philmoz-reorg includes quite a few changes to try and make the module code cleaner and less prone to including camera dependencies. This was a deeper rabbit hole than I first imagined

Main changes:
- remove camera dependencies from stdlib.h. All the work for the differences in the 'open', 'stat' and directory functions in now done in the wrapper code. Note this meant a change to the llibtst.lua script because the internal 'stat' structure is now simpler.
- split platform_camera.h and camera.h into pieces (e.g. camera_info.h, viewports.h, etc). These are now included in the module code and attempting to include either camera.h or platform_camera.h will give a compile error in a module.
- clean up include file usage in other source files
- simplify module header files
- remove the safe_XXX functions which are no longer needed for modules
- probably other stuff I've forgotten

I haven't encountered any problems so far, tested on both VxWorks and DryOS cameras; but the changes are extensive so I may have broken something.

Testing and feedback appreciated as always.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)