Source code re-org for modules? - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: [1] 2 3 4   Go Down

Source code re-org for modules?

  • 36 Replies
  • 18243 Views
*

Offline philmoz

  • *****
  • 3450
    • Photos
Source code re-org for modules?
« on: 30 / November / 2012, 21:34:19 »
Advertisements
The modules in the trunk version are now 100% platform independent across all cameras and firmware versions.
(I built them all and did a binary compare of all the module files.)

I'm currently testing a change to the batch build that will build the module files only once - hopefully will speed it up a bit.

I think a re-org of the source code is in order - e.g.:
- move the modules directory out of core to the top level
- move all the module code files out of core into the modules directory
- move the games directory out of core to be a sub-directory of modules.

Any comments, questions or suggestions?

Phil.



Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline waterwingz

  • ******
  • 12140
    • Waterwingz
Re: Source code re-org for modules?
« Reply #1 on: 30 / November / 2012, 21:43:03 »
Quote from: philmoz on 30 / November / 2012, 21:34:19
The modules in the trunk version are now 100% platform independent across all cameras and firmware versions.
...
Any comments, questions or suggestions?
Yea - I have a comment.  Thanks for doing this.

My only question is whether there is anything that needs to be done to ensure it stays this way ?  Or did they end up that way on their  own so will likely stay that way ?

Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Source code re-org for modules?
« Reply #2 on: 30 / November / 2012, 21:55:38 »
Quote from: waterwingz on 30 / November / 2012, 21:43:03
Quote from: philmoz on 30 / November / 2012, 21:34:19
The modules in the trunk version are now 100% platform independent across all cameras and firmware versions.
...
Any comments, questions or suggestions?
Yea - I have a comment.  Thanks for doing this.

My only question is whether there is anything that needs to be done to ensure it stays this way ?  Or did they end up that way on their  own so will likely stay that way ?



:)

It was one of the original goals; but not so simple to achieve.

Moving forward it will take a bit of diligence to ensure it's not broken by updates; although there might be some things that can be done in the build to help prevent it.
For example in the modified makefile I'm testing it bypasses loading the propset?.h file. So any attempt to use a propset value will generate a compile error.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14125
Re: Source code re-org for modules?
« Reply #3 on: 01 / December / 2012, 01:13:38 »
Quote from: philmoz on 30 / November / 2012, 21:55:38
For example in the modified makefile I'm testing it bypasses loading the propset?.h file. So any attempt to use a propset value will generate a compile error.
The "right" way to do this would be to make a set of module safe includes, and not include anything else in module code. Currently, the modules include things like platform.h and stdlib.h which have platform dependent ifdefs in them.

I'm somewhat wary of having the autobuild build modules only once without that sort of protection, it seems like this could lead to subtle errors which might go unnoticed for a long time.

The other possible concern is merging outstanding branches. The remote capture stuff is mostly in the core, but it does touch dng a bit. I suspect that trying to merge the "move" will not have the desired effect, but it can probably be resolved manually without a huge amount of pain. I don't know about tsvstars ui branch, it didn't start as a proper branch in the first place, and I don't know what if anything will go back to the trunk.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Source code re-org for modules?
« Reply #4 on: 01 / December / 2012, 01:55:08 »
Quote from: reyalp on 01 / December / 2012, 01:13:38
Quote from: philmoz on 30 / November / 2012, 21:55:38
For example in the modified makefile I'm testing it bypasses loading the propset?.h file. So any attempt to use a propset value will generate a compile error.
The "right" way to do this would be to make a set of module safe includes, and not include anything else in module code. Currently, the modules include things like platform.h and stdlib.h which have platform dependent ifdefs in them.

I'm somewhat wary of having the autobuild build modules only once without that sort of protection, it seems like this could lead to subtle errors which might go unnoticed for a long time.

The other possible concern is merging outstanding branches. The remote capture stuff is mostly in the core, but it does touch dng a bit. I suspect that trying to merge the "move" will not have the desired effect, but it can probably be resolved manually without a huge amount of pain. I don't know about tsvstars ui branch, it didn't start as a proper branch in the first place, and I don't know what if anything will go back to the trunk.

Good points; but making it completely foolproof will be hard.
I'll have a look at the current include files and see what can be done relatively easily.

The code location changes make sense to me even if the 'build-once' module change is left out - not sure about merging though. Need to think more about that.

I've created a test branch to play with in case anyones interested - it currently reduces the full build time from 40 minutes to 28 minutes on my system.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14125
Re: Source code re-org for modules?
« Reply #5 on: 01 / December / 2012, 02:14:16 »
Just to be clear, I'm not objecting to this going into the trunk. I don't think we need the module stuff to be completely foolproof.

Another impact on related to merging would be porting stuff between the release and trunk branches. Again, not a showstopper, just something to keep in mind.

Thanks for working on this, it's definitely a worthwhile improvement, both in terms of organization and batch build speed.
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Source code re-org for modules?
« Reply #6 on: 01 / December / 2012, 15:53:22 »
Quote from: philmoz on 30 / November / 2012, 21:34:19
Any comments, questions or suggestions?
Not related directly, but it would be nice if new modules could be added in a way that doesn't require modifying CHDK source. Maybe one subdir for a module somewhere in the source tree.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Source code re-org for modules?
« Reply #7 on: 03 / December / 2012, 03:45:26 »
Quote from: srsa_4c on 01 / December / 2012, 15:53:22
Quote from: philmoz on 30 / November / 2012, 21:34:19
Any comments, questions or suggestions?
Not related directly, but it would be nice if new modules could be added in a way that doesn't require modifying CHDK source. Maybe one subdir for a module somewhere in the source tree.

Not sure what you mean here.

I'm trying to consolidate all the module code into one directory.

As far as modifying CHDK for a new module you have to at least:
- add it to the makefile
- add it to a menu if it has a UI you want to invoke
- add code to ubasic, lua or other core routines to call the module
- potentially update the loader to handle load/unload of the module

I doubt it would be possible to get around most of this.

There was an experiment in dynamically building a module menu at startup; but it was not stable on all cameras and would also cause problems with the current 'user' menu implementation.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Source code re-org for modules?
« Reply #8 on: 03 / December / 2012, 11:54:49 »
Quote from: philmoz on 03 / December / 2012, 03:45:26
Not sure what you mean here.

I'm trying to consolidate all the module code into one directory.

As far as modifying CHDK for a new module you have to at least:
- add it to the makefile
- add it to a menu if it has a UI you want to invoke
- add code to ubasic, lua or other core routines to call the module
- potentially update the loader to handle load/unload of the module

I doubt it would be possible to get around most of this.

There was an experiment in dynamically building a module menu at startup; but it was not stable on all cameras and would also cause problems with the current 'user' menu implementation.
To work on a new module, one has to edit core/modules/Makefile. If the makefile is altered (due to a commit made by someone else), there's a chance for conflicts, and the makefile parts related to the new module have to be re-adjusted. I know that the same happens in case of non-module related development, and that it's a relatively minor annoyance.
There are modules which can be run directly from the file browser (games, benchmark, utilities), and do not necessarily need changes in CHDK code. As there's no SDK for building modules, a simple "put the module's files in a subdir of /modules" approach may make things a bit easier (especially in case of non-official modules).

The above is only my opinion, I'm not asking you to do this.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Source code re-org for modules?
« Reply #9 on: 07 / December / 2012, 18:40:50 »
The latest version in philmoz-reorg includes quite a few changes to try and make the module code cleaner and less prone to including camera dependencies. This was a deeper rabbit hole than I first imagined :)

Main changes:
- remove camera dependencies from stdlib.h. All the work for the differences in the 'open', 'stat' and directory functions in now done in the wrapper code. Note this meant a change to the llibtst.lua script because the internal 'stat' structure is now simpler.
- split platform_camera.h and camera.h into pieces (e.g. camera_info.h, viewports.h, etc). These are now included in the module code and attempting to include either camera.h or platform_camera.h will give a compile error in a module.
- clean up include file usage in other source files
- simplify module header files
- remove the safe_XXX functions which are no longer needed for modules
- probably other stuff I've forgotten

I haven't encountered any problems so far, tested on both VxWorks and DryOS cameras; but the changes are extensive so I may have broken something.

Testing and feedback appreciated as always.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
Pages: [1] 2 3 4   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal