new branch - CHDK : Elf Edition - Developers wanted - page 12 - General Discussion and Assistance - CHDK Forum

new branch - CHDK : Elf Edition - Developers wanted

  • 316 Replies
  • 131296 Views
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #110 on: 26 / December / 2011, 05:39:25 »
Advertisements
Before the export list gets too set it might be worth ordering it a bit more logically, and perhaps adding some more potentially useful functions even if they aren't needed yet.

One concern I have is with the handling of the 'conf' structure - I realise it can't be handled easily in the export / module load code due to the way the compiler generates code for conf.variable access.

But having to create pointer variables for any conf variable used and all the CONF_BIND_xxx macros with their magic numbers seems to me to be a complex and difficult system to use.

The OSD layout editor would be a good candidate for turning into a module; but will be very difficult with this CONF_BIND_xxx solution.

Can I suggest a simpler approach would be:
in conf.h change this line and add the pconf variable
Code: [Select]
extern Conf conf, *pconf;

in conf.c add
Code: [Select]
Conf conf;
Conf *pconf = &conf;

Then add pconf to the export list and replace any usage of "conf." in the modules with "pconf->"

Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

*

Offline vnd

  • *
  • 36
Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #111 on: 26 / December / 2011, 05:55:41 »
I can't compile the current trunk (rev. 1506).

The compilation ends with this:

Code: [Select]
>> Entering to core
>> Entering to core/modules
../module_exportlist.c -> exportlist
-> benchm.flt
In file benchm.elf:
 point not to the __div0. Something unusual.
make[2]: *** [benchm.flt] Error 8
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

I am using gcc 4.3.3 installed as described here: http://chdk.wikia.com/wiki/Gcc433.

Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #112 on: 26 / December / 2011, 05:59:33 »
Before the export list gets too set it might be worth ordering it a bit more logically, and perhaps adding some more potentially useful functions even if they aren't needed yet.
We can do that. No problem but future extending of export list again will separate similar functions to different parts (because new exported symbols will be added to the end).
Currently really no problem what exactly order in the export list is, because elf2flt convertor will notify about all required to import symbols. You need just add them to the end of list and nothing more (exclusions are: conf and unsafe io functions, this code should be slightly changed with simple rules)

Quote
One concern I have is with the handling of the 'conf' structure - I realise it can't be handled easily in the export / module load code due to the way the compiler generates code for conf.variable access.

But having to create pointer variables for any conf variable used and all the CONF_BIND_xxx macros with their magic numbers seems to me to be a complex and difficult system to use.

The OSD layout editor would be a good candidate for turning into a module; but will be very difficult with this CONF_BIND_xxx solution.
Then add pconf to the export list and replace any usage of "conf." in the modules with "pconf->"

There is absolutely no difference between use pointer or just "conf." variable in module. Any variables are compiled in modules as addresses because "long-call" directive and then resolved with real address when module load.

This complex solution is required because we can't guarantee that conf structure is binary same on different modules. New conf variables could be added at any place, size of existed members could be changed. So we need exact address of each conf variable to binary compatibility in futher versions.

I see two possible way how to reach this:
1) The way which is used now. It little bit complicated but requirements are obvious.
BTW we need to use same magic numbers in the scripts if need to access to conf variable, so I see no problem on this.
2) Each required conf variable will be declared in export list. In module such symbols are declared with specific names (for example CONF_variable name). We can't use common syntax (conf.) because it just get with exact binary offset instead of separated address. Then elf2flt resolve correspondence between declaration in exportlist and such special symbols.
I think this is much more entangleous and unobvious.

*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #113 on: 26 / December / 2011, 06:15:24 »
Quote
One concern I have is with the handling of the 'conf' structure - I realise it can't be handled easily in the export / module load code due to the way the compiler generates code for conf.variable access.

But having to create pointer variables for any conf variable used and all the CONF_BIND_xxx macros with their magic numbers seems to me to be a complex and difficult system to use.

The OSD layout editor would be a good candidate for turning into a module; but will be very difficult with this CONF_BIND_xxx solution.
Then add pconf to the export list and replace any usage of "conf." in the modules with "pconf->"

There is absolutely no difference between use pointer or just "conf." variable in module. Any variables are compiled in modules as addresses because "long-call" directive and then resolved with real address when module load.

This complex solution is required because we can't guarantee that conf structure is binary same on different modules. New conf variables could be added at any place, size of existed members could be changed. So we need exact address of each conf variable to binary compatibility in futher versions.

I see two possible way how to reach this:
1) The way which is used now. It little bit complicated but requirements are obvious.
BTW we need to use same magic numbers in the scripts if need to access to conf variable, so I see no problem on this.
2) Each required conf variable will be declared in export list. In module such symbols are declared with specific names (for example CONF_variable name). We can't use common syntax (conf.) because it just get with exact binary offset instead of separated address. Then elf2flt resolve correspondence between declaration in exportlist and such special symbols.
I think this is much more entangleous and unobvious.

If you use pconf->variable pattern in the module code you only need to export the single pconf variable and it works correctly. You don't need to explicitly export every individual variable inside conf.

This requires less changes to the existing code, and in my opinion is cleaner. It also results in smaller code in the modules.

Worrying about the conf binary structure changing is, in my opinion, taking the whole portability issue too far. Put some comments in the code to only add new variables at the end. Changing the size of an existing conf variable will also break the CCHDK.CFG config file so is not likely to happen.

The whole config system probably needs an overhaul; but I really think the CONF_BIND_xxx solution is the wrong way to go.

Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #114 on: 26 / December / 2011, 06:35:49 »
I can't compile the current trunk (rev. 1506).

The compilation ends with this:
I am using gcc 4.3.3 installed as described here: http://chdk.wikia.com/wiki/Gcc433.

Thanks for report.
1. Please provide more information: what is your OS? do you use CHDK-shell?
2. Please unpack attached file into core/modules/, then do complete compilation. Then replace Makefile with Makefile.1 there and repeat full compilation again. Then zip all *.dumpobj in the core/modules and send them to me. If any compilation errors appears in process please attach log files from bin/log also.
I will investigate them when I will have slightly more time.
3. Just for now try gcc4.5.1 or gcc.4.6.2. No errors was reported for them yet.

Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #115 on: 26 / December / 2011, 06:37:28 »
If you use pconf->variable pattern in the module code you only need to export the single pconf variable and it works correctly. You don't need to explicitly export every individual variable inside conf.
This requires less changes to the existing code, and in my opinion is cleaner. It also results in smaller code in the modules.
I do agree that this is more simple and slightly decrease module. And even more. As I say before there absolutely no difference between pointer and variable. So declare pconf is not required.
But my vote is safety.
Both conf.variable and pconf->variable works in same way - add offset of member to base address and then get value from result address.

Quote
Worrying about the conf binary structure changing is, in my opinion, taking the whole portability issue too far. Put some comments in the code to only add new variables at the end. Changing the size of an existing conf variable will also break the CCHDK.CFG config file so is not likely to happen.
Personally I add new variables into conf structure by logical group principle. I saw same logic in some previous changesets. Similar thing you propose in your first post today for export_list.
While for export_list there are no difference for developer which order is in it, because this is auxilary lists and developer should never investigate it (elf2flt say if something required by module absent there). But conf. list is something more common and logical grouping could be useful there. (And any "wrong" adding could be reason of error and they will be hard to catch if just "ask developers to add to the end only". one more unobvious restriction)

PS - I investigated OSD layout editor before my first version posting here as candidate to separation and found that it is so tiny that no benefit on separate will be reached.
« Last Edit: 26 / December / 2011, 06:45:25 by tsvstar »

Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #116 on: 26 / December / 2011, 08:34:34 »
New updates related to flat modules in trunk:

1508
Some fixes for module system:
- Correct processing of multiple call safe_closedir() for same handler
- Increased stability during startup (do not use console module before it is ready)
- Fixed hashing of names for usermenu

1509
DNG code separate to loadable module. Some rewrites done to avoid float arithmetic using in module (and so bloating module size significantly). Works ok on my S95, compile ok for other platforms (but should be tested)

*

Offline vnd

  • *
  • 36
Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #117 on: 26 / December / 2011, 13:34:30 »
I can't compile the current trunk (rev. 1506).

The compilation ends with this:
I am using gcc 4.3.3 installed as described here: http://chdk.wikia.com/wiki/Gcc433.

Thanks for report.
1. Please provide more information: what is your OS? do you use CHDK-shell?
2. Please unpack attached file into core/modules/, then do complete compilation. Then replace Makefile with Makefile.1 there and repeat full compilation again. Then zip all *.dumpobj in the core/modules and send them to me. If any compilation errors appears in process please attach log files from bin/log also.
I will investigate them when I will have slightly more time.
3. Just for now try gcc4.5.1 or gcc.4.6.2. No errors was reported for them yet.

1. It is openSUSE 11.4,  gcc is compiled from sources, I closely followed the wiki page mentioned above.
2. With Makefile.1 almost all modules failed. The attached file contains the results + log.
3. With gcc4.5.1 it compiles without problems.



*

Offline reyalp

  • ******
  • 14126
Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #118 on: 26 / December / 2011, 15:28:10 »
I have one additional note:
For this case it is ok to replace one function with another and insert new one in the middle of the list in module_export.c, because this branch was used by few experienced developers yet.
Thanks for that, I wondered about that after I did it. And yeah, I should have read the comments :)

I think we will end up reworking some of this before we get it all right, so for the unstable branch, we should expect different builds will not be compatible. Try to put something in the commit message if you change it, but I wouldn't worry about version to version compatibility yet.

Quote
Reason of this requirement: to minimize modules size and core code, symbols are exported as indexes. So, module have no idea what real symbols are in the core. If order of the list is broken, then correspondence symbol is changed - old modules will call different symbol than they expect - module will hangup/shutdown/make bad things.
I would be strongly tempted to export symbols by name. Yes, it takes some RAM, but by going to modules, we've save a lot of RAM. In some cases, we can re-factor the interfaces so not so many exports are needed. Alternately, you could build a symbol map file with each build, then the loader could refer to that (but I don't think there's a need for this yet)

I'd also suggest we might want to break things up into specific APIs. So a module that only needs stdio can just say it needs stdio, and changes in other areas won't affect it.

Quote
In my build I use it as shortcut to call module inspector. So I can easily check at any moment is module loaded/unloaded or not, what impact to the memory.
Module inspector would be a good option for debug shortcut used for ram dump, prop/param paging etc.

Quote
1509
DNG code separate to loadable module. Some rewrites done to avoid float arithmetic using in module (and so bloating module size significantly). Works ok on my S95, compile ok for other platforms (but should be tested)
I don't understand the need for this. I would like to get some of the big questions resolved before we go turning everything into modules. With the current module setup, most cameras should have plenty of RAM, there's no need to shove every possible part into a module right now. If we make changes to how the module system works, then we need to redo every modules.

I know both phil and I have concerns about the current implementation, I would rather get the basics sorted out before adding even more modules. As I said before, I would have much preferred to just put a few simple things in modules to start with. Since you already did the work, I took what you had done, but I really think this is making it harder for ourselves.
Don't forget what the H stands for.

*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: new branch - CHDK : Elf Edition - Developers wanted
« Reply #119 on: 26 / December / 2011, 15:29:45 »
Both conf.variable and pconf->variable works in same way - add offset of member to base address and then get value from result address.

This is not the case (at least on my system).

Try this:
- export conf
- add this line to module_inspector.c near the top of gui_module_draw
        draw_txt_string(0, 1,  conf.reader_file, MAKE_COLOR(SCREEN_COLOR, COLOR_WHITE));
- run CHDK, and open a file in the text reader
- run the module inspector, it should show the file name just opened in the reader

When I compile and run the above code the file name is not displayed.

When I do the same using pconf it displays correctly.

I presume this is because the code being generated does not work with the mechanism used to import the module.

Phil.

CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

 

Related Topics


SimplePortal © 2008-2014, SimplePortal