Parallel make - General Discussion and Assistance - CHDK Forum
supplierdeeply

Parallel make

  • 7 Replies
  • 3909 Views
*

Offline Hacki

  • ****
  • 359
  • EOS 80D
Parallel make
« on: 04 / August / 2008, 09:52:33 »
Advertisements
Hello folks,

I'd like to build chdk with parallel make (e.g.: make -j 4 batch-zip), to reduce build time. Juciphox autobuild is set up on a root server with an AMD Opteron dual core CPU, so parallel processing would probably reduce the time quite a bit. (18 minutes at the moment).

Unfortunately it doesnt work right away. First, it didnt work at all as the makefiles in loader/$cam werent suitable for parallel processing, i had to add the line:

Code: [Select]
$(RESET_FILE): all-recursive

otherwise it would have complained about a missing rule for resetcode/main.bin

Now i'm stuck with irregularly occuring errors like:
Code: [Select]
setjmp.o lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o
lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o
lundump.o lvm.o lzio.o lauxlib.o lbaselib.o ldblib.o ltablib.o lmathlib.o loadlib.o linit.o -> liblua.a
arm-elf-ar: setjmp.o: No such file or directory
make[3]: *** [liblua.a] Error 1

Or

Code: [Select]
../../../../tools/finsig.c:3:20: stdint.h: No such file or directory
../../../../tools/finsig.c:5:20: unistd.h: No such file or directory
../../../../tools/finsig.c:10: error: syntax error before "uint32_t"
../../../../tools/finsig.c:10: warning: no semicolon at end of struct or union
../../../../tools/finsig.c:11: warning: type defaults to `int' in declaration of `fail'
../../../../tools/finsig.c:11: warning: data definition has no type or storage class
../../../../tools/finsig.c:12: error: syntax error before "success"
[.. lot more like that ..]

And sometimes gcc complains about missing gensig.o (I guess that happens when "Cleaning tools/" happens right after compiling gensig.c, so the object file gets removed before the gensig programm gets generated)

Odd thing is, those errors dont happen everytime, and they dont happen for the same cams if they happen. They just seem to occur randomly. So, overall there seem to be a few of race conditions or missing dependencies. I never wrote or modified a makefile before today, so my skills are quite limited.

Still, i'd like to make it possible to build chdk with parallel make, first to make CHDK compile faster on multi-core platforms, and second.. just for the sake of it.

Some opinions and a little assistance would be really welcome. :)

Regards,
hacki

*

Offline GrAnd

  • ****
  • 916
  • [A610, S3IS]
    • CHDK
Re: Parallel make
« Reply #1 on: 04 / August / 2008, 15:02:05 »
Juciphox autobuild is set up on a root server with an AMD Opteron dual core CPU, so parallel processing would probably reduce the time quite a bit. (18 minutes at the moment).

18 minutes... Hmm... It takes less than 5 minutes for my autobuilding process without any parallelization. I use AMD Opteron server as well.
CHDK Developer.

*

Offline Hacki

  • ****
  • 359
  • EOS 80D
Re: Parallel make
« Reply #2 on: 04 / August / 2008, 15:14:53 »
18 minutes... Hmm... It takes less than 5 minutes for my autobuilding process without any parallelization. I use AMD Opteron server as well.

Make batch-zip of current trunk takes 5 minutes and 36 seconds.. I guess juciphox is just a little.. errh.. bloated ;)


*

Offline Jucifer

  • *****
  • 251
  • [A710IS]
Re: Parallel make
« Reply #3 on: 04 / August / 2008, 15:16:08 »
bloated it is. :]


*

Offline PhyrePhoX

  • *****
  • 2254
  • make RAW not WAR
    • PhyreWorX
Re: Parallel make
« Reply #4 on: 04 / August / 2008, 15:16:58 »
well, we do compress alotta things into the archive, don't we :D

btw wb grand :)

*

Offline GrAnd

  • ****
  • 916
  • [A610, S3IS]
    • CHDK
Re: Parallel make
« Reply #5 on: 04 / August / 2008, 15:29:29 »
well, we do compress alotta things into the archive, don't we :D

Then the parallelization wont help. Or you save just couple minutes...

btw wb grand :)

:D I still do not have enough free time to participate.
CHDK Developer.

*

Offline Hacki

  • ****
  • 359
  • EOS 80D
Re: Parallel make
« Reply #6 on: 04 / August / 2008, 15:55:04 »
Just did some testing:

Code: [Select]
juciphox:

make PLATFORM=sx100is PLATFORMSUB=100c : 12 sec
make -s --no-print-directory PLATFORM=sx100is  PLATFORMSUB=100c NO_INC_BUILD=1 firzipsubcomplete: 14 sec

trunk:

make PLATFORM=sx100is PLATFORMSUB=100c : 8 sec
make -s --no-print-directory PLATFORM=sx100is  PLATFORMSUB=100c NO_INC_BUILD=1 firzip: 8 sec

So.. batch-zip-complete *should* only take around 10 minutes. And its not the zip compression what takes that long.

Anyways, making it possible to use parallel compilation seems a little useful to me..

Edit:

Just noticed that i let the cronjob run with nicelevel 5 - with nice 0, juciphox compiles in 8 minutes. (although theres virtually no other load on that server, i dont know why it gets THAT slow with nice 5)


« Last Edit: 04 / August / 2008, 20:21:10 by Hacki »

*

Offline Hacki

  • ****
  • 359
  • EOS 80D
Re: Parallel make
« Reply #7 on: 05 / August / 2008, 12:17:30 »
I just had the idea to append a $(SUFFIX) consisting of $(PLATFORM) and $(PLATFORMSUB) to every "temporary" file generated (objects, stuff in tools/), so things wont get deleted during make clean.

Currently it sometimes happens that make clean for the next cam starts and deletes object files an older compile process still needs. Appending the platform & platformsub to every object file would ship around that issue, but increase the size of the build directory dramatically at the same time.

Another solution would be to tell make to build only one cam at a time, but still let make parallelize this task. (compiling the different files from core/ simultaneously)

I could do the first one, but thats really not an smart solution. (Could add a rule to remove those object files after a successful build, though) The second one is much more elegant i think, but i have no idea how to implement that.

regards, hacki


 

Related Topics


SimplePortal 2.3.6 © 2008-2014, SimplePortal