1.4 development planning thread

reyalp

14039

Re: 1.4 development planning thread

« Reply #160 on: 13 / September / 2015, 19:28:48 »

Advertisements

Quote from: philmoz on 13 / September / 2015, 17:52:52

Update patch. This should fix the sx275 c&p error, and auto_build_parallel.sh, also simplified auto_build.sh slightly.

Thanks. I've checked this in, with one minor change. auto-build.sh had ${copy} rather than ${cpy} in

Code: [Select]

$1 -s --no-print-directory TARGET_FW=${fw} PLATFORM=${cam} PLATFORMSUB=${srcfw} STATE=${state} SKIP_AUTOBUILD=${skip} ${2}${copy} || exit 1; \

Quote

I've left the OPT_GEN_STUBS override in the sx280 makefile.inc file.
My reasoning is that development of a finsig version for thumb2 would most likely begin with a single camera version. Having it in the makefile.inc means it can be turned on/off for individual versions for testing rather than being a global setting.

Makes sense.

Logged

Don't forget what the H stands for.

reyalp

14039

Re: 1.4 development planning thread

« Reply #161 on: 13 / September / 2015, 20:24:16 »

Note

rm -f -d -r *.dSYM

was added to tools 'clean'. I'm not sure what was intended there, I don't think our normal toochains generate anything called .dSYM, and -d is a pretty special case option. I've commented this line out for now to allow the build to run.

Logged

Don't forget what the H stands for.

philmoz

3450

Re: 1.4 development planning thread

« Reply #162 on: 13 / September / 2015, 20:44:11 »

Quote from: reyalp on 13 / September / 2015, 20:24:16

Note

rm -f -d -r *.dSYM

was added to tools 'clean'. I'm not sure what was intended there, I don't think our normal toochains generate anything called .dSYM, and -d is a pretty special case option. I've commented this line out for now to allow the build to run.

Sorry about that. It was for my OS/X build, where the *.dSYM directories gets created.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14039

Re: 1.4 development planning thread

« Reply #163 on: 13 / September / 2015, 21:09:32 »

Quote from: philmoz on 13 / September / 2015, 20:44:11

Sorry about that. It was for my OS/X build, where the *.dSYM directories gets created.

No problem, if you want it in the checked in version -rf should be ok, but -d fails on linux if you aren't root.

Logged

Don't forget what the H stands for.

philmoz

3450

Re: 1.4 development planning thread

« Reply #164 on: 13 / September / 2015, 22:25:52 »

Quote from: reyalp on 13 / September / 2015, 21:09:32

Quote from: philmoz on 13 / September / 2015, 20:44:11
Sorry about that. It was for my OS/X build, where the *.dSYM directories gets created.
No problem, if you want it in the checked in version -rf should be ok, but -d fails on linux if you aren't root.

It's fine as is. El Capitan broke the hack I was using to disable the OS/X PTPCamera app that takes over the camera when it's plugged in to the USB port.

I switched to building in a VM running Debian (Mint LMDE2), which lets me connect the camera to the VM USB port and use chdkptp again.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

srsa_4c

4451

Re: 1.4 development planning thread

« Reply #165 on: 15 / September / 2015, 20:05:08 »

I have two proposals.

The first one is rather simple: add an option that allows loading CHDK to a memory area other than Canon heap start/ARAM/EXMEM. Either for experimental reasons or perhaps permanently.

Code: [Select]

Index: arm_rules.inc
===================================================================
--- arm_rules.inc	(revision 4252)
+++ arm_rules.inc	(working copy)
@@ -69,6 +69,10 @@
 CFLAGS+=-DOPT_FORCE_LUA_CALL_NATIVE
 endif
 
+ifdef OPT_NOTCANONHEAP
+CFLAGS+=-DCHDK_NOT_IN_CANON_HEAP
+endif
+
 ifdef OPT_ARAM_MALLOC
 CFLAGS+=-DOPT_ARAM_MALLOC -DARAM_HEAP_START=$(ARAM_HEAP_START) -DARAM_HEAP_SIZE=$(ARAM_HEAP_SIZE)
 # Only allow OPT_CHDK_IN_ARAM when OPT_ARAM_MALLOC is also defined

Second one: a modification to the batch-rebuild-stubs-parallel build action (diff attached). Uses GNU Parallel (a utility written in Perl) to run one sigfinder process per CPU core. It should work on Windows too, but I did not try that.
I'm posting this mod because I use it whenever I work on the sigfinders.

batch-rebuild-stubs-parallel_mod.diff (2.31 kB - downloaded 21 times.)

Logged

reyalp

14039

Re: 1.4 development planning thread

« Reply #166 on: 20 / September / 2015, 14:53:25 »

Quote from: srsa_4c on 15 / September / 2015, 20:05:08

The first one is rather simple: add an option that allows loading CHDK to a memory area other than Canon heap start/ARAM/EXMEM. Either for experimental reasons or perhaps permanently.

I don't really follow this. By itself, the posted diff doesn't specify where CHDK is loaded, so just setting it would cause failures unless there is additional platform change. Is the idea that it would force the use of ARAM or EXMEM, or some other platform defined area? If you have to make other makefile.inc changes, then I'm not clear what the value of having the OPT_ would be?

Quote

Second one: a modification to the batch-rebuild-stubs-parallel build action (diff attached). Uses GNU Parallel (a utility written in Perl) to run one sigfinder process per CPU core. It should work on Windows too, but I did not try that.

With a few minutes of trying, I wasn't able to get parallel to work with mingw. configure/make/install runs, but any invocation of parallel (including --help) generates errors like

Code: [Select]

syntax error at -e line 1, near "and ="
                                       Execution of -e aborted due to compilation errors.
                                                                                         Usage:

                                                                                               parallel [options] [comma
nd [arguments]] < list_of_arguments
...

(weird formatting in original)

I'd suggest a separate script for GNU parallel, or a some kind of conditional for OS or user specified value.

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: 1.4 development planning thread

« Reply #167 on: 20 / September / 2015, 15:41:49 »

Quote from: reyalp on 20 / September / 2015, 14:53:25

I don't really follow this. By itself, the posted diff doesn't specify where CHDK is loaded, so just setting it would cause failures unless there is additional platform change.

I have this in the ixus150 makefile.inc (it's experimental):

Code: [Select]

# trial, put CHDK into unused space after DryOS kernel (size: 0x6f1000-0x6c2000=0x2f000, 188kB)
OPT_NOTCANONHEAP=1
MEMISOSTART=0x006c2000

Quote

Is the idea that it would force the use of ARAM or EXMEM, or some other platform defined area?

The idea is that I can place the core to whatever address I would like
- without having to edit the port's boot.c
- without using the standard EXMEM or ARAM #defines which use up the rest of the specified area as additional memory pool

Quote

If you have to make other makefile.inc changes, then I'm not clear what the value of having the OPT_ would be?

I can't just put CHDK_NOT_IN_CANON_HEAP=1 or CFLAGS+=-DCHDK_NOT_IN_CANON_HEAP in the port's makefile.inc because it won't get picked up from there.

Of course this is completely unsafe (no checks are provided), so it's really just an experimental option. I'm also open to other suggestions (I should probably just edit boot.c, I guess).

Quote

With a few minutes of trying, I wasn't able to get parallel to work with mingw. configure/make/install runs, but any invocation of parallel (including --help) generates errors like
Code: [Select]
syntax error at -e line 1, near "and =" Execution of -e aborted due to compilation errors. Usage: parallel [options] [comma nd [arguments]] < list_of_arguments ...(weird formatting in original)

That's unfortunate.

Quote

I'd suggest a separate script for GNU parallel, or a some kind of conditional for OS or user specified value.

I'll think about this.

Logged

reyalp

14039

Re: 1.4 development planning thread

« Reply #168 on: 20 / September / 2015, 16:25:36 »

Quote from: srsa_4c on 20 / September / 2015, 15:41:49

The idea is that I can place the core to whatever address I would like
- without having to edit the port's boot.c
- without using the standard EXMEM or ARAM #defines which use up the rest of the specified area as additional memory pool

Now it makes sense, thanks. We should probably have something like OPT_CHDK_LOAD_IN=aram|exmem|heap|... instead of having individual options for aram, exmem etc, since they are all mutually exclusive anyway, but I don't know that it's worth going through the whole tree to set up.

Quote

I'll think about this.

Could just add

Code: [Select]

batch-rebuild-stubs-gnu-parallel:
	sh tools/auto_build_gnu_parallel.sh $(MAKE) rebuild-stubs $(CAMERA_LIST) -noskip | parallel

where auto_build_gnu_parallel.sh is your new script. I wouldn't want to do it this way we had a lot of different parallel rules, but we don't right now.

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: 1.4 development planning thread

« Reply #169 on: 21 / September / 2015, 14:51:38 »

Quote from: reyalp on 20 / September / 2015, 16:25:36

We should probably have something like OPT_CHDK_LOAD_IN=aram|exmem|heap|... instead of having individual options for aram, exmem etc, since they are all mutually exclusive anyway, but I don't know that it's worth going through the whole tree to set up.

That would probably be nice but hand-editing lots of files would risk breaking some ports.

For the parallel stubs-rebuild:

Code: [Select]

Index: Makefile
===================================================================
--- Makefile	(revision 4260)
+++ Makefile	(working copy)
@@ -194,6 +194,10 @@
 		echo "missing primary for $(PLATFORM) $(PLATFORMSUB)" ; \
 	fi
 
+sigfinders:
+	$(MAKE) -C tools finsig_dryos$(EXE)
+	$(MAKE) -C tools finsig_vxworks$(EXE)
+
 rebuild-stubs: platformcheck
 	if [ -s $(TARGET_PRIMARY) ] ; then \
 		$(MAKE) -C tools finsig_dryos$(EXE) ;\
@@ -285,9 +289,14 @@
 # parallel version, starts each camera/firmware version build in a seperate session
 # Note:- Windows only, this will use all available CPU and a fair amount of memory
 #        but will rebuild much faster on a machine with many CPU cores
-batch-rebuild-stubs-parallel:
-	sh tools/auto_build_parallel.sh $(MAKE) rebuild-stubs $(CAMERA_LIST) -noskip
+batch-rebuild-stubs-parallel: sigfinders
+	sh tools/auto_build_parallel.sh $(MAKE) rebuild-stubs $(CAMERA_LIST) start -noskip
 
+# *nix version, may work on Windows too
+# Note:- needs GNU Parallel, runs one job per CPU core
+batch-rebuild-stubs-gnu-parallel: sigfinders
+	sh tools/auto_build_parallel.sh $(MAKE) rebuild-stubs $(CAMERA_LIST) echo -noskip | parallel
+
 batch-clean:
 	$(MAKE) -C tools clean
 	$(MAKE) -C modules clean THUMB_FW=
Index: tools/auto_build_parallel.sh
===================================================================
--- tools/auto_build_parallel.sh	(revision 4260)
+++ tools/auto_build_parallel.sh	(working copy)
@@ -3,16 +3,17 @@
 # Read the 'camera_list.csv' file ($3) and run the
 # CHDK build action ($2) using the designated
 # make program ($1)
-# ($4) = -noskip disable SKIP_AUTOBUILD
+# ($4) = 'start' (for Windows) or 'echo' (for creating a batch file)
+# ($5) = -noskip disable SKIP_AUTOBUILD
 # - also see main Makefile
 # parallel version - starts each build in a seperate session (Windows only)
 while IFS=, read cam fw state srcfw skip
 do
-  if [ ${cam} != "CAMERA" ] && ( [ "$4" = "-noskip" ] || [ "${skip}" = "" ] ); then
+  if [ ${cam} != "CAMERA" ] && ( [ "$5" = "-noskip" ] || [ "${skip}" = "" ] ); then
     if [ "${state}" != "" ]; then state=_${state}; fi
 # skip copied firmwares
     if [ "${srcfw}" = "" ]; then
-    	start $1 -s --no-print-directory TARGET_FW=${fw} PLATFORM=${cam} PLATFORMSUB=${fw} STATE=${state} SKIP_AUTOBUILD=${skip} $2 || exit 1;
+    	$4 $1 -s --no-print-directory TARGET_FW=${fw} PLATFORM=${cam} PLATFORMSUB=${fw} STATE=${state} SKIP_AUTOBUILD=${skip} $2 || exit 1;
 	fi
   fi
 done < $3

Perhaps better than making a copy of that script.

edit:
Finally managed to try GNU parallel on Windows (Vista x86, on a Cygwin based install), with success. Little problem is that recent Cygwin no longer includes stricmp in strings.h, so the sigfinder source needs a trivial fix (and the makefiles don't recognize Cygwin either, so that also needed a fix).

« Last Edit: 25 / September / 2015, 17:41:26 by srsa_4c »

Logged