Author Topic: Autobuilding of trunk  (Read 11938 times)

Offline GrAnd

  • Developers
  • Hero Member
  • ****
  • Posts: 916
  • [A610, S3IS]
    • CHDK
Autobuilding of trunk
« on: 22 / March / 2008, 02:18:07 »
I've set up a script on one server, which checks every 15 minutes for the new revision in svn repository and builds it if it's found. The build takes 5 minutes. Then puts zip-files to http://grandag.nm.ru/hdk/autobuild/download.htm. So in the worst case the new build will be available in 20 minutes (in 5 minutes if the check-in was just before the check).

Very experimental feature...  :D

edit by phyrephox:

HERE you can find an autobuild server that not only hosts the autobuilds grand hosts, but the EXTENDED zips as well (including readmes and files etc). Provided by Hacki. Don't ask about the domain name, it's weird german.
So if you need the extended zips - go there. If you got problems connecting to russian domains - go to Hackis place.
Dont just always download the "big" zips though, as the extra files arent updates as often as the binaries - so better check the versions first.
« Last Edit: 25 / September / 2008, 21:16:09 by PhyrePhoX »
CHDK Developer.

Offline GrAnd

  • Developers
  • Hero Member
  • ****
  • Posts: 916
  • [A610, S3IS]
    • CHDK
Re: Autobuilding of trunk
« Reply #1 on: 22 / March / 2008, 02:25:34 »
Here is the script in case somebody wish to improve it :D
Code: TCL
  1. #!/bin/sh
  2. # \
  3. exec tclsh $0 $*
  4.  
  5. #-------------------------------------------------------------------
  6. # PATH settings
  7. set BUILD_PATH      "/project/home/grand/chdk/autobuild"
  8. set BUILD_LOCK_FILE "${BUILD_PATH}/.build_lock"
  9. set LAST_REV_FILE   "${BUILD_PATH}/.last_revision"
  10. set LOG_FILE        "${BUILD_PATH}/compile.log"
  11. set COMPILER_PATH   "/project/home/grand/chdk/gcc/arm-elf/bin"
  12.  
  13. #-------------------------------------------------------------------
  14. # List of branches to automatic rebuild
  15. set BRANCHES_TO_BUILD [list "trunk"]
  16.  
  17. #-------------------------------------------------------------------
  18.  
  19.  
  20.  
  21. #-------------------------------------------------------------------
  22. proc run {cmd resVar} {
  23.     upvar $resVar x
  24.     if {[catch {set res [eval exec $cmd 2>@ stdout]} errMsg]} {
  25.         set x $errMsg
  26.         return 0
  27.     } else {
  28.         set x $res
  29.         return 1
  30.     }
  31. }
  32.  
  33. #-------------------------------------------------------------------
  34. proc send {file resVar} {
  35.     upvar $resVar x
  36.     set ok 0
  37.     set USER "**********"
  38.     set PASSWD "***********"
  39.  
  40.     lappend cmds "user $USER $PASSWD"
  41.     lappend cmds "cd hdk/autobuild"
  42.     lappend cmds "binary"
  43.     lappend cmds "mdelete $file"
  44.     lappend cmds "mput $file"
  45.     lappend cmds "pwd"
  46.     lappend cmds "quit"
  47.  
  48.     if {![catch {set f [open "|ftp -n -i ftp.newmail.ru 2>@ stdout" "w+"]} res]} {
  49.         puts $f [join $cmds "\n"]
  50.         flush $f
  51.         set res ""
  52.         while {![eof $f]} {
  53.             gets $f ll
  54.             lappend res $ll
  55.         }
  56.         if {![catch {close $f} res2]} {
  57.             foreach ll $res {
  58.                 if {[regexp -- {hdk/autobuild} $ll]} {
  59.                     set ok 1
  60.                     break
  61.                 }
  62.             }
  63.         } else {
  64.             lappend res $res2
  65.         }
  66.     }
  67.     set x $res
  68.     return $ok
  69. }
  70.  
  71. #-------------------------------------------------------------------
  72. proc finish {err} {
  73.     global BUILD_PATH BUILD_LOCK_FILE LOG_FILE info log
  74.  
  75.     # Save last compiled revision
  76.     if {![catch {set f [open $LOG_FILE "w"]}]} {
  77.         if {$err} {
  78.             #errors
  79.             puts $f "Errors occured during build process: $err."
  80.         } else {
  81.             puts $f "Build process finished normally."
  82.         }
  83.  
  84.         puts $f "================================================\n"
  85.         puts $f [join $log "\n++++++++++++++++++++++++++++++++++++++++++++++++\n"]
  86.         catch {close $f}
  87.         set res ""
  88.         cd $BUILD_PATH
  89.         send [file tail $LOG_FILE] res
  90.     }
  91.  
  92.     catch {file delete $BUILD_LOCK_FILE}
  93.     exit
  94. }
  95.  
  96.  
  97.  
  98. #-------------------------------------------------------------------
  99. # Check for lock-file and wait until lock is away
  100. while {[file exists $BUILD_LOCK_FILE]} {
  101.     lappend log "Lock is detected. Waiting..."
  102.     catch {exec sleep 1m}
  103. }
  104.  
  105. # Create lock-file
  106. #!!!catch {set f [open $BUILD_LOCK_FILE "w"]; close $f}
  107.  
  108. set errors 0
  109. set info(last_revision) 0
  110.  
  111. # Get last compiled revision
  112. if {![catch {set f [open $LAST_REV_FILE "r"]}]} {
  113.     catch {gets $f info(last_revision)}
  114.     catch {close $f}
  115. }
  116.  
  117. # Get last revision info
  118. lappend log "Getting last revision info:"
  119. set ok [run [list svn log --verbose -r HEAD "http://tools.assembla.com/svn/chdk/"] res]
  120. lappend log $res
  121. if {!$ok} {
  122.     incr errors
  123.     lappend log "***FATAL***: Cannot obtain last revision information."
  124.     finish $errors
  125. }
  126. set res [split $res "\n"]
  127.  
  128. # Parse the revision information
  129. # r243 | grand | 2007-12-07 19:18:54 +0300 (Fri, 07 Dec 2007) | 1 line
  130. set info(info) [lindex $res 1]
  131. set list [split $info(info) "|"]
  132. foreach name {revision person date changes} value $list {
  133.     set info($name) [string trim $value]
  134. }
  135. set info(revision) [string trim $info(revision) "rR"]
  136.  
  137. if {$info(revision) == $info(last_revision)} {
  138.     exit
  139. }
  140.  
  141. # Search for branch names
  142. foreach str [lrange $res 3 end] {
  143.     if {[regexp -- " \/(\[^\/ \]+)\[\/ \]" $str dummy branch]} {
  144.         if {$branch == "branches"} {
  145.             if {[regexp -- " \/(\[^\/\]+\/\[^\/ \]+)\[\/ \]" $str dummy branch]} {
  146.                 set branches($branch) ""
  147.             }
  148.         } else {
  149.             set branches($branch) ""
  150.         }
  151.     }
  152. }
  153.  
  154. set env(PATH) "${COMPILER_PATH}:/bin:/usr/bin:$env(PATH)"
  155. #set env(HOME) "/var/www"
  156.  
  157. # Build only required branches
  158. foreach branch $BRANCHES_TO_BUILD {
  159.     if {[info exists branches($branch)]} {
  160.         lappend log "\n"
  161.         lappend log "Removing old sources of branch: '${branch}'"
  162.         catch {cd $BUILD_PATH}
  163.         catch {file delete -force -- "${BUILD_PATH}/${branch}"}
  164.         # Get sources
  165.         lappend log "Getting the sources for branch: '${branch}', revision: $info(revision)"
  166.         set ok [run [list svn export -r $info(revision) "[url=http://tools.assembla.com/svn/chdk/]Revision 338: /[/url]${branch}"] res]
  167.         lappend log $res
  168.         if {!$ok} {
  169.             incr errors
  170.             lappend log "***FATAL***: Cannot get sources."
  171.             continue
  172.         }
  173.        
  174.         lappend log "Compiling branch '${branch}'..."
  175.         if {[catch {cd "${BUILD_PATH}/${branch}"}]} {
  176.             incr errors
  177.             lappend log "***ERROR***: cannot change the current directory to '${branch}'."
  178.             continue
  179.         }
  180.  
  181.         set ok [run [list make NO_INC_BUILD=1 batch-zip] res]
  182.         lappend log $res
  183.         if {!$ok} {
  184.             incr errors
  185.             lappend log "***FATAL***: Error occured during compilation."
  186.             continue
  187.         }
  188.  
  189.         lappend log "Uploading files..."
  190.         if {[catch {cd "${BUILD_PATH}/${branch}/bin"}]} {
  191.             incr errors
  192.             lappend log "***ERROR***: cannot change the current directory to '${branch}/bin'."
  193.             continue
  194.         }
  195.         set ok [send "*.zip" res]
  196.         lappend log $res
  197.         if {!$ok} {
  198.             incr errors
  199.             lappend log "***FATAL***: Cannot upload files for branch '${branch}'."
  200.             continue
  201.         }
  202.     }
  203. }
  204.  
  205. # Save last compiled revision
  206. if {![catch {set f [open $LAST_REV_FILE "w"]}]} {
  207.     catch {puts $f $info(revision)}
  208.     catch {close $f}
  209. }
  210.  
  211. finish $errors
  212.  
CHDK Developer.

Offline ewavr

  • Developers
  • Hero Member
  • ****
  • Posts: 1057
  • A710IS
Re: Autobuilding of trunk
« Reply #2 on: 22 / March / 2008, 02:39:37 »
It is possible to supress warnings:
Warning: File `******' has modification time #### s in the future 
warning:  Clock skew detected.  Your build may be incomplete.
?



Offline GrAnd

  • Developers
  • Hero Member
  • ****
  • Posts: 916
  • [A610, S3IS]
    • CHDK
Re: Autobuilding of trunk
« Reply #3 on: 22 / March / 2008, 02:44:12 »
It is possible to supress warnings:
Warning: File `******' has modification time #### s in the future 
warning:  Clock skew detected.  Your build may be incomplete.
?

Possible. That "server" just do not sync its time...
CHDK Developer.

Offline PhyrePhoX

  • Global Moderator
  • Guru Member
  • *****
  • Posts: 2254
  • make RAW not WAR
    • PhyreWorX
Re: Autobuilding of trunk
« Reply #4 on: 22 / March / 2008, 03:20:36 »
hey, thats a good birthday present. but hey, it's YOUR birthday, we should be giving YOU something.
here's my present for you:



other then that, very good idea, with the autobuilds.

Offline GrAnd

  • Developers
  • Hero Member
  • ****
  • Posts: 916
  • [A610, S3IS]
    • CHDK
Re: Autobuilding of trunk
« Reply #5 on: 22 / March / 2008, 12:18:49 »
@PhyrePhoX
Thank you for your present. I like it. :)
CHDK Developer.

Offline Barney Fife

  • Hero Member
  • *****
  • Posts: 1157
    • Gay Outdoorsmen
Re: Autobuilding of trunk
« Reply #6 on: 23 / March / 2008, 09:43:31 »
Deleted
« Last Edit: 22 / April / 2008, 23:42:21 by Barney Fife »
[acseven/admin commented out: please refrain from more direct offensive language to any user. FW complaints to me] I felt it imperative to withdraw my TOTAL participation. Nobody has my permission, nor the right, to reinstate MY posts. Make-do with my quoted text in others' replies only. Bye

Offline GrAnd

  • Developers
  • Hero Member
  • ****
  • Posts: 916
  • [A610, S3IS]
    • CHDK
Re: Autobuilding of trunk
« Reply #7 on: 24 / March / 2008, 13:52:37 »
Windows GUI is discussing here - Windows GUI for trunk building
CHDK Developer.

Offline adcz

  • Full Member
  • ***
  • Posts: 151
  • Canon S5 | Before Thou Smiteth Me, Readeth Sig
    • OSFlower
Re: Autobuilding of trunk
« Reply #8 on: 08 / June / 2008, 06:56:07 »
I've set up a script on one server, which checks every 15 minutes for the new revision in svn repository and builds it if it's found. The build takes 5 minutes. Then puts zip-files to http://grandag.nm.ru/hdk/autobuild/download.htm. So in the worst case the new build will be available in 20 minutes (in 5 minutes if the check-in was just before the check).

Very experimental feature...  :D




Oooooh! Now I get it!!! All this time I was thinking...what in the world do they mean by auto-build? Almost asked...then bumped into this thread :) That is so cool :D Can you email/pm/post the script you used? Might come in useful sometime for me :)

Edit: oh woops! never mind :D found it (I really should read before I post more :P)! By the way...what language is this in? Doesn't look like php...is this a cron job?
« Last Edit: 08 / June / 2008, 07:00:02 by adcz »
Email me at adz@jewc.org for questions/comments/etc or aim me at adzempire!
---
STOP!! Before thou smiteth me, please PM me the reason! . Criticism = bad, constructive criticism = good.

If you wanna applaud, no reason necessary, I know I'm wonderful :P

CHDK Forum

Re: Autobuilding of trunk
« Reply #8 on: 08 / June / 2008, 06:56:07 »

Offline GrAnd

  • Developers
  • Hero Member
  • ****
  • Posts: 916
  • [A610, S3IS]
    • CHDK
Re: Autobuilding of trunk
« Reply #9 on: 08 / June / 2008, 09:12:03 »
By the way...what language is this in? Doesn't look like php...is this a cron job?


Tcl. Yes, it's executed as a cron job.
CHDK Developer.

Offline geekmug

  • Rookie
  • *
  • Posts: 10
Re: Autobuilding of trunk
« Reply #10 on: 07 / August / 2008, 23:19:58 »
An alternative to the cronjob would be to have whoever is hosting the SVN repo setup a post-commit hook that triggers a build. But, since GrAnd's builds are hosted on another server, I would guess the SVN hosting server is not capable of building the repo for everyone.

Offline PhyrePhoX

  • Global Moderator
  • Guru Member
  • *****
  • Posts: 2254
  • make RAW not WAR
    • PhyreWorX
Re: Autobuilding of trunk
« Reply #11 on: 25 / September / 2008, 21:16:44 »
updated first post to include Hackis Autobuild Server.

Offline PhyrePhoX

  • Global Moderator
  • Guru Member
  • *****
  • Posts: 2254
  • make RAW not WAR
    • PhyreWorX
Re: Autobuilding of trunk
« Reply #12 on: 09 / October / 2008, 22:23:04 »
update: as of now grands autobuild server seems to not be working anymore, as it is stuck at revision #527. i informed him of this matter. for now use hackis autobuild server please :)

Offline PhyrePhoX

  • Global Moderator
  • Guru Member
  • *****
  • Posts: 2254
  • make RAW not WAR
    • PhyreWorX
Re: Autobuilding of trunk
« Reply #13 on: 29 / October / 2008, 02:23:11 »
grands autobuild still is defect, he doesnt seem to read his mails either :(

anyhow, Hacki just told me in IRC some interesting data:

around 55.000 downloads have been leeched off of his autobuild server since it started (about 3 months ago).
this amounts to a total of 24 GB downloaded. gigabyte.

i think this is impressive (even if you consider the fact that many people regularly update their version).

thanks Hacki for the hosting!

Offline user1

  • Rookie
  • *
  • Posts: 20
Re: Autobuilding of trunk
« Reply #14 on: 16 / February / 2009, 21:32:58 »
mighty-hoernsche._de reported now: "Errors occured during this build! Old version available here".
But old version are too old (#622)  :(


 


SimplePortal 2.3.3 © 2008-2010, SimplePortal