mighty-hoernsche.de autobuild retirement - General Discussion and Assistance - CHDK Forum

mighty-hoernsche.de autobuild retirement

  • 19 Replies
  • 1553 Views
*

Offline reyalp

  • ******
  • 14039
mighty-hoernsche.de autobuild retirement
« on: 28 / January / 2023, 19:58:08 »
Advertisements
Hacki informed me that server which hosts the official autobuild on mighty-hoernsche.de is being shut down in the next month or so.

He is not interested in setting up and supporting a new server, which is quite understandable given that he hasn't been much involved in the project for many years. He did offer to redirect the domain if we set up a new one.

I can put something together if needed, but if anyone else (with a history of involvement in the community) is interested in hosting and maintaining it, please post here.

Some notes on requirements:
* People with CHDK commit access should have a way of viewing build logs
* Little other interaction from developers is required. The fi2.inc occasionally needs to be updated, but given the cameras CHDK can possibly support, there should be few if any updates in the future
* The build process inherently allows anyone with CHDK commit access to execute code on the autobuild server, though it would be recorded in commit history
* While the bandwidth / storage requirements are modest, IIRC hacki has noted in the past that some crawlers liked to repeatedly download every zip
* It's possible that Canon will someday get mad and send lawyers after whoever hosts it, though given that we aren't likely to ever support anything after digic 7, I'd suspect the odds are going down
* AFAIK the existing autobuild was on a cron job which checked for new reversions every 15 minutes. I believe assembla does have the capability to call external URLs on commit, but given their general flakiness over the last few years, the time based approach is probably better


My thoughts on implementation:
* I would be tempted to make an autobuild variant of the docker build environment (or a similar container platform), with the binaries uploaded to something that servers static files (like s3)
* This should make it easy to host, and keep server costs minimal. For storage, I think a full batch build is something like 400 MB.
* Rather than generating an html table in the autobuild, I could consider making the autobuild just publish JSON file, which would allow the download UI to be maintained separately, and would also be easily usable by installer tools (but more on that below).
* Regarding download pages, msl pointed me to the chdkde autobuild page at https://chdkde.akalab.de/, which has a javascript (client side only) solution for identifying the camera from exif. This and the list box selection UI seems like an improvement over a giant table.
* If a build fails, the previous build should be kept, rather than presenting broken build page.
* I'd personally be tempted to drop support for the "small" zips.

Installer tool considerations:
Stick and CHIMP presumably expect to parse the HTML table on mighty-hoernsche.de. I don't know how much interest @zeno and @dmitrys have in updating these tools, but it would be nice to keep them working.
Don't forget what the H stands for.

Re: mighty-hoernsche.de autobuild retirement
« Reply #1 on: 28 / January / 2023, 21:01:28 »
Stick and CHIMP presumably expect to parse the HTML table on mighty-hoernsche.de.
CHIMP certainly does, although https://autobuild.akalab.de/release.txt is much preferable.

The CHDK distro metadata is taken from https://github.com/CHDKUtil/CHIMP/blob/master/src/CHIMP/CHIMP/Data/Product/CHDK/distros.json; I'll gladly accept a pull request once a new server is set up (What about hacki.someserver.de BTW?).
« Last Edit: 28 / January / 2023, 21:13:47 by dmitrys »
Author of CHIMP, Canon Hack Installation and Management Platform

*

Offline reyalp

  • ******
  • 14039
Re: mighty-hoernsche.de autobuild retirement
« Reply #2 on: 28 / January / 2023, 21:11:40 »
Stick and CHIMP presumably expect to parse the HTML table on mighty-hoernsche.de.
CHIMP certainly does, although https://autobuild.akalab.de/release.txt is much preferable.
Yes, that seems much simpler.
Quote
(What about hacki.someserver.de BTW?).
I expect that's just an alias of mighty-hoernsche.de and will go away at the same time.
Don't forget what the H stands for.

*

Offline zeno

  • *****
  • 891
Re: mighty-hoernsche.de autobuild retirement
« Reply #3 on: 29 / January / 2023, 05:51:53 »
If and when there's an alternative server holding builds I guess I could update STICK and ACID to use it, though at 81 (and several years absence from doing any maintenance) it may take some time!
A570, S100, Ixus 127
Author of ASSIST, STICK, WASP, ACID, SDMInst, LICKS, WICKS, MacBoot, UBDB, CFGEdit


*

Offline reyalp

  • ******
  • 14039
Re: mighty-hoernsche.de autobuild retirement
« Reply #4 on: 09 / February / 2023, 17:17:43 »
For anyone wondering, I'm working on setting up a new autobuild. More soon ;)

If and when there's an alternative server holding builds I guess I could update STICK and ACID to use it, though at 81 (and several years absence from doing any maintenance) it may take some time!
Thanks, I'll ping you when the new server is up. Your contributions over the years are greatly appreciated :)
Don't forget what the H stands for.

Re: mighty-hoernsche.de autobuild retirement
« Reply #5 on: 13 / February / 2023, 12:24:49 »
If and when there's an alternative server holding builds I guess I could update STICK and ACID to use it, though at 81 (and several years absence from doing any maintenance) it may take some time!
Hi Dave.  Having you any thoughts about someday releasing an archive of your source code? Or will your various utilities just go away some sad day in the future?
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline reyalp

  • ******
  • 14039
Re: mighty-hoernsche.de autobuild retirement
« Reply #6 on: 13 / February / 2023, 19:17:34 »
Update: Turn's out Hacki's hosting isn't getting cut off after all, so the mighty-hoernsche.de autobuild will continue to be available for the immediate future.

It has been updated to use a gcc 5 toolchain moved to a new server.  Please post any issues here. The only other change I see is the server appears to force https now.

I mostly done putting together the AWS based implementation I was working on, so I still plan to run that, and will put the docker configuration and scripts in svn.
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14039
Re: mighty-hoernsche.de autobuild retirement
« Reply #7 on: 02 / April / 2023, 22:26:25 »
I mostly done putting together the AWS based implementation I was working on, so I still plan to run that, and will put the docker configuration and scripts in svn.
"mostly done"  :haha :-[
Anyway, after some severe bouts of shipwright's disease, this is now available on https://build.chdk.photos/

The autobuild portion is in the trunk tools directory since r6226. Note the actual image that runs the autobuild is maintained manually, it will not automatically update when the scripts in CHDK SVN are updated.

The front end source is at on github. There's still a few things I want to add (mainly stealing @rudi's model select by exif code), but it should be usable.

Note, if for some reason anyone runs another public instance of this, please update the "about" section to reflect who actually maintains it, or at least not say that I do.

The build uses gcc 8.3.1, and includes FI2 files where supported. If the build fails due to a compile error, the files are not updated, so the most recent successful build will remain available.

The front end is react garbage, so scraping it is definitely not recommended for things like @zeno or @dmitrys install tools. Instead, simple flat file lists are available for each build:
https://build.chdk.photos/builds/trunk/meta/files_full.txt
https://build.chdk.photos/builds/trunk/meta/files_small.txt
and similarly for release under /builds/release/meta

The corresponding zip files are under
/builds/trunk/bin and /builds/release/bin

A file with sha256 sums for all files in the build is also provided, in the usual format, under meta/sha256sums.txt

The front end uses a json file, inspired by / stolen from, but not compatible with, the one used on the chdk de alternate autobuild. The format should be self explanatory (I recommend firefox built in json viewer) and I'll theoretically update the version if the format changes.

Finally build_status.json should reflect whether the most recent build attempt succeeded. If it didn't, you could check the log.

The meta files have permissive CORS headers, so javascript front ends hosted on other sites can use them if they wish.
Don't forget what the H stands for.


*

Offline reyalp

  • ******
  • 14039
Re: mighty-hoernsche.de autobuild retirement
« Reply #8 on: 29 / April / 2023, 00:16:49 »
The https://build.chdk.photos/ now supports identifying model and firmware version by image.

It should handle the ambiguous cases of N / N facebook and Ixus 132 / 135 by prompting the user to select which they have. However I don't have images from any of those cameras. Any samples would be welcome.

When new models are added, the exif model ID will need to be added to the autobuild docker image data/platform_model_info.json, but if it's missing, this should only mean image selection will incorrectly say no build is available for the affected camera, manual selection should still work.

Also, I should note I stole the initial model ID data from the chdkde https://chdkde.akalab.de/ autobuild and haven't attempted to verify all the cameras. It seems correct for all I tested.

The front end now also support linking directly to a specific branch / family / model / firmware, like
https://build.chdk.photos/#build=trunk/D
or
https://build.chdk.photos/#build=release/G/g7x2/110b
Don't forget what the H stands for.

Re: mighty-hoernsche.de autobuild retirement
« Reply #9 on: 01 / May / 2023, 20:06:07 »
It should handle the ambiguous cases of N / N facebook and Ixus 132 / 135 by prompting the user to select which they have. However I don't have images from any of those cameras. Any samples would be welcome.

Powershot N attached.

Edit : I'm guessing you need the original image and not the munged one that gets posted here?
Ported :   A1200    SD940   G10    Powershot N    G16

 

Related Topics