ideas for improving testing

reyalp

13792

ideas for improving testing

« on: 29 / November / 2009, 21:50:00 »

Advertisements

Testing CHDK is hard, because there are many different cameras and no one has all of them. It would be nice to have an easy way to find someone who can test something on a particular camera, or let everyone know that there is a new feature that needs exercising.

Not everyone reads everything in the forum (for example, general users might not pay attention to the development section at all), so just posting in a development thread related to a specific feature is hit or miss for getting someone to test.

I'm not sure how to organize this, but I've had two general ideas:
A (sticky ?) thread where people who are willing to do some testing list what cameras they have. If testing is needed, the developer PMs them.

A (sticky ?) thread for test requests. Basically, each post would just be a link to the thread of whatever it was that needed testing.

Either of these could be done on the wiki instead of the forum. Maybe that would be better, since old stuff could be erased easily.

Thoughts ? Suggestions ?

Logged

Don't forget what the H stands for.

barberofcivil

Re: ideas for improving testing

« Reply #1 on: 29 / November / 2009, 22:24:12 »

Good idea I think.

I think it may be better on the forum as it can be linked to the user for messaging.
What I would suggest is a stickied topic. The top item would list all cameras and known FW versions. Users can submit their names, modesl and versiosn and have them added to the top message if they are interested in testing.
The testing request sticky also sounds like a good idea. I would suggest that anyone who submits to thread #1, should set an email watch to the test request thread.

Logged

mattkime

Re: ideas for improving testing

« Reply #2 on: 29 / November / 2009, 22:49:02 »

i'm not certain of the wiki/forum choice but i do know that any system which would spread testing would be very welcome. it may be necessary to suggest test cases when camera owners aren't familiar or interested in specific features.

seems this might be worthy of a forum specific to testing.

Logged

fe50

3139
IXUS50 & 860, SX10 Star WARs-Star RAWs

Re: ideas for improving testing

« Reply #3 on: 30 / November / 2009, 07:19:30 »

I'd prefer to have it on the wikia, much easier to handle, with history...IMO in the forum this will become chaotic very fast.

Logged

-->FIRMWARE DUMPS<-- * Manual * Lua * me

fudgey

1705
a570is

Re: ideas for improving testing

« Reply #4 on: 30 / November / 2009, 12:43:23 »

http://chdk.setepontos.com/index.php/topic,1838.msg16710.html#msg16710

Indeed something we desperately need. My opinion still is that we should have a way to push these requests to volunteers instead of hoping someone comes and checks.

Subscribing to a specific (developer write only) thread concerning a certain port and enabling emails for new messages is one way, but the more threads there are, the more tedious it is for a developer to mass mail testers for a large number of ports unless we also create threads (and have those same people monitor those) for VxWorks, DryOS etc (there may be a huge amount of useful ways to group them, would need to identify what's most useful).

A single thread for a huge amount of ports won't be enough because the forum only notifies me of the first unread message (and if it doesn't concern your camera, why would you check back?). I could change that, but I believe that option is global (affecting all threads I have notify enabled for) so I'd be receiving a ton of messages and turning the dang thing off.

IMO, volunteers shouldn't be forced to make their effor public (e.g. to prevent random people in-forum from choosing to spam them for help) or permanent -- we should just gather as many capable people to receive our requests so that they can act whenever they feel they can help. Each request will naturally need to have a forum link (or something similar) where someone can post their results (or just an "I'll do it, wait a sec") to prevent unnecessary double work.

Logged

reyalp

13792

Re: ideas for improving testing

« Reply #5 on: 30 / November / 2009, 18:18:05 »

Agree with all of that fudgey.

A couple further thoughts.
There aren't currently a huge number of requests. One per day would be a lot more than I'd expect to generate.

I'd rather avoid having yet another medium (i.e. mailing list) unless it was really dramatically better than the other options and someone was willing to actively maintain it in whatever way was needed. The mantis tracker suffers from the fact that there doesn't seem to be anyone who really knows the system maintaining it (and I personally find it a much clunkier means of communication than the forum)

The wiki allows you to watch pages. Email notification is again a global option, although you can get a summary instead of individual messages, and you can ignore minor edits.

Hmm, thinking out loud ... probably overkill but you could do this with a very simple custom web app, and produce an rss based on the users selection (I'm thinking foo?dryos+sx10+generic gives you a feed general dryos, sx10 and non-camera specific requests, the site lets you build the query by checking boxes and giving you the url to subscribe to). They could subscribe to in their favorite newsreader, or use http://chdk.wikia.com/wiki/Help:RSS2Wiki to make it appear in a wiki page. You could also use this so that each cameras page kept a feed of active requests for that camera. Each item would just be a link to the relevant forum post.

Logged

Don't forget what the H stands for.

mattkime

Re: ideas for improving testing

« Reply #6 on: 30 / November / 2009, 23:01:37 »

I tilt toward using mailing list mainly because its a more familiar medium. the main problem i see with a mailing list is that we'd need _a_lot_ of lists if we wanted to send out notices regarding specific models.

this starts to get into contact management territory.

If I understand correctly, we want to notify testers in the following groups -

All Testers
By Digic Chip / OS
By Model

Logged

fudgey

1705
a570is

Re: ideas for improving testing

« Reply #7 on: 01 / December / 2009, 17:47:19 »

If global notify options are our only problem with the forum (or wiki), could we create a separate forum (or wiki site) for this? It would be lower in message volume (being developer moderated) and shouldn't attract a huge number of members as long as its attachments are downloadable without registration.

mattkime: categories that might end up being useful are vast...chip/os/model yes, but also things like ixus/A/S/SX/G series, reference firmware resemblance grouping, FW update coding style, miscellaneous user interface and feature grouping (video button, swivel screen, super zoom, wheel, external flash, usb trigger voltage, usb wakeup, usb remote...).

The list is near endless but most of this isn't really important as long as messages can be sent and will be received by suitable people who understand that most messages aren't going to be very interesting to them. Reyalp is correct, the volume of messages will likely be pretty low (way less than the number of viagra ads the average Joe receives). Unfinished ports most definitely require a separate threads, but this forum right here suits porting fairly well the way it is now I suppose.

Logged