Hi,
[too bad it's too late to rename this thread, now that there have been
6 replies, to it.]
anonym wrote (21 Oct 2015 12:46:37 GMT) :
>> FTR I dislike the idea of blacklisting such branches based on their
>> name. I'm not going to debate it here [...]
> Please do it here, then! :) I guess it's related to that you want at
> least doc/ branches to be tested, since our automated test suite
> actually depend on the documentation (see two quotes below for more on
> this).
My concerns about skipping branches based on their name are
essentially the same as the ones I've just posted in another
sub-thread of this one
(
https://mailman.boum.org/pipermail/tails-dev/2015-December/009836.html).
Besides, I think they are addressed by the content-based filtering (as
opposed to name-based filtering) that seems to be the most current
proposal in this thread, so no need for me to elaborate about it now.
> Ok, let's forget about this now and return to the "workaround", i.e.
> skipping tests depending on branch name:
>> Still, this idea is probably OK as a short-term workaround if we need
>> it for some reason, but then we need a clear description of the
>> problem,
> The problem, as I see it, is: we run the full automated test suite for
> doc/ and web/ branches, which wastes a lot of valuable isotester time
> given that all but a few tests are completely orthogonal to changes in
> the wiki sub directory of Tails Git sources.
About isotester time, I computed some stats in order to make my
opinion a bit better informed: in November, among 314 non-aborted ISO
test suite runs, 24% (75) were about branches whose name starts with
"doc" or "web". So yes, that's wasting a lot of resources. It will
probably be a problem, we'll see on #9264, so all the work put into
this thread is definitely useful :)
It seems that the proposed solution also addresses the problem that
was raised initially, which was rather about latency for testing
non-doc branches, so it sounds like we can avoid introducing clever
stuff like prioritization + quiet time for a bit longer (#9760 will
likely need it anyway, though). Yay!
>> [...] making sure that the workaround is not worst than the problem
>> it fixes
> The only effect should be that we won't get automated tests of the few
> scenarios that looks at the wiki shipped inside Tails. I think that
> currently is one scenario: The Report an Error launcher will open the
> support documentation in supported non-English locales.
> So, if some wiki
> change makes the 'the support documentation page opens in Tor Browser'
> step fail. The only way that can happen is if:
> [...]
> Only in the first case would our test suite find an actual error, the
> other two just implies that our test is now out of sync and must be
> updated. Clearly it's not worth wasting that much isotester time on this
> at the moment.
Thanks for gathering all the data I need to be convinced :)
I'll comment on this topic, about the actual proposed solution, later
on this thread if needed.
Cheers,
--
intrigeri
