[Moving discussion to tails-dev@]
Given the trimming that has happened, some context may have been lost.
The discussion is about that we now, in our Jenkins setup, automatically
test images built from doc/ and web/ branches, which wastes a lot of
time on our isotesters.
> From: intrigeri <intrigeri@???>
> Date: Tue, 20 Oct 2015 13:08:31 +0200
>
>> From: bertagaz <bertagaz@???>
>> Date: Mon, 19 Oct 2015 12:29:26 +0200
>
>>> From: anonym <anonym@???>
>>
>>> Still, once we release 1.7, then all doc/ and web/ branches will become
>>> tested. I suspect we will need a permanent fix for only building (not
>>> testing) these branches -- it's useless to test them 99.9% of the time,
>>> and they will block (for ~5 hours) test runs from relevant branches that
>>> got something pushed to them right after them.
>
>> That's something we didn't decide when during the design round. Sounds
>> doable, but I wonder if there are still some valid points to still test
>> that branches.
True, but there's an overwhelming amount of them, and their
modifications are limited to something that is completely isolated from
most of Tails, the OS, meaning that a huge part of each test run on them
is just a (possibly out-dated) re-run of master/devel/stable depending
on which branch it was based on. That is unlike feature/, bugfix/ and
test/ branches, that need a full run in general. Perhaps you can see
where I'm going:
As an optimization, we could introduce @web and @doc tags, and run the
test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
respectively. Then we could even have automated tests of @web changes
before deploying them by browsing the local wiki in Tails. :)
Note that I may not have the correct understanding of the doc/ vs web/
distinction, so I'd like a clarification just in case so we don't design
something stupid. I suspect that since we don't have any automated tests
for the *website* (as opposed to the docs) we only care about doc/ and
only need to test web/ if we want to start testing the website.
> FTR I dislike the idea of blacklisting such branches based on their
> name. I'm not going to debate it here [...]
Please do it here, then! :) I guess it's related to that you want at
least doc/ branches to be tested, since our automated test suite
actually depend on the documentation (see two quotes below for more on
this).
I wonder if we could have a configuration filem like
config/jenkins_testing, which has settings controlling how the automated
test suite is invoked on Jenkins. I imagine something like this:
ENABLE = {true,false}
Whether to run the test suite at all. Hence, when merging another base
branch into master as part of the release process, we just make sure to
set this to `false` and we're done. When merging master into a base
branch, it must be set to `true`. So it's similar to config/base_branch,
except that it includes master as a special case. Is this problematic?
Won't merge conflicts save us?
If we prefer it, we could implement my tagging idea with this:
CUCUMBER_ARGUMENTS = <string>
This makes Jenkins invoke run_test_suite with extra CUCUMBER_ARGUMENTS.
And it could be used for the @web/@doc idea above if we have it set to
`--tag @doc` or whatever in master. We could even test master then, in
the same way. We just have to update the release process in the same way
as for the ENABLE setting, but for CUCUMBER_ARGUMENTS.
Since this affects how Jenkins will invoke a script, there's a security
concern vs the shell that has to be considered. Also arbitrary paths can
be given. Hence we probably may want to have this option *instead*:
RUN_WITH_TAGS = <list of strings>
So if it's set to ["tag1", "~tag2,tag3"] it would be translated to these
distinct arguments:
['--tag', 'tag1', '--tag', '~tag2,tag3']
in the usual safe (i.e. shell-free) list syntax used for exec in many
programming languages (python, ruby, perl...). This would also improve
the situation for @fragile, and also be subject to the same release
process update as ENABLE above.
<off-topic>
Note that ENABLE gives an alternative implementation for the problem
that #10389 wants to solve, i.e. "don't run tests for WIP branches". I
think I prefer this approach, so we avoid excessive branch re-naming.
Also note that with the flexibility from
CUCUMBER_ARGUMENTS/CUCUMBER_TAGS we can remove `--tag ~@fragile` from
the Jenkins wrapper script (#10288) by setting it in the config instead.
Using CUCUMBER_ARGUMENTS/CUCUMBER_TAGS the true power users could even
do "focus" testing of specific features/scenarios via a temporary tag
and then making Jenkins only test that tag -- now the test suite will
not waste time on other scenarios. FWIW. I don't think this will play
nicely with our future ideas about Jenkins updating tickets in Redmine
given the test results. That could be fixed if we added another option:
JENKINS_REVIEW = {true,false}
It would be true by default (I guess), and would then enable Jenkins
"reviewing" (i.e. ticket updating). So when doing focus testing like
described above, one could set it to false, temporarily. Alternatively
this setting could be removed and its funcitonality implied by jenkins
checking the CUCUMBER_ARGUMENTS, and only update tickets if it is the
expected value(s), e.g. `--tag ~@fragile` (or `--tag @website` or whatever).
Also, per our discussion on #10117 we may want:
OLD_TAILS_ISO = <string>
which sets the test suite's OLD_TAILS_ISO option (via run_test_suite's
`--old-iso`, or features/config). As part of the release process, we
could set this to the old release. In feature branches that we know
there will be issues if the previous release is used could unset this.
One would have to be careful about resetting it after merging such
branches into a base branch, though. I guess we could add a @source test
checks this (i.e. if a base branch is checked out, OLD_TAILS_ISO must be
set, and we can perhaps even parse debian/changelog in some intelligent
way).
Speaking about Jenkins + Redmine, perhaps some additional settings here
can help us with the issues we discussed in the specification thread? I
don't have the context now to come up with something specific, but I
recall something about the need of an extra field in Redmine, which
perhaps would fit better in this config?
We could also add an option for increasing the priority of the tests of
that branch. In combination with "focus testing", this could be useful
for those that cannot run the automated test suite on their own hardware.
More crazy ideas?
</off-topic>
BTW, to ease (= isolate) merge conflicts, separating them to one setting
per file is probably a good idea (think of the reason why we ended up
with config/APT_overlays.d/ instead of the original
config/APT_overlays). Which makes me thinking...
Actually, we probably want to introduce the above settings to
features/config (so we need YAML parsing) instead of
config/jenkins_testing or similar. The above settings (except possibly
OLD_TAILS_ISO, see below) could be prefixed "JENKINS_" or whatever. I
suggest features/config/jenkins.d (which we want to be tracked by Git,
so we have to add a ! for it to .gitignore) which is supported by our
current options parsing inside the test suite. We could leverage that in
interesting ways: we could make the test suite ignore that sub dir
unless some JENKINS envvar (that we set any way in the Jenkins
environment) is set, and then we can also set OLD_TAILS_ISO in there
appropriately (release process++) so Jenkins uses the previous release.
All-in-all, this sounds pretty interesting to me.
Ok, let's forget about this now and return to the "workaround", i.e.
skipping tests depending on branch name:
> Still, this idea is probably OK as a short-term workaround if we need
> it for some reason, but then we need a clear description of the
> problem,
The problem, as I see it, is: we run the full automated test suite for
doc/ and web/ branches, which wastes a lot of valuable isotester time
given that all but a few tests are completely orthogonal to changes in
the wiki sub directory of Tails Git sources.
Given my conclusion in the next quote, I think the branch matching idea
is a reasonable short-term solution that should replace my silly
features/tao.feature hack that currently is deployed ASAP.
> [...] making sure that the workaround is not worst than the problem
> it fixes
The only effect should be that we won't get automated tests of the few
scenarios that looks at the wiki shipped inside Tails. I think that
currently is one scenario: The Report an Error launcher will open the
support documentation in supported non-English locales. So, if some wiki
change makes the 'the support documentation page opens in Tor Browser'
step fail. The only way that can happen is if:
* the url to the "Help and Support" section changes so the desktop
shortcut is wrong
* there's a change to the "Help and Support" section, or its German
translation
* we change the website layout of the "parentlinks" thing, or the "home"
icon there
Only in the first case would our test suite find an actual error, the
other two just implies that our test is now out of sync and must be
updated. Clearly it's not worth wasting that much isotester time on this
at the moment.
> [...] and then some timeline + tickets + assignees to find a real
> solution (my intuition tells me that we'll want 1. prioritization +
> quiet period based on our domain knowledge; and then 2. more hardware;
> not that the order matters, because fixing the problem described by
> anonym with hardware only would require a fuckin lot of powerful gear
> that'll be sleeping most of the time, which would be kinda stupid; but
> anyway, let's focus on the defining the timeline and responsibilities,
> not on what the solution might be).
Right. Let's first investigate the options we have.
Cheers!
