Re: [Tails-dev] Releasing automated tests

anonym:
> [Moving discussion to tails-dev@]

Meta: I really don't want to understand everything that's in this email
but I felt you would want me to answer this one. But if you think that
you can have this discussion without me I would be super happy as well.

> Given the trimming that has happened, some context may have been lost.
> The discussion is about that we now, in our Jenkins setup, automatically
> test images built from doc/ and web/ branches, which wastes a lot of
> time on our isotesters.
>
>> From: intrigeri <intrigeri@???>
>> Date: Tue, 20 Oct 2015 13:08:31 +0200
>>
>>> From: bertagaz <bertagaz@???>
>>> Date: Mon, 19 Oct 2015 12:29:26 +0200
>>
>>>> From: anonym <anonym@???>
>>>
>>>> Still, once we release 1.7, then all doc/ and web/ branches will become
>>>> tested. I suspect we will need a permanent fix for only building (not
>>>> testing) these branches -- it's useless to test them 99.9% of the time,
>>>> and they will block (for ~5 hours) test runs from relevant branches that
>>>> got something pushed to them right after them.
>>
>>> That's something we didn't decide when during the design round. Sounds
>>> doable, but I wonder if there are still some valid points to still test
>>> that branches.
>
> True, but there's an overwhelming amount of them, and their
> modifications are limited to something that is completely isolated from
> most of Tails, the OS, meaning that a huge part of each test run on them
> is just a (possibly out-dated) re-run of master/devel/stable depending
> on which branch it was based on. That is unlike feature/, bugfix/ and
> test/ branches, that need a full run in general. Perhaps you can see
> where I'm going:
>
> As an optimization, we could introduce @web and @doc tags, and run the
> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
> respectively. Then we could even have automated tests of @web changes
> before deploying them by browsing the local wiki in Tails. :)
>
> Note that I may not have the correct understanding of the doc/ vs web/
> distinction, so I'd like a clarification just in case so we don't design
> something stupid. I suspect that since we don't have any automated tests
> for the *website* (as opposed to the docs) we only care about doc/ and
> only need to test web/ if we want to start testing the website.
>
>> FTR I dislike the idea of blacklisting such branches based on their
>> name. I'm not going to debate it here [...]

The prefixes doc/ and web/ are used loosely to differentiate work on the
"documentation" (/doc /support) and the "website" in general (structure,
stuff not in /doc, etc.) but the difference is not strict.

I also don't think they should be tested. Maybe you could exclude them
from testing by their diff to their base branch. If all the diff is
under wiki/src then don't test that branch.

> Please do it here, then! :) I guess it's related to that you want at
> least doc/ branches to be tested, since our automated test suite
> actually depend on the documentation (see two quotes below for more on
> this).
>

[...] Skipping tons of stuff I don't understand.

>> [...] making sure that the workaround is not worst than the problem
>> it fixes
>
> The only effect should be that we won't get automated tests of the few
> scenarios that looks at the wiki shipped inside Tails. I think that
> currently is one scenario: The Report an Error launcher will open the
> support documentation in supported non-English locales. So, if some wiki
> change makes the 'the support documentation page opens in Tor Browser'
> step fail. The only way that can happen is if:
>
> * the url to the "Help and Support" section changes so the desktop
> shortcut is wrong
>
> * there's a change to the "Help and Support" section, or its German
> translation
>
> * we change the website layout of the "parentlinks" thing, or the "home"
> icon there
>
> Only in the first case would our test suite find an actual error, the
> other two just implies that our test is now out of sync and must be
> updated. Clearly it's not worth wasting that much isotester time on this
> at the moment.

Agreed. If I understand correctly, these scenarios have a *dependency*
on what is on the local copy of the website, but they are actually
testing the Tails software (the "Report an Error" launcher for example).
They are not testing the content of the local copy of the website itself.