Re: [Tails-dev] Releasing automated tests

sajolida:
> anonym:
>> [Moving discussion to tails-dev@]
>
> Meta: I really don't want to understand everything that's in this email
> but I felt you would want me to answer this one. But if you think that
> you can have this discussion without me I would be super happy as well.

I believe you have answered the question that was (mostly) directed to
you, but you also added an interesting idea, so... :)

>> Given the trimming that has happened, some context may have been lost.
>> The discussion is about that we now, in our Jenkins setup, automatically
>> test images built from doc/ and web/ branches, which wastes a lot of
>> time on our isotesters.
>>
>>> From: intrigeri <intrigeri@???>
>>> Date: Tue, 20 Oct 2015 13:08:31 +0200
>>>
>>>> From: bertagaz <bertagaz@???>
>>>> Date: Mon, 19 Oct 2015 12:29:26 +0200
>>>
>>>>> From: anonym <anonym@???>
>>>>
>>>>> Still, once we release 1.7, then all doc/ and web/ branches will become
>>>>> tested. I suspect we will need a permanent fix for only building (not
>>>>> testing) these branches -- it's useless to test them 99.9% of the time,
>>>>> and they will block (for ~5 hours) test runs from relevant branches that
>>>>> got something pushed to them right after them.
>>>
>>>> That's something we didn't decide when during the design round. Sounds
>>>> doable, but I wonder if there are still some valid points to still test
>>>> that branches.
>>
>> True, but there's an overwhelming amount of them, and their
>> modifications are limited to something that is completely isolated from
>> most of Tails, the OS, meaning that a huge part of each test run on them
>> is just a (possibly out-dated) re-run of master/devel/stable depending
>> on which branch it was based on. That is unlike feature/, bugfix/ and
>> test/ branches, that need a full run in general. Perhaps you can see
>> where I'm going:
>>
>> As an optimization, we could introduce @web and @doc tags, and run the
>> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
>> respectively. Then we could even have automated tests of @web changes
>> before deploying them by browsing the local wiki in Tails. :)
>>
>> Note that I may not have the correct understanding of the doc/ vs web/
>> distinction, so I'd like a clarification just in case so we don't design
>> something stupid. I suspect that since we don't have any automated tests
>> for the *website* (as opposed to the docs) we only care about doc/ and
>> only need to test web/ if we want to start testing the website.
>>
>>> FTR I dislike the idea of blacklisting such branches based on their
>>> name. I'm not going to debate it here [...]
>
> The prefixes doc/ and web/ are used loosely to differentiate work on the
> "documentation" (/doc /support) and the "website" in general (structure,
> stuff not in /doc, etc.) but the difference is not strict.

ACK, as I expected, than.

> I also don't think they should be tested. Maybe you could exclude them
> from testing by their diff to their base branch. If all the diff is
> under wiki/src then don't test that branch.

I guess you mean the diff against the base branch (but base branches
themselves would *always* build, of course). Hm. Technically we'd have
to do something slightly different since a `git diff` would show changes
in the base branch since the point they diverged. We'd have to look at
all files touches in `git log $base_branch..` or something like that.
It's an interesting approach, which I think I like.

>>> [...] making sure that the workaround is not worst than the problem
>>> it fixes
>>
>> The only effect should be that we won't get automated tests of the few
>> scenarios that looks at the wiki shipped inside Tails. [...]
>
> Agreed. If I understand correctly, these scenarios have a *dependency*
> on what is on the local copy of the website, but they are actually
> testing the Tails software (the "Report an Error" launcher for example).

Correct so far.

> They are not testing the content of the local copy of the website itself.

Actually, they *are* testing the content of the local copy, e.g. that
the support page is properly localized. Hence there could be a subset of
automated tests that would make sense to run for doc/ and web/ branches.

Cheers!