Re: [Tails-dev] Releasing automated tests

anonym:
> sajolida:
>> anonym:
>>> [Moving discussion to tails-dev@]
>>
>> Meta: I really don't want to understand everything that's in this email
>> but I felt you would want me to answer this one. But if you think that
>> you can have this discussion without me I would be super happy as well.
>
> I believe you have answered the question that was (mostly) directed to
> you, but you also added an interesting idea, so... :)
>
>>> Given the trimming that has happened, some context may have been lost.
>>> The discussion is about that we now, in our Jenkins setup, automatically
>>> test images built from doc/ and web/ branches, which wastes a lot of
>>> time on our isotesters.
>>>
>>>> From: intrigeri <intrigeri@???>
>>>> Date: Tue, 20 Oct 2015 13:08:31 +0200
>>>>
>>>>> From: bertagaz <bertagaz@???>
>>>>> Date: Mon, 19 Oct 2015 12:29:26 +0200
>>>>
>>>>>> From: anonym <anonym@???>
>>>>>
>>>>>> Still, once we release 1.7, then all doc/ and web/ branches will become
>>>>>> tested. I suspect we will need a permanent fix for only building (not
>>>>>> testing) these branches -- it's useless to test them 99.9% of the time,
>>>>>> and they will block (for ~5 hours) test runs from relevant branches that
>>>>>> got something pushed to them right after them.
>>>>
>>>>> That's something we didn't decide when during the design round. Sounds
>>>>> doable, but I wonder if there are still some valid points to still test
>>>>> that branches.
>>>
>>> True, but there's an overwhelming amount of them, and their
>>> modifications are limited to something that is completely isolated from
>>> most of Tails, the OS, meaning that a huge part of each test run on them
>>> is just a (possibly out-dated) re-run of master/devel/stable depending
>>> on which branch it was based on. That is unlike feature/, bugfix/ and
>>> test/ branches, that need a full run in general. Perhaps you can see
>>> where I'm going:
>>>
>>> As an optimization, we could introduce @web and @doc tags, and run the
>>> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
>>> respectively. Then we could even have automated tests of @web changes
>>> before deploying them by browsing the local wiki in Tails. :)
>>>
>>> Note that I may not have the correct understanding of the doc/ vs web/
>>> distinction, so I'd like a clarification just in case so we don't design
>>> something stupid. I suspect that since we don't have any automated tests
>>> for the *website* (as opposed to the docs) we only care about doc/ and
>>> only need to test web/ if we want to start testing the website.
>>>
>>>> FTR I dislike the idea of blacklisting such branches based on their
>>>> name. I'm not going to debate it here [...]
>>
>> The prefixes doc/ and web/ are used loosely to differentiate work on the
>> "documentation" (/doc /support) and the "website" in general (structure,
>> stuff not in /doc, etc.) but the difference is not strict.
>
> ACK, as I expected, than.
>
>> I also don't think they should be tested. Maybe you could exclude them
>> from testing by their diff to their base branch. If all the diff is
>> under wiki/src then don't test that branch.
>
> I guess you mean the diff against the base branch (but base branches
> themselves would *always* build, of course). Hm. Technically we'd have
> to do something slightly different since a `git diff` would show changes
> in the base branch since the point they diverged. We'd have to look at
> all files touches in `git log $base_branch..` or something like that.
> It's an interesting approach, which I think I like.

Did you took into account the '...' (THREE DOTS) operator which is
slightly different than '..' and I *think* might be helpful here to diff
only the changes that happened on this branch. I'm not sure what it does
exactly (couldn't understand the man page).

>>>> [...] making sure that the workaround is not worst than the problem
>>>> it fixes
>>>
>>> The only effect should be that we won't get automated tests of the few
>>> scenarios that looks at the wiki shipped inside Tails. [...]
>>
>> Agreed. If I understand correctly, these scenarios have a *dependency*
>> on what is on the local copy of the website, but they are actually
>> testing the Tails software (the "Report an Error" launcher for example).
>
> Correct so far.
>
>> They are not testing the content of the local copy of the website itself.
>
> Actually, they *are* testing the content of the local copy, e.g. that
> the support page is properly localized. Hence there could be a subset of
> automated tests that would make sense to run for doc/ and web/ branches.

I didn't know that, sorry!