Hi,
[replying to the kinda off-topic part in a dedicated sub-thread]
anonym wrote (21 Oct 2015 12:46:37 GMT) :
> I wonder if we could have a configuration filem like
> config/jenkins_testing, which has settings controlling how the automated
> test suite is invoked on Jenkins. I imagine something like this:
> ENABLE = {true,false}
> Whether to run the test suite at all. Hence, when merging another base
> branch into master as part of the release process, we just make sure to
> set this to `false` and we're done. When merging master into a base
> branch, it must be set to `true`. So it's similar to config/base_branch,
> except that it includes master as a special case. Is this problematic?
> Won't merge conflicts save us?
> If we prefer it, we could implement my tagging idea with this:
> CUCUMBER_ARGUMENTS = <string>
> This makes Jenkins invoke run_test_suite with extra CUCUMBER_ARGUMENTS.
> And it could be used for the @web/@doc idea above if we have it set to
> `--tag @doc` or whatever in master. We could even test master then, in
> the same way. We just have to update the release process in the same way
> as for the ENABLE setting, but for CUCUMBER_ARGUMENTS.
> Since this affects how Jenkins will invoke a script, there's a security
> concern vs the shell that has to be considered. Also arbitrary paths can
> be given. Hence we probably may want to have this option *instead*:
> RUN_WITH_TAGS = <list of strings>
> So if it's set to ["tag1", "~tag2,tag3"] it would be translated to these
> distinct arguments:
> ['--tag', 'tag1', '--tag', '~tag2,tag3']
> in the usual safe (i.e. shell-free) list syntax used for exec in many
> programming languages (python, ruby, perl...). This would also improve
> the situation for @fragile, and also be subject to the same release
> process update as ENABLE above.
> <off-topic>
> Note that ENABLE gives an alternative implementation for the problem
> that #10389 wants to solve, i.e. "don't run tests for WIP branches". I
> think I prefer this approach, so we avoid excessive branch re-naming.
> Also note that with the flexibility from
> CUCUMBER_ARGUMENTS/CUCUMBER_TAGS we can remove `--tag ~@fragile` from
> the Jenkins wrapper script (#10288) by setting it in the config instead.
> Using CUCUMBER_ARGUMENTS/CUCUMBER_TAGS the true power users could even
> do "focus" testing of specific features/scenarios via a temporary tag
> and then making Jenkins only test that tag -- now the test suite will
> not waste time on other scenarios. FWIW. I don't think this will play
> nicely with our future ideas about Jenkins updating tickets in Redmine
> given the test results. That could be fixed if we added another option:
> JENKINS_REVIEW = {true,false}
> It would be true by default (I guess), and would then enable Jenkins
> "reviewing" (i.e. ticket updating). So when doing focus testing like
> described above, one could set it to false, temporarily. Alternatively
> this setting could be removed and its funcitonality implied by jenkins
> checking the CUCUMBER_ARGUMENTS, and only update tickets if it is the
> expected value(s), e.g. `--tag ~@fragile` (or `--tag @website` or whatever).
> Also, per our discussion on #10117 we may want:
> OLD_TAILS_ISO = <string>
> which sets the test suite's OLD_TAILS_ISO option (via run_test_suite's
> `--old-iso`, or features/config). As part of the release process, we
> could set this to the old release. In feature branches that we know
> there will be issues if the previous release is used could unset this.
> One would have to be careful about resetting it after merging such
> branches into a base branch, though. I guess we could add a @source test
> checks this (i.e. if a base branch is checked out, OLD_TAILS_ISO must be
> set, and we can perhaps even parse debian/changelog in some intelligent
> way).
> Speaking about Jenkins + Redmine, perhaps some additional settings here
> can help us with the issues we discussed in the specification thread? I
> don't have the context now to come up with something specific, but I
> recall something about the need of an extra field in Redmine, which
> perhaps would fit better in this config?
> We could also add an option for increasing the priority of the tests of
> that branch. In combination with "focus testing", this could be useful
> for those that cannot run the automated test suite on their own hardware.
> More crazy ideas?
> </off-topic>
I see all the nice ad-hoc features we could get from encoding such
settings in Git branches.
Let me state some concerns I have (this is absolutely not a veto, all
these can be addressed). If we allow code to specify if/how it's going
to be tested, then:
* If we allow Git branches to specify such things, then "Scenario 1:
reviewer" [1] becomes more complex: the reviewer has to ensure that
no such setting was left by mistake in the branch, before they can
consider an OK from Jenkins as valid. Similarly, the RM somehow has
to make sure that no such setting goes into a release branch
by mistake.
[1]
https://tails.boum.org/blueprint/automated_builds_and_tests/automated_tests_specs/
* When we're further down the road of gatekeeping our main Git
branches with QA tools, we'll need to special-case merge requests so
that all such per-branch settings are ignored, the entire test suite
is run.
* I'm not 100% convinced by the "focussed testing on Jenkins" idea:
I do a lot of focussed testing _locally_, but I like it that Jenkins
is giving me the full regression test suite in the background, so if
I broke something else that I didn't think about and did not include
in my focussed testing, then it'll tell me before I start working
again on this branch tomorrow. Of course it may be a matter of
personal inclination, and nobody forces me to push commits that
enable focussed testing to Jenkins. My concern is: giving incentives
to run only part of the test suite on Jenkins delays when we'll
detect some classes of regressions, and will add pressure and stress
close to deadlines, which is good neither for morale, nor for
collective dynamics, nor for the quality of the stuff we release.
I see Jenkins and our test suite as a way to avoid some of
these problems.
Cheers,
--
intrigeri
