Re: [Tails-dev] Testing with openqa?

On 09/15/2015 11:03 AM, intrigeri wrote:
> Alan wrote (23 Aug 2015 13:28:41 GMT) :
>> but I encourage the people working on the test suite to have a look
>> at OpenQA and consider wether it would make sense to use it. It has
>> a nice web interface to update screenshots,

Sorry for not responding earlier!

I looked at OpenQA ~4 years ago, before we started our own thing, but it
seemed to me that it solved quite little of what we actually needed, at
least on the top; it didn't support fine-grained hardware configuration
(and not hotplugging at all), for instance. I felt weary of using
something that we'd have to extend endlessly to make it useful for us,
and while it would have been a potentially better longer-term solution,
I'm not sure we'd have started to benefit from it even now. (Also, perl!)

When bertagaz made the initial version of the cucumber + sikuli +
libvirt based thing we have now, I was blown away at how easily
extensible it was (despite me having no previous experience with Ruby).
I still think that is true, despite some painful surprises (JRuby). We
need so much flexibility to do all the things we want to do, so I'm not
sure if anything else than a solution of our own would work efficiently.

Or maybe I'm just having a case of NIH syndrome. :)

>> and we would share maintenance with other distros.

This I find really sad, however.

> I've had a look during DebConf, and indeed the web interface for
> developers is much better than what Jenkins will give us as-is.
> Perhaps at some point we'll need $something that extracts data from
> Jenkins and presents it to developers, and at that point it may be
> worth looking into OpenQA.

I'm unsure what the "$something that extracts data from Jenkins and
presents it to developers" part refers to. If it refers to Alan's "it
has a nice web interface to update screenshots", then I don't see the
benefit in it. It's quite rare that only pictures need updating when
stuff changes, and when they do, often more than one (used in the same
scenario) needs updating. Our --retry-find option will let us solve that
in (ideally) one babysitted run, while relying on automation + a web gui
to update images could result in some crazy long back and forth, or that
you must run a VM in parallel to take screenshots from. I'm not
convinced we'll have much use of that feature except for trivial cases
where only one image needs updating. And in trivial cases we can just
extract the new image from the test suite failure screenshot.

> Note that tests are written directly in Perl, with no overlay language
> like Gherkin. Not as if we were taking much benefit from Gherkin for
> {intra,inter}-team communication yet, but I like it that we're able to;
> and I intend to try to use Gherkin more in the future when discussing
> changes with UX folks.

I personally like the clarity of what is (well, *should be*) going on
that they provide to me, as a developer, even without the "easier
communication with non-developers" part.

Cheers!