Hi,
here's a report of the second reproducible sprints that just ended.
Ulrike volunteered to handle broader communication about this topic,
so this report is only meant to share the news within our community.
Completed
=========
After many iterations we finally made our ISO image build
reproducibly!
The build environment variations we've tested include: build system
clock (last month, next month; could not test next year yet), number
of CPU cores, CPU brand and model, building in Vagrant or not.
This implied fixing a number of things:
* APT auto-removal file (#11986): patch submitted and accepted
upstream, backported in Tails
* Switched to the new squashfs-tools upstream, that builds SquashFS
in a reproducible manner (#12032).
* Various non-determinism issues in the content of the files included
in our SquashFS, including fixing incorrect metadata in old blog
posts and their translations (#11966 – who would have guessed this
affected build determinism? :)
* Various non-determinism issues in the mtimes of the files included
in our SquashFS, that made not only the SquashFS non-reproducible,
but also made the initrd non-reproducible despite the patches we
sent upstream for initramfs-tools (#12330).
* Drop the "Posted on" timestamp ikiwiki added to some pages on
our website (#11987).
Also:
* Made diffoscope *way* faster when comparing SquashFS'es:
changes made directly upstream
* Improved performance of generating CA certificates databases on
boot (#11971)
In progress
===========
* Review'n'merge the feature/5630-deterministic-builds branch into
feature/stretch: one review happened, now blocked by a couple of
the other WIP items and waiting for a second review, so it's
unlikely these changes make it into 3.0~beta3, but I'm confident
they'll be in 3.0~rc1 (mid-May)!
* Ensure the reproducibly built ISOs pass our test suite (#11983):
done for the subset of tests we run on Jenkins, left to be done for
the other tests. Plus some new failures left to be investigated.
* Build our IUKs reproducibly: branch ready for QA (#11974).
* Avoid boot performance problems while generating the fontconfig
cache: we've optimized this a bit with fancy systemd ordering,
but since then one of us came up with a solution that's probably
better (#11971).
* Lots of progress was made to have static build environments:
- Move the apt-cacher-ng data to a dedicated disk that can be shared
among many Vagrant build VMs (#11979).
- Create and provision a new Vagrant VM for every ISO build
(#11980).
- Switch our Jenkins ISO build system to vagrant-libvirt (#11972).
Next steps are to make the whole thing robust enough both for
developers and for our Jenkins CI environment. We expect this will
be merged and deployed either very soon, or between April 19 and
May 12.
To be done
==========
Not that much as far as we know! See remaining open tickets on
https://labs.riseup.net/code/issues/5630.
Cheers,
--
intrigeri
