Hi,
anonym wrote (24 Mar 2015 16:50:06 GMT) :
> However, I think the reason I asked for this feature was to trigger
> rebuilds when the feature branch's APT suite has been updated. From
> reading the blueprint it only mentions "Git", so I assume the "watchdog"
> won't look at the feature branch's APT suite state?
Short-term mitigation:
1. if anyone really want to trigger an immediate rebuild, they can do
it manually in the Jenkins interface (people who have upload
rights to our APT repo also have powerful-enough access to Jenkins
to trigger builds);
2. the proposal says that all active branches are built daily, in
addition to post-Git-push => worst case, the branch will be
rebuilt within 24h;
3. if in a hurry, or for whatever reason, you can still force
a rebuild by pushing a dummy commit (ugly, but it works).
Long-term: it seems quite clear to me that any upload to an APT suite
should trigger a rebuild of the *directly* affected base branch.
And also, more generally: at some point, whenever a base branch's
current build is marked as outdated and needing to be rebuilt, be it
because the base branch was updated in Git or via its APT suite, we'll
want to trigger rebuilds of indirectly affected active branches as
well (e.g. topic branches based on that base branch) somehow.
Note that our resource estimates (and thus, our last round of hardware
shopping) didn't take this cascade of triggered builds, so we simply
can't implement that at the moment.
BTW, I've read a bit about Zuul
(
http://ci.openstack.org/zuul/zuul.html) recently, and this made me
aware of quite a few issues similar to the one you're raising here.
Lots of food for thought, forwarded to bertagaz already.
Now, let's put things back into perspective: what bertagaz and Jurre
are working on so far is expending what we currently have to all
active branches. Of course it doesn't cover everything that should
ideally be done yet, simply because what we have so far itself
doesn't. That's merely yet another step towards implementing the ideal
CI system we need :)
=> bertagaz and Jurre, may you please capture this problem, and the
long-term solution ideas we had, in the "Future ideas" section of
the blueprint?
On a more general note, all this discussion and thinking convinces me
more and more that we should really capture in Git everything that
matters for a given branch/build, including the state of the custom
source packages it needs. Lots of technical details (Git submodules,
when/where packages shall be built, etc.) clearly need more thinking,
but the general concept, once implemented, will make most of this
discussion moot.
> Another thing that *just* struck me (and has a lot of consequences...
> sorry for coming with this at this late stage) is that updates to the
> base branch's APT suite will affect all feature branches based on it, so
> a rebuild for all of them would make sense immediately after such an
> update.
I agree in principle, but we can't do the "immediately" part yet, as
explained above in more general terms (since the same is true for Git
changes in a base branch).
> OTOH, they may lack important Git state (since the base branch
> wasn't merged back) that otherwise will make the builds break. Luckily,
> if we have #8654 and never modify any base branch's APT suite directly
> but only add APT suites to the base branche's config file *in* *Git*, we
> should be fine since a Git merge into the feature branch is needed to
> make it aware of the APT changes.
(In what follows, I'm assuming we will have #8654 soonish, since
otherwise the rest of the extended autobuild system wouldn't be
very useful.)
I don't get what's the problem you're raising here. My understanding
is that there are few potential problems:
a) Changes made directly in the base branch' APT suite are not taken
into account by a rebuild of a topic branch.
b) Changes made directly in the base branch' APT suite don't trigger
a rebuild of the affected topic branches.
c) Changes made in the base branch' APT suites config file are not
taken into account by a rebuild of a topic branch.
d) Changes made in the base branch' APT suites config file don't
trigger a rebuild of the affected topic branches.
Triggering rebuilds (b and d) has been discussed above, and in short:
we can't do that yet, too bad.
I believe that (a) and (c) are non-issues once we have #8654.
So, what's the remaining problem we should solve *now*?
> There's a similar case that also needs some consideration, and changes
> in how we work: imagine we have a feature branch F based on base branch
> B. We upload a package to F's APT suite, all looks good at the moment,
> and we merge F into B. Then a feature branch E has B merged into. All
> good so far. However, at this point we discover an issue with F and have
> to rebuild the package, or otherwise change F's APT suite. Then we
> cannot do anything with F's APT suite, because that would affect E
> without it merging B again. Hence, the fixup for F has to be made in a
> new feature branch, say F-fixup. When F-fixup is merged into B, E won't
> be affected (and need a rebuild) until B is merged into E.
I agree with your conclusions, except "E won't be affected (and need
a rebuild) until B is merged into E", which is untrue once we have
#8654. But this only makes things better IMO, so no problem.
> To conclude, we should update our merge policy (or whatever) with the
> following rules:
> * Never upload packages directly to any base branch, not even during
> during release time. Never!
I don't understand why we shouldn't do that => please elaborate.
> * Once a feature branch has been merged into a base branch, the feature
> branch's APT suite should be frozen forever. Any fixups that affect the
> feature branch's APT suite *must* be done in the APT suite of a *new*
> feature branch.
I think that's true not only for its APT suite, but also for its state
in Git: follow-up fixes need new branches. This was captured in
https://tails.boum.org/blueprint/delete_obsolete_Git_branches/
already, for other reasons:
After a Git branch has been merged and we discover that it
introduces a regression, the cleanest way would be to work on
a resolution in a new Git branch, and not reuse the old Git branch.
Hence, we consider it safe to delete branches that were
already merged.
And in any case, it'll be cleaner, and will make the workflow easier
to understand, than reusing an already merged branch (which makes
things confusing to me: was it merged yet, or not? if it has to be
merged again, why does its name embeds the number of a ticket that's
marked as resolved? etc.)
> I also think we have some problems around release time when we merge all
> APT suites of the feature branches that *have* been Git-merged into the
> base branch being released (I'm gonna refer to this procedure simply as
> "squashing" below). Specifically, all feature branches based on the
> to-be-released base branch that has *not* been merged will be affected
> by this. Example:
> 1. Let F1 and F2 be divergent feature branches based on base branch B.
> 2. F1 is merged into B.
> 3. B is released, so F1's APT suite is merged into B's APT suite, i.e.
> the "squashing".
> 4. Note that at no point was F1 or B merged into F2, but right now it's
> like the APT suite of F1 is merged into F2. Also note that F1's Git is
> not merged into F2.
> Now we have two issues:
> * F2 has (indirectly, via the base branch's APT suite) been modified
> without its Git state being changed at all.
It seems that we don't have in mind the same semantics regarding
relationship between a topic branch and the base branch it's based
upon => let me explain how I see it.
The issue you're raising has been here since we have an APT repo, and
I don't see this as a problem at all. A base branch can evolve (be it
in Git or APT), and it does impact all topic branches based on it:
that's why it's called a "base" branch. IMO a topic branch is merely
an overlay on top of a base branch. The fact that it may be lagging
behind it (as far as Git is concerned) at a given point of time is
a mere fact of life (things change), but does not mean that it
shouldn't include recent changes made in its base branch when
building: for all practical purposes I can think of, what we're
interested about is "what product would be built from base branch B if
we merged F2 into it?"
> * However, F2 doesn't have the Git changes made in F1, and it could very
> well be that the APT stuff introduced by F1's APT suite depend on those
> Git changes in a way so that builds will break without them.
That's a non-issue once we have #8654 for all practical purposes I can
think of: building F2 without merging B into it first, doesn't make
much sense to me.
Cheers!
--
intrigeri
