On 25/03/15 16:44, intrigeri wrote:
> Hi,
>
> anonym wrote (24 Mar 2015 16:50:06 GMT) :
>> However, I think the reason I asked for this feature was to trigger
>> rebuilds when the feature branch's APT suite has been updated. From
>> reading the blueprint it only mentions "Git", so I assume the "watchdog"
>> won't look at the feature branch's APT suite state?
>
> Short-term mitigation:
>
> 1. if anyone really want to trigger an immediate rebuild, they can do
> it manually in the Jenkins interface (people who have upload
> rights to our APT repo also have powerful-enough access to Jenkins
> to trigger builds);
Of course! Can't believe I forgot about this!
> [...]
> Long-term: it seems quite clear to me that any upload to an APT suite
> should trigger a rebuild of the *directly* affected base branch.
I assume you didn't intend "base" to be there, right? I mean, whatever
branch we're dealing with, feature or base, it should be rebuilt when
its APT suite is modified. If that's what you meant, great!
> And also, more generally: at some point, whenever a base branch's
> current build is marked as outdated and needing to be rebuilt, be it
> because the base branch was updated in Git or via its APT suite, we'll
> want to trigger rebuilds of indirectly affected active branches as
> well (e.g. topic branches based on that base branch) somehow.
> Note that our resource estimates (and thus, our last round of hardware
> shopping) didn't take this cascade of triggered builds, so we simply
> can't implement that at the moment.
Cool!
[...]
> On a more general note, all this discussion and thinking convinces me
> more and more that we should really capture in Git everything that
> matters for a given branch/build, including the state of the custom
> source packages it needs. Lots of technical details (Git submodules,
> when/where packages shall be built, etc.) clearly need more thinking,
> but the general concept, once implemented, will make most of this
> discussion moot.
Agreed.
>> Another thing that *just* struck me (and has a lot of consequences...
>> sorry for coming with this at this late stage) is that updates to the
>> base branch's APT suite will affect all feature branches based on it, so
>> a rebuild for all of them would make sense immediately after such an
>> update.
>
> I agree in principle, but we can't do the "immediately" part yet, as
> explained above in more general terms (since the same is true for Git
> changes in a base branch).
Right. Take my "immediately":s with a grain of salt.
>> OTOH, they may lack important Git state (since the base branch
>> wasn't merged back) that otherwise will make the builds break. Luckily,
>> if we have #8654 and never modify any base branch's APT suite directly
>> but only add APT suites to the base branche's config file *in* *Git*, we
>> should be fine since a Git merge into the feature branch is needed to
>> make it aware of the APT changes.
>
> (In what follows, I'm assuming we will have #8654 soonish, since
> otherwise the rest of the extended autobuild system wouldn't be
> very useful.)
For the record, everything I wrote was with the assumption that we have
#8654.
> I don't get what's the problem you're raising here. My understanding
> is that there are few potential problems:
>
> a) Changes made directly in the base branch' APT suite are not taken
> into account by a rebuild of a topic branch.
> b) Changes made directly in the base branch' APT suite don't trigger
> a rebuild of the affected topic branches.
> c) Changes made in the base branch' APT suites config file are not
> taken into account by a rebuild of a topic branch.
> d) Changes made in the base branch' APT suites config file don't
> trigger a rebuild of the affected topic branches.
>
> Triggering rebuilds (b and d) has been discussed above, and in short:
> we can't do that yet, too bad.
>
> I believe that (a) and (c) are non-issues once we have #8654.
Urgh. I had forgotten this part of the build specification:
A topic branch may be lagging behind the base branch it's based
upon. What we're interested in is generally not whether a (possibly
outdated) topic branch builds fine, but whether it would build fine
once merged into its base branch:
This should fix all issues I came up with (except that already-merged
branches should be frozen and not reused). Sorry for the noise. I had
already written some answers below before realizing this, so I leave
them, but you can just ignore them.
>> There's a similar case that also needs some consideration, and changes
>> in how we work: imagine we have a feature branch F based on base branch
>> B. We upload a package to F's APT suite, all looks good at the moment,
>> and we merge F into B. Then a feature branch E has B merged into. All
>> good so far. However, at this point we discover an issue with F and have
>> to rebuild the package, or otherwise change F's APT suite. Then we
>> cannot do anything with F's APT suite, because that would affect E
>> without it merging B again. Hence, the fixup for F has to be made in a
>> new feature branch, say F-fixup. When F-fixup is merged into B, E won't
>> be affected (and need a rebuild) until B is merged into E.
>
> I agree with your conclusions, except "E won't be affected (and need
> a rebuild) until B is merged into E", which is untrue once we have
> #8654. But this only makes things better IMO, so no problem.
My understanding is the following, of the scenario I made above:
1 ."we have a feature branch F based on base branch B. We upload a
package to F's APT suite, all looks good at the moment, and we merge F
into B."
At this stage, config/APT_suites in B contains F.
2. "Then a feature branch E has B merged into."
Hence E has F in its config/APT_suites.
3. "at this point we discover an issue with F and have to rebuild the
package, or otherwise change F's APT suite."
Since E has F in its config/APT_suites, any change to F's APT suite will
affect E without any change to E in Git. This would be particularly bad
if some change in F is needed for the new state of F's APT suite to not
break something, e.g. the build itself.
4. "E won't be affected (and need a rebuild) until B is merged into E"
This should be true because no change is made to F's APT suite; the
change is made to F-fixup, but F-fixup is not in E's config/APT_suites,
so E won't be affected. Either B or F-fixup would have to be merged into
E for it to be affected by the fixup.
>> To conclude, we should update our merge policy (or whatever) with the
>> following rules:
>
>> * Never upload packages directly to any base branch, not even during
>> during release time. Never!
>
> I don't understand why we shouldn't do that => please elaborate.
Otherwise all branches based on the base branch will be affected by that
APT suite change despite nothing changing in their Git state. Example
when this goes bad:
Feature branch F is based on base branch B (i.e. B is in F's
config/APT_suites). We push an updated package P into B's APT suite, and
into B's Git we push a commit C that is essential -- without commit C,
installing P will break the build. Now building F is broken, despite
nothing happening to its Git state. We must merge B (or cherry-pick C)
to fix F.
>> * Once a feature branch has been merged into a base branch, the feature
>> branch's APT suite should be frozen forever. Any fixups that affect the
>> feature branch's APT suite *must* be done in the APT suite of a *new*
>> feature branch.
>
> I think that's true not only for its APT suite, but also for its state
> in Git: follow-up fixes need new branches. This was captured in
> https://tails.boum.org/blueprint/delete_obsolete_Git_branches/
> already, for other reasons:
>
> After a Git branch has been merged and we discover that it
> introduces a regression, the cleanest way would be to work on
> a resolution in a new Git branch, and not reuse the old Git branch.
> Hence, we consider it safe to delete branches that were
> already merged.
>
> And in any case, it'll be cleaner, and will make the workflow easier
> to understand, than reusing an already merged branch (which makes
> things confusing to me: was it merged yet, or not? if it has to be
> merged again, why does its name embeds the number of a ticket that's
> marked as resolved? etc.)
Yes, great!
>> I also think we have some problems around release time when we merge all
>> APT suites of the feature branches that *have* been Git-merged into the
>> base branch being released (I'm gonna refer to this procedure simply as
>> "squashing" below). Specifically, all feature branches based on the
>> to-be-released base branch that has *not* been merged will be affected
>> by this. Example:
>
>> 1. Let F1 and F2 be divergent feature branches based on base branch B.
>> 2. F1 is merged into B.
>> 3. B is released, so F1's APT suite is merged into B's APT suite, i.e.
>> the "squashing".
>> 4. Note that at no point was F1 or B merged into F2, but right now it's
>> like the APT suite of F1 is merged into F2. Also note that F1's Git is
>> not merged into F2.
>
>> Now we have two issues:
>
>> * F2 has (indirectly, via the base branch's APT suite) been modified
>> without its Git state being changed at all.
>
> It seems that we don't have in mind the same semantics regarding
> relationship between a topic branch and the base branch it's based
> upon => let me explain how I see it.
>
> The issue you're raising has been here since we have an APT repo, and
> I don't see this as a problem at all. A base branch can evolve (be it
> in Git or APT), and it does impact all topic branches based on it:
> that's why it's called a "base" branch. IMO a topic branch is merely
> an overlay on top of a base branch. The fact that it may be lagging
> behind it (as far as Git is concerned) at a given point of time is
> a mere fact of life (things change), but does not mean that it
> shouldn't include recent changes made in its base branch when
> building: for all practical purposes I can think of, what we're
> interested about is "what product would be built from base branch B if
> we merged F2 into it?"
The new issue is that once we automatically build all active branches,
and send notifications on build failure, we want to minimize the amount
of build breakage that can be prevented.
>> * However, F2 doesn't have the Git changes made in F1, and it could very
>> well be that the APT stuff introduced by F1's APT suite depend on those
>> Git changes in a way so that builds will break without them.
>
> That's a non-issue once we have #8654 for all practical purposes I can
> think of: building F2 without merging B into it first, doesn't make
> much sense to me.
Right... and this was the point were I realized that we will merge topic
branches into their respective base branches before building, which will
fix all issues (except that already-merged branches should be frozen and
not reused). :S
Cheers!
