Hi,
anonym wrote (17 Mar 2016 17:18:11 GMT) :
> intrigeri:
>> anonym wrote (10 Mar 2016 20:06:31 GMT) :
>>>> Freeze exceptions
> [...]
>>> BTW, it would be great to have a linting tool that compared the current
>>> APT pinnings vs what is available in the current Debian branches used
>>> given some Tails source checkout.
>>
>> I'm open to adding ideas of helpful tools to the blueprint.
>>
>> I'll need help to specify more clearly what problem we want desired
>> tools to solve, and how.
>>
>> If I got it right, you want to know something like "what would happen
>> if we dropped our APT pinning", right? Do we want to know that for the
>> case when we remove APT pinning we have set up to grant freeze
>> exceptions only, or all APT pinning? The former, I guess, right?
> Well, I'm not sure how it would be determined if a pinning was added for
> a freeze exception exactly, and not some other purpose.
We could for example have a filename pattern, for snippets dropped in
/etc/apt/preferences.d/, that is specific to those added for
freeze exceptions.
> Any way, this
> tool seems to be useful in the latter case you talk about too, to keep
> our pinnings trimmed, especially if we become a rolling distro, and may
> have to frequently pin stuff (security updates) from Debian Unstable.
> Furthermore, I expect this latter case to be easier to solve, and I
> think I'd be happy enough with that one solved -- with informative
> enough output it will be easy enough to use it for the first case too.
OK. To be clear, I won't move forward on it until it's clarified what
problem this would help solve in the current state of things (we're
not a rolling distro), as this is not obvious to me yet. Besides, I'm
not sure it's feasible to easily solve any such interesting problem we
would have in real world situations.
And of course, I'll prioritize higher designing and implementing
a solution to the problems raised in the "Freeze exceptions" section,
that have to face somehow.
>>>> Garbage collection
>>> [...]
>>> but my point is that the garbage collector will have to
>>> chech each branch, right?
>>
>> I think this would be over-engineering it a lot, given what our actual
>> use cases are.
> Indeed, some minor manual work can work around this, as you point out.
> [...]
>> Speaking of which, I see two main ways to handle the garbage
>> collection process:
>>
>> a. use a manually maintained list of snapshots that need to be
>> kept around, as the blueprint currently suggests;
>>
>> b. rely on Valid-Until; i.e. the way to express "I want to keep
>> a given snapshot around" would be to postpone its expiration date;
>> I see no reason to differenciate "keep a given snapshot around"
>> from "keep a given snapshot usable".
>>
>> I think we should do (b), _and_ have some cronjob warn us if we're
>> going to have serious problems, e.g. if the snapshot used by a frozen
>> testing branch is going to expire (and be deleted); this avoids the
>> need to maintain a list of exceptions.
> This sounds reasonable.
>> Let's discuss separately the two main cases:
>>
>> * frozen testing branch: we rarely freeze for more than 10 days, so
>> in the general case there's no problem; and the cronjob check
>> mentioned above should help us deal with corner cases.
> Sure, but it *does* happen. Let's just make it explicit in the freeze
> section of the release docs to explicitly set the snapshot expiry to the
> expected release date + 5 days to account for release delays.
Right, thanks. feature/5926-freezable-APT-repository had something
that was too vague about it, now clarified with e51c9aa.
>>> However, I see nothing about how to deal with Debian packages that
>>> fetches something external at install time (firmwares, sources illegal
>>> in some jurisdictions). This sounds like a non-trivial problem, and I
>>> really wonder what your thoughts on solutions are.
>>
>> Indeed, that's outside of the scope of the current "freezable APT
>> repository" project.
> I see. This implies that there is no explicit goal (yet) to make each
> Tails release buildable "forever", correct?
Correct. Even the "tagged APT snapshots" feature we are working on is
a bonus, that is not part of the formal goals of the deliverable this
is part of.
>> My current best solution for that is to package
>> all these things as .deb's somewhere (possibly in a very ad-hoc way in
>> our own overlay APT repo), so we get them handled (snapshotted etc.)
>> for free just like any other package. What do you think?
> Isn̈́'t the problem that some of these Debian packages fetch these blobs
> from static URLs during package installation? How would a .deb
> containing the blobs help, then? Is there some Debian packaging
> mechanism that all of these use that looks for files in some cache first
> where you intend to place them?
I've not put much thought into it yet, and will skip it for now.
But yes, probably something like that.
>>> Crazy idea: along with the snapshots we also have a particular cache of
>>> a caching proxy. For instance, we set `http_proxy` in the build env to
>>> point to some caching proxy running in our infra, and when we build with
>>> snapshot X, we set the proxy username to X to tell the proxy that cache
>>> X is to be used. Cache X contains exactly the cached files used when
>>> building with snapshot X the first time, because that is when they were
>>> seeded. Crazy! We'd need some sort of access control for this, though.
>>> :)
>>
>> Yes, crazy :) Frankly, I need to focus on stuff that's in the scope
>> of the project I'm working on right now, so I haven't thought of
>> it much.
> Ack! It's mostly relevant for the reproducible builds part, so let's not
> focus on it until then; but let's keep this future design change in mind
> while implementing this design, ok?
I'm not sure I understand correctly what you mean here. What shall we
keep in mind, and during what?
>>> And I also wonder if the same mechanism can be used to determine the
>>> "tagged, partial snapshots", instead of the "'build the almost-final
>>> image' to generate the package manifest" thing I mentioned I didn't like
>>> above. Think of it as an apt-cacher-ng instance that is seeded when
>>> doing the canonical release build, and then frozen forever.
>>
>> Indeed, that might have been another way to implement it.
>> We're "almost done" with our current implementation so I doubt we want
>> to switch to this new idea now, though :)
> So you send and RFC on a *design* when it already is *implemented* and
> cannot be changed? A true RFC (in the IEEE sense) indeed! :) (This is
> not an actual critique, just a bad joke! :))
Yes, we had to do enough of the implementation to be 95% sure that the
design could possibly work :)
Cheers!
--
intrigeri
