Hi,
Jacob Appelbaum wrote (24 Jun 2014 10:56:54 GMT) :
> I think agreeing on a specific user agent and having a central place
> to find it makes the job much easier to tackle. In any case, I think
> setting a few shell aliases would not hurt and if they source a common
> file for a user agent, it should be straight forward to keep things in
> sync with perhaps no upstream modifications?
> For example:
> wget --user-agent="$useragent"
> curl --user-agent "$useragent"
> GET -H "User-Agent:$useragent"
This would definitely work. We ship a getTorBrowserUserAgent program,
that's used by the curl processes started by htpdate. Its results
could be cached at ISO build time, and then used by these aliases.
One should look for other instances of using wget, curl, LWP and
friends without going through the shell, too. Any taker?
> For the discussion at hand, I sniffed my own sessions and saw the
> following data transmissions.
Woohoo \o/ .. and sorry for the delay.
> wget:
> [...]
> Accept: */*
> Connection: keep-alive
> curl:
> [...]
> Accept: */*
> Proxy-Connection: Keep-Alive
> GET:
> [...]
> TE: deflate,gzip;q=0.3
> Connection: TE, close
> This is Tor Browser on Tails for the same file but on a different web server:
> [...]
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-us,en;q=0.5
> Accept-Encoding: gzip, deflate
> Connection: keep-alive
So, this shows that we have an identifying set of headers for each of
these four clients, even if we ignore the user-agent information.
... but, in the following tests (with a forged user-agent), most of
these discrepancies disappear, so I'm confused:
> Here are the same clients with a forged User Agent:
> wget --user-agent="User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:24.0)
> Gecko/20100101 Firefox/24.0"
> [...]
> Accept: */*
> Connection: keep-alive
> curl --user-agent "User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:24.0)
> Gecko/20100101 Firefox/24.0"
> http://people.torproject.org/~ioerror/misc/tor-ips.txt; shows:
> [...]
> Accept: */*
> Connection: keep-alive
I'm surprised: without faking the user-agent, we had
"Proxy-Connection" instead of "Connection". Is one of these results
wrong, or is curl behaving erratically, or is there another
rational explanation?
> GET -H "User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:24.0)
> Gecko/20100101 Firefox/24.0"
> [...]
> Connection: keep-alive
Same here, we had "Connection: TE, close" previously => same question.
> My conclusion is that setting the user agent for curl and wget to
> match Tor Browser isn't a horrible idea. It even seems like on a
> single GET request, it would be helpful for privacy and anonymity set
> reasons. It certainly reduces the version information leakage that is
> absolutely useful for fingerprinting and exploitation. For `GET` - we
> might also add -H="Accept: */*" and then all three would be aligned.
I'll wait for the surprising things highlighted above to be clarified,
before commenting on this one.
Thanks a lot!
Cheers,
--
intrigeri
