Hello I am Jonathan Wright, Infrastructure Team Lead for AlmaLinux. I manage most of the plumbing that keeps things humming smoothly along and I’ve been working on some improvements to some parts of it to make things more user friendly for our community.
AlmaLinux values transparency https://wiki.almalinux.org/Transparency.html and communal decision making, it’s one of the reasons why I decided to become a contributor. As part of some of the work I’m doing I’d like to request some feedback from the community on a proposal to enable `dnf countme` similar to the way the Fedora project does.
countme is a core feature of DNF implemented upstream in Fedora 32 (dnf 4.2.9). It is described by the docs as such:
Determines whether a special flag should be added to a single, randomly chosen metalink/mirrorlist query each week. This allows the repository owner to estimate the number of systems consuming it, by counting such queries over a week's time, which is much more accurate than just counting unique IP addresses (which is subject to both overcounting and undercounting due to short DHCP leases and NAT, respectively).
The flag is a simple "countme=N" parameter appended to the metalink and mirrorlist URL, where N is an integer representing the "longevity" bucket this system belongs to. The following 4 buckets are defined, based on how many full weeks have passed since the beginning of the week when this system was installed: 1 = first week, 2 = first month (2-4 weeks), 3 = six months (5-24 weeks) and 4 = more than six months (> 24 weeks). This information is meant to help distinguish short-lived installs from long-term ones, and to gather other statistics about system lifecycle.
countme was designed with privacy in mind and does not add any identifying or unique information to requests so there is no tracking involved. Just a simple “hello” to the repository.
Currently, AlmaLinux does not track any sort of usage statistics for our distribution at all. We can technically try to aggregate basic metrics from HTTP logs on our mirrorlist servers but the reliability of the data will not be the best since counting unique IPs is undermined by things like NAT and dynamic addressing. So, I’d like to propose we implement “countme=1” in our repository configs just as Fedora and EPEL have done. I’d also like to propose that the aggregated data be made available publicly, similar to https://data-analysis.fedoraproject.org/ for the community to see.
I’ve setup a form for feedback at https://forms.gle/BShXoxJmsjNbMXCk6 in case you’d like to give any input on this proposal. We will keep this form open for about a week.
FAQ:
Q: When are “countme” requests sent? A: Once a week at random during normal dnf activity. If you do not use dnf calls that would otherwise trigger mirrorlist requests (makecache, install, update) this flag will NOT cause dnf to go out of its way and make special requests.
Q: What extra data will be sent that is not currently collected? A: “countme=X” will be added to a random mirrorlist request each week from DNF where X is a number, 1-4 which represents the number of weeks your system has been installed. See above for the explanation of this from the DNF documentation.
Q: Will aggregated data be made publicly available? A: Yes
Q: What data do you use? A: The only data we look at is in the HTTP request itself. Our log lines are in the standard Combined Log Format. Ex: 172.30.61.81 - - [15/Dec/2021:17:02:12 +0000] "GET /mirrorlist/8/baseos?countme=4 HTTP/1.1" 200 629 "-" "libdnf (AlmaLinux 8.3; generic; Linux.x86_64)"
We only look at log lines where the request is "GET", the query string includes "countme=N", the result is 200 or 302, and the User-Agent string matches the libdnf User-Agent header.
The only data we use are the timestamp, the query parameters (repo, arch, countme), and the libdnf User-Agent data.
In the future we will also aggregate data by country using GeoIP. Our processing and aggregation does not care about IPs themselves or their uniqueness. When we implement the aggregation of geographic data it will use MaxMind’s GeoIP database locally to turn the IP into a region which will be used for tallying generalized metrics for that region.
Raw access logs are archived in case we find major issues in any of our processing which would allow us to re-parse the data in the future and correct the published statistics.
Q: Can I opt out? A: Yes, but we’d prefer you not since the data is very helpful. The only extra data you’ll be submitting is “countme=X” in one request per week.
If you’d like to opt out you can comment out the “countme=1” line in the repository config files in /etc/yum.repos.d/
Discussion for this should be directed to the AlmaLinux Infrastructure mailing list. You can join the list at https://lists.almalinux.org/mailman3/lists/infra.lists.almalinux.org/