What is FOSDEM?

FOSDEM is the Free and Open-Source Developers European Meeting. Every year the Université Libre de Bruxelles (ULB) turns over their campus for one weekend around the end of January to a group of open source developers. It is a completely free meeting, but with an amazing amount of organization and the quality of presentations and discussions is second to none. Since no registration is required, nobody really knows how many attendees there are, but a rough estimate based on devices connected to the LAN is around 5000 (a blog article which mentions this is is here: http://daniel.haxx.se/blog/2014/02/03/my-fosdem-2014/; also fun to note is that the default network is IPv6-only, with a NAT64 connection to the legacy IPv4 network).

If you are an open source developer, I recommend attending FOSDEM. It’s awesome.

DNS at FOSDEM

While DNS has a lot of proprietary implementations - both products and services - there is a lot of DNS work that is open source. Since I am a DNS person, I spent some time at various DNS talks. These were spread across a lot of different sessions; there was no DNS or even tracks or developer rooms. I surely missed some talks though, given the scale of FOSDEM, but hopefully I hit most of the interesting ones.

systemd

The opening plenary talk was by Lennart Pöttering, well-known for his previous work on PulseAudio and more recently for his work on systemd.

https://fosdem.org/2016/schedule/event/systemd/

systemd has been adopted by most major Linux distributions (Debian, Ubuntu, Fedora, CentOS, …). It has been the source of controversy, partially because it replaces a lot of system functionality that has historically been provided by other means, such as the venerable System V init script system.

The title of the talk was quite generic, and I was pleasantly surprised to learn that about 80% of talk was actually about DNS and DNSSEC.

systemd is known for re-implementing existing functionality, and it turns out that DNS resolution is no exception. The developers are busily implementing systemd-resolved. This may be a good idea, or not. DNS developers that I talked to seem to regard this as a bad idea, although the reasons for this vary. (Personally I think that re-implementing the protocol is a classic beginner’s mistake with DNS. It looks like a simple protocol, but very quickly the complications of real-world, modern DNS become daunting.)

It turns out that Lennart is a convert to DNSSEC, and even though he reports that only 2% of DNS is secured with DNSSEC systemd is going to validate responses.

Lennart did give a good overview of some of the problems with implementing DNSSEC on the resolver of a user computer. These include:

  • Private DNS zones (for example FRITZ.BOX) don’t work
    These get a authoritative NXDOMAIN reply, and so can’t be seen. The systemd solution is to special-case NXDOMAIN for TLD and allow that.

  • Broken resolvers
    The systemd resolver uses the local resolver provided by the network, and often times these fail in weird and wonderful ways when a client tries to use DNSSEC. systemd will turn of DNSSEC in these cases.

  • Captive portals require registration
    If you are a hotel is redirects browser lookups to a registration page. Part of this often involves returning bogus answers to DNS queries. systemd works with Network Manager or systemd networkd or whatever other tool is used to allow DNSSEC to be disabled until registration is complete.

The basic approach is “downgrade aggressively by default”. Users can require that DNSSEC always be applied, but in the current Internet that seems to be too difficult by default. :(

I did ask about TLS or other privacy plans, but Lennart responded that DNS is public. I pushed back on this given pervasive monitoring, and Lennart said that he did not care because servers do not provide TLS and he only wants things that he can use. This seems like a fair, although disappointing and short-sighted, approach.

Knot DNS Resolver

Ondřej Surý used a lightning talk to present CZ.NIC’s latest work in the DNS software space: the Knot DNS Resolver.

https://fosdem.org/2016/schedule/event/knot_dns/

It’s GPLv3+, which is nice. (The only other GPL DNS resolver is the PowerDNS Recursor. Personally I am a fan of copyleft licenses, since they limit the ability of companies to incorporate it into proprietary products, giving little or nothing back to the community.)

The Knot DNS Resolver is written in C and LuaJIT. An audience member asked why it is written in C, and the answer was “because we like C”, although Ondřej noted that the Knot server is written in C. Ondřej also mentioned that CZ.NIC takes care to insure that it is written and tested in as safe a way as possible.

The software uses no internal threads, relying on multiple processes for scaling. There is a long list of supported features, some not implemented anywhere else, such as QNAME minimization.

Especially unique about the software is the cache design. It supports flexible cache backends. By default this is lmdb, but memcachedb and redis are also supported. One benefit of this is support for a persistent cache.

Anyone interested in seeing a pretty web view will be happy to know that it supports a tinyweb module. You can see a demo of this at:

http://kitsune.labs.nic.cz/

Managing a complex DNS environment

One of the Facebook engineers, Stephan Gorget, gave a presentation to a standing-room only crowd talking about how they manage their DNS.

https://fosdem.org/2016/schedule/event/managing_complex_dns_environment/

He gave an overview of their service, and then started with an inspiring quote:

Managing DNS is easy. 
      - said no one ever

The rest of the talk was split into two parts, one covering the internal DNS and the other the external DNS.

Internal DNS at Facebook

Facebook internal DNS runs Unbound, which is not surprising, and also TinyDNS, which is less expected. Apparently the TinyDNS zone file format is a compelling feature in the choice of TinyDNS versus BIND 9, which was the only other DNS server mentioned. For a DNS professional it certainly seems like an odd choice, and may be part of Facebooks resistance to adopting DNSSEC (this is just my theory - nobody asked about it).

On the internal side there are both human-generated and machine-generated changes, and a pipeline to reconsile changes and create a final version. A torrent seed is used to push zone changes out to all internal servers, which is a nice touch.

External DNS at Facebook

On the external side the main issue is getting users to the best site to deliver them data as quickly as possible.

Facebook has a clever system which manages TCP and TLS termination near users, and then tunnels requests for data after that to more distant PoP if necessary. This setup can save several slow packet round trips. The example gave was turning a 600 msec time into 240 msec - still not very fast but undeniably a massive savings.

Facebook uses a trick where some small percentage of users are given a unique name to look up, which is then matched on the HTTP side. This allows Facebook to build a map of the relationship between users, resolvers, and their infrastructure. This map is updated every 2 minutes, and send to their DNS servers at that time.

Talk message points

Facebook’s engineers seem justifibly proud of their creation. It is a load-balancing DNS service that automatically generates and updates configration, with direct user benefit.

They give the following recipe for success, which seem applicable to almost any DNS setup:

  • Use simple sevices as building blocks
  • Keep no state in your system - or as little state as possible
  • Focus on reliability, reproducability, and traceability

The promise they make is that this will make things “easier and more complex”. :)

DNS ad-hoc gathering

In addition to presentations, Peter van Dijk organized an ad-hoc gathering for anyone interested in DNS. It’s always nice to chat with people working on DNS about the problems they are seeing as well as things they are excited about.

There ended up being about 10 people who showed up. There was no fixed agenda, but topics ranged from dnsdist to LDAP, from ICANN RSSAC Caucus to Paul Hoffman’s DNS testing, and the always-popular observations that DNS software never gets upgraded.

If I had been more diligent I would have taken some notes, but I admit I was enjoying the conversation too much….

Other Stuff

Even with all of this DNS, there was plenty of time at FOSDEM to learn about all kinds of great stuff, including tons of IoT presentations and demos, an update about the RIPE NCC’s efforts to open the source of RIPE Atlas, CZ.CNIC’s fabby Turris Omnia router project, and an interesting talk by the insightful-but-abrasive Richard Stallman of the Gnu project & FSF.