Why we overhauled our network infrastructure, and what it means for users.
A post from Webtrekk Head of System Administration
This is the second installment of the Dev Blog series, posts written in the language of programmers.
The first post, "Determining Conversion Probability", is available here.
The internet accelerates a bit every day. Fueled by national grants, EU subsidies and the natural expiration of old telecommunications systems, the expansion of fibre-optic networks in Germany and Europe is picking up speed.
Though this development delights end users, it poses a genuine challenge for companies like Webtrekk. Since tracking is expected to be imperceptible, all subsequent processing needs to happen instantaneously: The faster a user can access websites, the faster tracked data needs to be processed.
Moreover, the quicker websites load, the more sites users can access. So the challenge of rapid data processing is compounded by the challenge of processing not only faster data, but ever-growing amounts of data.
High bandwidths also present opportunities for cyber criminals, who are refocusing on private computers and their increased network capacities. Since private computers are often less protected, they can be turned into “bots” more easily. These bots are then used together with a multitude of other hacked computers in so-called bot-nets constructed for cyberattacks.
Even though Webtrekk has never suffered so-called distributed denial-of-service (DDoS) attacks, we noticed connection bottlenecks originating from attacks on our data centre operator’s other clients. This necessitated imporovements to our external connection, as well.
One and a half years ago, Webtrekk decided to overhaul its network infrastructure to meet modern requirements and anticipate future developments. Amongst the requirements for the new infrastructure were higher bandwidth, increased scalability, expanded redundancy and industry-leading security.
Spatial Conditions – Legacy Examination
Since constructing networks isn’t Webtrekk’s core business, we opted to bring an experienced partner – Opteamax GmbH – on board to jointly prepare, plan and implement a concept.
Pooling the racks was necessary to restructure the network using modern concepts. After talks with our data centre operator, QSC in Nuremberg, a solution for gradually consolidating racks – which were previously spread out over six rooms – and rooms was developed. A closed cold aisle was provided for each of two fire compartments, which were then filled with new and existing systems.
New LAN Structure
A Brocade Ethernet fabric was introduced to reach a significantly higher degree of provisioning automation. This fabric offers provisioning via OpenStack (an upcoming standard for software-defined networking), 10Gbit interfaces, simple scalability and redundancy options.
Every rack is equipped with two switches containing 24 or 48 10Gbit ports each. Furthermore, these switches come with four 40Gbit ports; a meshed ring structure connects all switches.
Without having to care about loops, these cross-links can be extended spontaneously via a specific proprietary protocol. As soon as one link fills up, a new Inter-Switch Link (ISL) can be added and will promptly integrate into all path calculations. It doesn’t matter which links connect which switches – the fabric will always calculate the ideal path. If a path drops out, the fabric is able to shift to alternative paths, without loss, in less than a second.
The next requirement was to give each server a redundant connection, ensuring functionality even if something goes awry. Luckily, the Ethernet fabric also serves our needs beyond edge ports. Since all switches within the fabric are addressed as one big switch, it’s easy to create a load-balanced group of connections using Link Aggregation Control Protocol (LACP). Every new computer is equipped with two 10Gbit network cards. These can be connected to the Ethernet fabric via OpenVSwitch with LACP trunks in “balancing mode”.
This way, every server can use a capacity of 20 Gbit during regular operation. Even if one connection dies – e.g., due to a broken switch, a faulty cable or during software updates for switches – there will be no perceptible service downtimes. Only the connection’s capacity will decrease.
The whole system’s performance has been increased by permanent active multipathing (distributing load amongst paths) and by the significant increase in bandwidth from 1Gbit to 2*10Gbit. This is detectable not only during regular operation, but also during maintenance. Due to considerably faster live migration of databases, most of our clients are hardly affected by maintenance work. Even large amounts of data that previously took several days to migrate can now be moved within mere hours.
As mentioned earlier, the external connection had to be improved. Since we couldn’t influence attacks on QSC’s other clients, we decided to take the external connection into our own hands. With this in mind, Webtrekk joined RIPE (Réseaux IP Européens) and has been running its own “autonomous system” within the internet for well over a year now.
The Juniper routers that were purchased for this purpose are connected to both QSC and Opteamax. Thanks to their flexibility, Opteamax can react spontaneously to changing requirements by configuring new connection paths on demand.
This was advantageous for establishing connections between Webtrekk and the Asian region via China-Telecom’s landlines. As a result, response times for Eastern China have dropped from 380-400ms to 160-180ms and packet loss has become a thing of the past.
The new WAN infrastructure has also significantly increased general performance. Most of all, we were unable to detect any noteworthy downtimes caused by DDoS attacks since Opteamax took charge of routing.
Load Balancing and Security
What good is a fast network if transferred data cannot be processed quickly enough? To avoid a bottleneck, we also reviewed load balancing and firewalling.
Test results convinced us to choose F5 BIGIP for load balancing. These load balancers are able to use SSL encryption on both the client side and the server side, ensuring that all transferred data is encrypted until reaching its destination. As with our fabric, all load balancers are equipped with fully redundant 10Gbit connections towards LAN and WAN. Compared to our previous systems, this enables us to process more than 50 times as many connections at once.
Security has been enhanced as well, resulting in increased performance thanks to improved concepts and hardware. Juniper’s Carrier Grade firewalls offer enough capacity to properly process the new network’s speed.
The choice to drop NAT (Network Address Translation), and to instead work with cleanly defined packet filters, dramatically increased the possible number of simultaneous connections while decreasing firewall load. It also noticeably increased the firewall’s throughput rate.
However, performance improvements were not the main reason for this choice. It was rather based on the firewall functionality for IPv4 being as identical to the IPv6 configurations as possible. Since IPv6 doesn’t innately support NAT, it only seemed natural to not use NAT for IPv4 either. The only servers still utilising NAT are servers that are never addressed from the outside and exclusively process data within the “trust zone”. And even these only use it for accessing external services like update repositories. Without this unfortunately necessary step, the available IPv4 address space would not suffice.
The whole new network concept was created considering the IPv6 issue – that is, the ongoing migration from the IPv4 version of the Internet Protocol to IPv6. The entire backbone, for example, has already been fully prepared for IPv6. All of our servers are able to obtain IPv6 addresses and communicate with the IPv6-using world. Webtrekk’s software is currently being adapted and thoroughly tested for IPv6. However, as far as the network is concerned, all wheels have been set in motion for the protocol evolution.
As soon as IPv6 has been rolled out for all servers and services, we will gradually decommission the private IPv4 addresses used within our network.
In the meantime, we are set up to handle the speed, volume and security issues of today and tomorrow.