Bits of networks

Patching OpenSSH in Debian

2026-08-17T00:00:00+00:00

This article was written entirely without the help of any generative AI tool.

I needed to test an OpenSSH patch. I thought that rebuilding the Debian package would be easy, but it wasn't.

My case for rebuilding the OpenSSH Debian package</h1>
My main use-case when building or rebuild packages is to use patched or backported versions locally or for internal server use, not to push them in the Debian archive. It means I don't need the same kind of quality process, but I would still like the resulting packages to be as close to Debian as possible, hoping it helps my system to remain stable.
I love Debian for its stability and community, but there's one thing that has always irritated me: there are so many available tools to build Debian packages, and there is no particular recommendation to use any of them. Besides, the whole building process feels quite obscure and magical.
In that particular case, I wanted to test a patch to the OpenSSH client. Since I use SSH daily, I definitely don't want to run a weird build that could break any of my workflows. I thought I was starting to master the art of building Debian packages, but there were new interesting surprises!
Rebuilding Debian packages is easier than before</h1>
In recent years, the ease of building Debian packages has improved, notably because many packages are now maintained as a git repository.
Obtaining and modifying the source Debian package is easier for me thanks to git:

I go to the package details</a></li>
I jump to the source package details</a></li>
I follow the link Debian Source Repository</a></li>
I clone the git repository</li>
I checkout the correct git branch for the target version of Debian</li>
I add my own commits to the git repository if needed</li>
I can now build the package!</li> </ul>
That last step can still be difficult, but at least it's nice to be able to work with a familiar git environment with branches, stashes, cherry-picks, etc.
In addition, there is (yet another) wrapper to facilitate working with Debian packages that are maintained in git repositories: git-buildpackage</code>. It can integrate withpbuilder</a> to build package in a chroot.
I found this particular guide to be quite useful: https://people.debian.org/~debalance/packaging-with-git.html</a>. The Debian maintainer guide</a> is a good resource too.
Hitting a wall at first</h1> Armed with that knowledge, I quickly tried to build my patched OpenSSH package. Let's clone the repository and apply our patch: $ git clone https://salsa.debian.org/ssh-team/openssh.git $ git checkout trixie $ git apply my-openssh-patch.patch $ git commit -am "Apply local patch to fix race condition"</code></pre> Then we need to initialize a fresh pbuilder chroot (see the guides above): $ DIST=trixie git-pbuilder create</code></pre> Now let's try to build in the chroot: $ DIST=trixie gbp buildpackage --git-ignore-branch dpkg-source: info: building openssh using existing ./openssh_10.0p1.orig.tar.gz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: local changes detected, the modified files are: openssh/clientloop.h openssh/mux.c openssh/ssh.c dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/openssh_10.0p1-4.diff.R0ebST dpkg-source: hint: make sure the version in debian/changelog matches the unpacked source tree dpkg-source: hint: you can integrate the local changes with dpkg-source --commit gbp:error: 'git-pbuilder' failed: it exited with 2</code></pre> OK, it seems that we need to maintain the patch with quilt instead of committing the code change directly to the git repository. This is annoying but doable. I must admit I did it by hand by simply copying the patch and editing debian/patches/series</code>, and committing the patch (replacing the previous commit). Let's try again: $ DIST=trixie gbp buildpackage --git-ignore-branch dpkg-source: info: local changes detected, the modified files are: openssh/clientloop.h openssh/mux.c openssh/ssh.c dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/openssh_10.0p1-4.diff.R0ebST</code></pre> In short, we get the same error, even though this time I didn't touch the actual openssh code in the git repository. I verified several times that I didn't make a mistake somewhere, but no, the only commit I added is touching debian/patches</code>, not the upstream code directly. At the point, I'm in WTF mode, and I leave this rest for a while. Hunting for the culprit</h1> Several days later, looking at the problem again, I found a blog post from Colin Watson, OpenSSH's Debian maintainer, precisely about converting the OpenSSH package to the "new" (that is, new as in 2010) quilt format: https://www.chiark.greenend.org.uk/~cjwatson/blog/thoughts-on-3.0-quilt-format.html According to his article, Colin takes the unusual (but well-motivated) approach of having an already-patched tree in the main git branch, while other maintainers prefer having their git repository reflect the upstream code: they manage patches either with quilt or in a separate branch. Due to this choice, Colin says "it’s a bit awkward to set things up when checking out from revision control", because quilt doesn't understand that the patches are already applied. AH AH! It seems we got our culprit! The article mentions an helpful script that sets things up properly, let's use it: $ debian/rules quilt-setup dh quilt-setup --with=runit dh: error: Unknown sequence quilt-setup (choose from: binary binary-arch binary-indep build build-arch build-indep clean install install-arch install-indep) make: *** [debian/rules:122: quilt-setup] Error 255</code></pre> Err, so the helpful script doesn't exist! Looking around some more, I found out that the script was removed in 2014</a> with a commit message "Initialize git-dpm". We are on a good follow-up trail here. Trying out git-dpm</h1> So, let's learn to use (yet another) tool to manage Debian packages in a git repository! The manpage for git-dpm</a> is actually rather helpful. The only confusing aspect is that Colin is again storing the applied state of patches in the git tree, while git-dpm defaults to store the unapplied state (i.e. upstream state). It means the branches explanations from the manpage don't actually apply to the Debian openssh repository. It's a choice to make when initializing a new repository, see the --patches-applied</code> option). Anyway, with the right tool, it's now straightforward to patch and build the package: $ git-dpm checkout-patched $ git cherry-pick XXX # or git am, git apply + commit, etc $ git-dpm dch -- -i $ DIST=trixie gbp buildpackage --git-ignore-branch</code></pre> And I got my patched package openssh-client_10.0p1-7+deb13u4.2_amd64.deb</code>! Conclusion</h1> Building Debian package still feels obscure, but at least I learned new tools to manage packages in git. It's a bit annoying that different Debian maintainers have different habits, but I feel the ecosystem is already more homogenenous than it was a few years ago. Besides, once you understand the maintainer's logic (especially through their useful blog posts!), things become easier. About git-dpm, there is a small downside for me: most commits are merges, which makes it difficult to review historical diff. My first reflex to understand the packaging of openssh was to look at the commit history: $ git log -p debian/rules</code></pre> but this was mostly useless: I could see the introduction of quilt-setup</code> in 2010 but not its removal afterwards, WTF. It turns out that git doesn't display diff for merges by default, but you can ask for them: $ git log -p --first-parent debian/rules</code></pre> That's already more useful, and one more thing I learnt during this small experience!

BananaPi BPI-F3: bootstraping upstream Debian 2024-09-30T00:00:00+00:00 This post is part of the Upstream RISC-V serie: The upstream RISC-V experience: running RISC-V hardware with upstream distros</a></li> VisionFive 2 Benchmarking RISC-V: VisionFive 2 vs the world</a></li> VisionFive 2: the hardware</a></li> </ul> </li> BananaPi BPI-F3 BananaPi BPI-F3: hardware and software overview</a></li> BananaPi BPI-F3: bootstraping upstream Debian</a></li> </ul> </li> </ul> Debian bootstrap method</h2> Since July 2023, riscv64 is an official Debian architecture</a>. It means that generating a RISC-V Debian rootfs is really straightforward. However, without kernel support</a>, it is not that straightforward to actually install Debian on RISC-V hardware. Here, we will run the Bianbu Linux system from a SD card and use it to bootstrap Debian on the internal eMMC storage, using debootstrap. For the kernel, we will simply re-use the Bianbu kernel. This is clearly not ideal but it is the only way currently (all distributions with early support for this hardware are doing the same). Prerequisite</h2> To follow this tutorial, you will need: a BananaPi BPI-F3, of course :) I tried two versions of the hardware: one with 4 GB RAM + 16 GB eMMC, and one with 16 GB RAM + 128 GB eMMC</li> a UART serial-to-USB converter, necessary to debug things and connect through the serial console</li> a good-quality USB-C power supply. Don't underestimate this requirement: I spent too much time debugging weird issues on various boards, and 90% of the time it was caused by the power supply. I am now using a 45 W laptop power supply that is capable of delivering 12 V and everything works fine.</li> a good-quality micro-SD card to be able to boot Bianbu Linux. Again, I spent hours failing to boot any image on this board, even though I was trying two different SD cards. It turns out that both were old and not booting correctly for some reason. A third recent SD card, a SanDisk Ultra Plus, worked fine.</li> optionally, a NVMe drive. Typically you can setup /home</code> on the NVMe (this is quite standard Linux sysadmin and not explained here)</li> </ul> Debian installation</h2> Step 1: boot Bianbu Linux</h3> Get the most recent version of Bianbu from http://archive.spacemit.com/image/k1/version/bianbu/</a>. In particular, version 2 images have a newer kernel. Use the minimal image for SD card, for instance bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808.img.zip</code> On your laptop, unzip the image and flash it to a SD card, for instance using a standard dd</code> copy. After inserting the SD card on the BPI-F3, it should boot without problem and you will see the login prompt on the serial console. Step 2: prepare the internal eMMC with partitions and bootloader</h3> Login with user root</code> and password bianbu</code> on the serial console. Then either continue through the serial console, or setup networking and SSH to login through SSH. Let's partition the internal eMMC storage mmcblk2</code>: export LANG=C.UTF-8 cat <<EOF | sfdisk /dev/mmcblk2 label: gpt first-lba: 256 1 : start=256, size=512, name=unused 2 : start=768, size=128, name=env 3 : start=2048, size=2048, name=opensbi 4 : start=4096, size=4096, name=uboot 5 : start=8192, size=400M, name=bootfs 6 : name=rootfs EOF</code></pre> For simplicity, this partition scheme is very similar to the vendor scheme, with a larger boot filesystem. The first partition is not actually useful on the eMMC, but we need it, otherwise u-boot gets really confused when it tries to compute which partition is the root partition to pass to the kernel. Next, you need to obtain the magic bootinfo specific for the eMMC. Without it, booting will fail. It can be found in the other zip file (e.g. bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808.zip</code>). This bootinfo magic and the FSBL (copied from the SD card) will need to be flashed to a special portion of the eMMC. You can get it directly from the board: wget http://archive.spacemit.com/image/k1/version/bianbu/v2.0rc1/bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808.zip mkdir bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808 cd bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808 unzip ../bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808.zip cd ..</code></pre> Then flash bootinfo and FSBL (thanks stintel</a> for the trick to make this part of the eMMC writable): echo 0 > /sys/block/mmcblk2boot0/force_ro dd if=bianbu-24.04-minimal-k1-v2.0rc1-release-20240909131808/factory/bootinfo_emmc.bin of=/dev/mmcblk2boot0 bs=512 count=1 dd if=/dev/mmcblk0p1 of=/dev/mmcblk2boot0 bs=512 seek=1</code></pre> Then copy all other bootloader stages from the SD card: # u-boot env dd if=/dev/mmcblk0p2 of=/dev/mmcblk2p2 bs=64K status=progress # opensbi dd if=/dev/mmcblk0p3 of=/dev/mmcblk2p3 bs=64K status=progress # u-boot dd if=/dev/mmcblk0p4 of=/dev/mmcblk2p4 bs=64K status=progress</code></pre>Step 3: debootstrap Debian</h3> Prepare the filesystems for the future /boot</code> and /</code> and mount them: # Create boot and root filesystems mkfs.ext4 -m 0 /dev/mmcblk2p5 mkfs.ext4 -m 0 /dev/mmcblk2p6 # Mount everything mkdir -p /mnt/target mount /dev/mmcblk2p6 /mnt/target/ mkdir -p /mnt/target/boot mount /dev/mmcblk2p5 /mnt/target/boot</code></pre> Then run debootstrap. Here we include some base packages, feel free to modify the list. # Debootstrap with some base packages PACKAGES=openssh-server,systemd-timesyncd,bash-completion,vim,htop,wget,xz-utils,zstd,file,locales,dbus,openssl,ca-certificates debootstrap --arch=riscv64 --include="$PACKAGES" unstable /mnt/target http://deb.debian.org/debian</code></pre>Step 4: finishing touches before first boot</h3> Copy the kernel, kernel modules, and necessary firmware: cp -a /boot/* /mnt/target/boot/ cp -a /usr/lib/modules /mnt/target/usr/lib/ mkdir -p /mnt/target/usr/lib/firmware cp -a /usr/lib/firmware/esos.elf /mnt/target/usr/lib/firmware/</code></pre> Make sure you will be able to login to the system: # Setup a root password for the console echo 'root:CHANGEME' | chpasswd -R /mnt/target # Setup a SSH key mkdir /mnt/target/.ssh echo "YOUR_SSH_KEY" > /mnt/target/.ssh/authorized_keys</code></pre> Setup target fstab, using blkid</code> to get the UUID of the new boot and root partitions, and then editing fstab manually: blkid cp /etc/fstab /mnt/target/etc/fstab nano /mnt/target/etc/fstab</code></pre> Your fstab file should look like this: # <file system> <dir> <type> <options> <dump> <pass> UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX / ext4 defaults,noatime,errors=remount-ro 0 1 UUID=YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY /boot ext4 defaults 0 2</code></pre> Finally, setup basic DHCP network, adapt to your need: cat <<EOF >> /mnt/target/etc/network/interfaces auto end0 iface end0 inet dhcp EOF</code></pre> You are now done with the setup, you can umount and shutdown the board: umount /mnt/target/boot/ umount /mnt/target sync poweroff</code></pre>Step 5: first boot on Debian</h3> Boot without the SD card: Debian should come up on the serial console! Login on the serial console and/or SSH, and finish the setup according to your needs, e.g. set a locale: localectl set-locale C.UTF-8</code></pre>Step 6 (optional): fix initramfs generation</h3> If you are playing with kernels, you will quickly discover that an initramfs built from Debian will hang during boot. The initramfs provided by Bianbu, /boot/initrd.img-6.6.36</code>, works fine though. This is because a firmware needs to be added to the initramfs (thanks aurel32 for the tip and hook script below). First, backup the vendor initramfs: cp -a /boot/initrd.img-6.6.36 /boot/backup-initrd.img-6.6.36</code></pre> Install initramfs-tools if needed: apt install initramfs-tools</code></pre> Then add a hook /etc/initramfs-tools/hooks/bianbu-firmware</code> that copies the required firmware into the initramfs: #!/bin/sh # # List the soft prerequisites here. This is a space separated list of # names, of scripts that are in the same directory as this one, that # must be run before this one can be. # PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions # Ensure the esos.elf firmware is included copy_file firmware "/usr/lib/firmware/esos.elf"</code></pre> Don't forget to make the hook executable, and rebuild the initramfs: chmod +x /etc/initramfs-tools/hooks/bianbu-firmware update-initramfs -u</code></pre> Reboot to check if everything works fine. Otherwise, you can also boot back on Bianbu using the SD card to fix your system. BananaPi BPI-F3: hardware and software overview 2024-09-29T00:00:00+00:00 This post is part of the Upstream RISC-V serie: The upstream RISC-V experience: running RISC-V hardware with upstream distros</a></li> VisionFive 2 Benchmarking RISC-V: VisionFive 2 vs the world</a></li> VisionFive 2: the hardware</a></li> </ul> </li> BananaPi BPI-F3 BananaPi BPI-F3: hardware and software overview</a></li> BananaPi BPI-F3: bootstraping upstream Debian</a></li> </ul> </li> </ul> The BananaPi BPI-F3 hardware</h2> I recently received several BananaPi BPI-F3 boards from RISC-V International, thanks to them for their support. After some software experimentation, the boards will be made available to all users of cfarm.net</a>. This hardware is interesting for a couple of reasons: it is the first widely available hardware with RISC-V Vector Extensions (RVV 1.0)</li> it comes with 8 CPU cores and up to 16 GB of RAM, which is double that of the VisionFive 2</li> it comes with complete storage connectivity (NVMe, internal eMMC, SD card) and even has a mini-PCIe slot for e.g. a Wi-Fi card</li> this is a completely different SoC compared to the VisionFive 2: it uses the SpacemiT K1 SoC</a> with SpacemiT X60 cores at 1.60 GHz</li> </ul> Power consumption and performance</h2> Initial power measurements and performance benchmarks look quite good. Overall, it is very similar to the VisionFive 2 as measured in a previous article</a>: single-threaded performance is roughly the same as the VisionFive 2 (but the BPI-F3 has twice as many cores)</li> power consumption is also similar to the VisionFive 2: around 5 to 8 W when idle, around 13 to 15 W when fully loaded. These power figures are measured on the 230 V AC side, which means that they are very dependent on the power adapater: take them with a grain of salt.</li> CPU frequency scaling works, with 5 steps: 614 MHz, 819 MHz, 1000 MHz, 1.23 GHz, 1.60 GHz. Idling at 614 MHz saves around 1 W of power compared to idling at 1.60 GHz. Again, this is very similar to the VisionFive 2.</li> </ul> These measurements were done on Bianbu Linux 2.0rc6, with their vendor kernel based on Linux 6.6. EDIT 2024-10-03: actually, CPU frequency scaling has unfortunate side-effects with the vendor kernel. When using a dynamic cpupower governor such as powersave</code> or schedutil</code>, a kernel thread starts spinning at 100% CPU on one core: this is unexpected. When using a fixed frequency with performance</code> or powersave</code>, everything is working fine. Operating system support</h2> As usual, the operating system offer from the vendor is clearly unsatisfying. They provide a weird "Bianbu Linux" distribution, which seems to be a heavily customised version of Ubuntu. When trying it, I managed to crash their version of openssl. They also provide modified versions of OpenWrt, Fedora, Armbian</a>... Overall, I have zero confidence in their software, so I would like to run a standard Debian OS on the board. The next article</a> will focus on how to install an upstream Debian on the board. It should be noted that third-party distributions started supporting this board: Armbian</a> (official support), Gentoo</a> (unofficial), Irradium</a> (never heard of it before). They all use the vendor u-boot and Linux kernel, for very good (or rather very bad) reasons, see below. Linux kernel support</h2> Unfortunately, we are very far from upstream support in the Linux kernel. The vendor maintains a custom Linux kernel, with two base kernel versions: bianbu 1.0 based on linux 6.1</a>, and bianbu 2.0 based on linux 6.6</a>. These kernels are themselves based on a kernel from SINOVOIP</a>, with small changes to DTS and cpufreq code. All the sources are available, which is good, but they are not based on the upstream kernel git tree, so it's hard to figure out what they changed. I experimented a bit with the bianbu 2.0 kernel, and the code seems to be low-quality. For instance, many of their drivers completely fail to build when built as module, so it's likely that they were never tested this way. On the upstream side, some initial patches have been posted by the community</a>, independently from the vendor. As of now, no code has been merged in the upstream kernel. Let's hope that progress will be made on this front. Rebuilding a kernel for cfarm95</h2> Somebody reported an issue when using vector instructions and signals on cfarm95</a> with the vendor kernel. It turns out to be a kernel bug that was fixed recently upstream</a>, but the vendor kernel does not include this fix. So, we are now building our own kernel for cfarm95, based on the vendor source + assorted patches. The build happens here: https://framagit.org/compile-farm/riscv64-kernel-build-bananapi-f3</a> VisionFive 2: the hardware 2024-05-06T00:00:00+00:00 This post is part of the Upstream RISC-V serie: The upstream RISC-V experience: running RISC-V hardware with upstream distros</a></li> VisionFive 2 Benchmarking RISC-V: VisionFive 2 vs the world</a></li> VisionFive 2: the hardware</a></li> </ul> </li> BananaPi BPI-F3 BananaPi BPI-F3: hardware and software overview</a></li> BananaPi BPI-F3: bootstraping upstream Debian</a></li> </ul> </li> </ul> The VisionFive 2 hardware: TL;DR</h2> The board comes with a cute transparent case</li> The hardware is relatively fast and stable</li> You absolutely need a good power supply, especially if you want to use a NVMe SSD</li> </ul> Hardware assembly</h2> The kit came with a cute transparent case that you have to assemble. Peeling off the protective adhesive can be a bit challenging at first, but it's otherwise straightforward to assemble. On the board itself, you have to apply the thermal pad on the CPU, then attach the fan on top of it (again, it can a bit challenging the first time). The kit also came with a 16 GB eMMC module that you can use as a replacement for a SD card, and an adapter to be able to plug your eMMC module on a standard micro-SD slot. I guess you could technically use both a SD card and the eMMC module simultaneously on the board, but it has limited usefulness (has anyone tried RAID?) </a> Regarding the NVMe, I found it difficult to attach a SSD firmly in the M.2 slot, but this is a general woe I have with all M.2 slots on all motherboards. Most SSD vendors don't give you a screw to attach it, and even if they do, it's often not working terribly well. Here, I ended up bending the NVMe so that it stays in place even with no screw. It doesn't feel right, but maybe this is how it's designed to be attached? </a> The DIP switch to select the boot mode is well-documented</a> elsewhere</a>, but let's include it for completeness. Interestingly, I couldn't manage to load u-boot directly from the eMMC while it works from the SD card, but that will be a topic for the next article. </a> Finally, you will certainly need a serial console, and again the UART pins are well-documented</a>. Hardware performance</h2> The VisionFive 2 has a StarFive JH7110 SoC, with 4 SiFive U74 cores at 1.5 GHz. More than one year ago, I had run benchmarks of the VisionFive 2</a> (version 1.2A). The numbers are very encouraging: the board is about as fast and power-efficient as a Raspberry Pi 3B/3B+. Crucially, it is also 50% to 75% faster per-core than previous RISC-V boards. In practice, it feels quite responsive in interactive usage over SSH, where the VisionFive 1 or the HiFive Unmatched always felt a bit slow. I suspect this is combination of reasonable CPU performance and good NVMe performance. A tale of weird kernel woes with a NVMe drive</h2> When trying out the board initially, I simply used a SD card and left the NVMe slot empty. It was working completely fine. However, as soon as I tried a NVMe SSD (with a patched kernel because there is still no upstream support), I had weird issues: while a Crucial P3 drive worked correctly, a Samsung 970 EVO Plus drive would fail spectacularly. The system completely locked up shortly after boot, with RCU stalls and hard lockup printed to the serial console: [ 34.799589] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 34.805520] rcu: 3-...0: (10 ticks this GP) idle=d754/1/0x4000000000000000 softirq=2411/2411 fqs=1053 [ 34.814830] rcu: (detected by 2, t=5256 jiffies, g=1493, q=1059 ncpus=4) [ 34.821618] Task dump for CPU 3: [ 34.824849] task:(udev-worker) state:R running task stack:0 pid:274 ppid:261 flags:0x0000000a [ 34.834769] Call Trace: [ 34.837220] [<ffffffff80944056>] __schedule+0x346/0xb0e [ 48.279586] watchdog: Watchdog detected hard LOCKUP on cpu 3</code></pre> I initially thought it could be a software issue in the kernel, since the patch series is not yet merged in the upstream kernel. But at this point the patch series was in its 9th iteration and other people seemed fine. The next logical culprit is the hardware. I had a second Samsung 970 EVO Plus drive and two other VisionFive 2 boards, so I tried all combinations: I got mostly the same lockups in all cases, while all boards were still working fine with the Crucial P3 drive. At this point, it started to smell like a power supply issue: maybe the Samsung SSD was simply consuming more power than the Crucial SSD? So I tested several power supplies and got interesting results: Akashi smart charger 5V/2.4A using 230V, model ALT2USBACCH: kernel stalls as soon as the NVMe drive is present, even if not using it at all</li> ALLNET USB charger 5V/3A using 230V, model KA1803A-EU: works fine when the NVMe drive is present and unused, but produces kernel stalls when trying to actually read/write data from the NVMe</li> my phone's power supply at 5V/2A (but 5V/6A in "WarpCharge mode", whatever it is), using 230V, model WC0506A3HK: works correctly in all cases, even when actively using the NVMe</li> </ul> Clearly, the rating of the power supply had a big impact, and the board seems to be able to make use of voltage above 5V. I ended up buying a laptop-class USB-C power supply, a 45 W Bluestork NB-PW-45-C</a>, and I had absolutely no NVMe issue since then. Conclusion</h2> While we are still very far from mainstream RISC-V adoption outside of the embedded space, this is a good board to start tinkering with Linux on RISC-V. In particular, this is the first RISC-V board with both reasonably good performance and reasonably good upstream software support. The rest of the serie will be dedicated to software support. The upstream RISC-V experience: running RISC-V hardware with upstream distros 2024-05-01T00:00:00+00:00 This post is part of the Upstream RISC-V serie: The upstream RISC-V experience: running RISC-V hardware with upstream distros</a></li> VisionFive 2 Benchmarking RISC-V: VisionFive 2 vs the world</a></li> VisionFive 2: the hardware</a></li> </ul> </li> BananaPi BPI-F3 BananaPi BPI-F3: hardware and software overview</a></li> BananaPi BPI-F3: bootstraping upstream Debian</a></li> </ul> </li> </ul> A foreword on cfarm.net</h2> In case you don't know about cfarm.net</a> (formerly the GCC Compile Farm), it is the longest-running and most comprehensive compile farm for free software developers. We provide SSH access to machines with many different operating systems and many different CPU architectures. Currently, we have 6 Linux distributions, 5 other OS, and 7 architecture families (with several variants such as big-endian and little-endian ppc64). The goal is to help developers to port, build and debug their projets on less common or hard-to-get hardware. So, of course, we have been very interested in RISC-V: it's not every day that a whole new CPU architecture gets its place in the landscape, and this specific architecture holds very interesting promises of openness. The many challenges of a new architecture</h2> Adding RISC-V machines to cfarm.net turned out to be a big challenge: for many years, there was no hardware that was both capable of running Linux and stable enough to provide a compilation service to many concurrent users. Like many projects, we resorted to QEMU emulation. It was working mostly fine (especially because user-mode emulation is reasonably fast on server-class x86 hardware), but it did not provide everything needed: debugging with gdb</code> was a challenge, and microarchitectural details are obviously different in QEMU compared to real hardware. When capable RISC-V hardware eventually became available, it came with very custom software, because the software ecosystem needed time and hardware to support this new architecture properly. In practice, it meant custom bootloader, custom kernel, and hacked-up Linux distros. Now, the lowest-levels of software support (libc, compiler, bootloader, kernel) are definitely ready for RISC-V. But support for specific hardware is another story. Making the case for upstream software and distros</h2> To properly integrate new RISC-V hardware in cfarm.net, we set ourselves a challenge: try to run them with upstream software and distros as much as possible. On our side, this makes the machines easier to manage and is more future-proof. In addition, this offers a more consistent environment to developers using the farm: if their code works on the farm, there is a good chance it will work on different RISC-V hardware and in a different software environment. This may feel a bit theoretical, but if you have ever worked with vendor SDKs in the embedded world, you know the nightmare it quickly becomes: ancient kernels with very questionable hacks, old compilation toolchains, low-level code that is highly specific to a particular hardware, poor security support, custom and hacky implementations of features that have since been upstreamed in a different way... Unfortunately, the RISC-V ecosystem tends to go a bit into this direction, although it is very far from the worst and we see RISC-V vendors making real efforts to upstream their code. To give more substance to the argument, here is a list of issues we have encountered so far because of non-upstream code: Starfive only provides custom Debian images with a full graphical environment</a>. This is something we can't use in the farm because it's a headless environment. In addition, they seem to be based on a Debian snapshot from 2022</a>, and many packages are customized by Starfive. This is not a good base to build and debug free software projects on top of it, because there is no guarantee that another RISC-V system would work the same way. </li> the u-boot shipped with the VisionFive boards is old and highly misconfigured, needing workarounds in the operating system. I reported the issues for the VisionFive v1 and even proposed fixes (one</a>, two</a>, three</a>). They were eventually merged after one year, but by this time the VisionFive v1 was already more or less abandoned. The u-boot shipped with VisionFive v2 is better configured, but doesn't provide an easy way to boot an upstream distro from NVMe</a>. </li> the non-upstream kernel on the VisionFive v1 caused confusion when reporting an issue in a userspace software</a>. I used the kernel of the VisionFive v1 as an example of correct behaviour, but that was actually due to a downstream change in Starfive's patched kernel: the upstream kernel code was still buggy. This wrong assumption caused much head-scratching to the third-party kernel developer working on a fix</a>. </li> </ul> Of course, I understand that upstream code does not appear magically, and there is always a fuzzy phase during which code is only available in downstream vendor forks. The point I want to make is the following: we should not be satisfied until the code is available upstream, as it is the only way to ensure its quality and longevity. Targeted hardware and current plans</h2> Thanks to the awesome support from RISC-V International</a>, we already have received several boards to work on: VisionFive v2</li> Milk-V Pioneer</li> Lichee Pi 4A</li> </ul> We plan to experiment with upstream distros on these boards, document the installation process, submit fixes upstream if required, and generally smooth out the path to get upstream software up and running on this hardware. Then, of course, we will make these boards and upstream software freely available to all cfarm.net developers</a> for the foreseeable future. Expect more blog articles and news as we progress through this plan. So far, we mostly target Debian, Alpine Linux and Arch Linux, but we are also exploring other OSes such as OpenBSD. It should be noted that Ubuntu generally has first-class upstream support for RISC-V boards</a>. Of course they still need to patch the Linux kernel to support the hardware, but this is currently unavoidable. In any case, it's very nice that a distro is able to offer official images with this level of quality. Given this, I believe that we have very little to contribute to Ubuntu to make it better. Long-term plans</h2> In the long term, we would like to work on vector instructions and virtualisation instructions support, but this is still a long way to go given the lack of hardware supporting these instructions. Moving from Pelican to Zola 2024-01-19T00:00:00+00:00 I have been using Pelican</a> for almost 10 years for this blog. It's very good and does the job, but I do find it a bit complex to use, and I wanted to switch to a new theme, so I looked at alternatives. Among the current alternatives, Zola</a> clearly emerged as the simplest and most efficient solution. From Pelican ...</h2> I must admit that Pelican is still working fine after 10 years, which is quite a feat for a Python project. However, over time, I found it was not that intuitive to use, mostly because I write blog articles very infrequently. Each time, I had to remember how it works: do I have to use the Makefile or the develop_server.sh</code> script? Did I have a virtualenv somewhere or is it a global install? Why is there a specific config file for publishing? I had to reinstall Pelican, and now it is complaining that some of my configuration is deprecated (but amazingly, my old Pelican configuration from 2014 still works without any change) But the real reason to switch is the theme. I don't like HTML/CSS, so I was quite happy with a very basic Pelican theme at the time, but it is showing its age now. While looking for a newer Pelican theme, I found nothing that I liked much. So, time to switch! Since Deuxfleurs</a> is using Zola a lot and it seems really simple to use, I went this route. ... to Zola</h2> Most tools advertise themselves as "easy to use", but that often means "easy to use for the authors". But in the case of Zola, it is actually impressively easy to use. Having a single binary instead of having to play with virtualenvs: a big win. Having just two main commands zola serve</code> and zola build</code>: a big win. There are many great themes to use as a base, and extending themes is simple and really well documented. Keeping compatibility with old URLs</h2> While migrating my content, I encountered a difficulty to maintain compatibility with the old URLs. I would like to avoid breaking URLs to old articles. Pelican generates URLs such as /blog/my-article-slug.html</code>. But Zola would generate /blog/my-article-slug/</code> instead, by putting the content in a index.html</code> file within that directory. This behaviour is currently not configurable</a>. Given this, keeping compatibile URLs seemed like an impossible task, but there is actually a way if you are a bit creative with aliases. You just have to add an alias with the old name in the front matter of your article: +++ title = "My page title" aliases = ["posts/old-page-name.html"] +++</code></pre> This will still generate the file as usual for Zola (let's say blog/my-page/index.html</code>, exposed as /blog/my-page/</code>), BUT it will also generate a redirect page at /posts/old-page-name.html</code>. This way, old URLs still work and they redirect to the new URL. Check issue #840</a> for more details. Theme fork</h2> I chose the jimmyff/zola-inky</a> theme because it fits my requirements: it's very simple but elegant. In addition, I really like the way it lists articles by year, and the fact that you can show a small excerpt of each article in this list. However, it was missing a few things for my use-case, so I forked it to add the following features: styling for tables</li> allow to override the content of footers easily</li> use a different set of icons (I needed the Mastodon logo)</li> stop loading external fonts from Google and scripts from CDN</li> </ul> The fork is available here</a>. I should find time to try to upstream some of these changes. In addition, I changed the style of tags</a>, but this is a local change for my blog, not part of the theme (it feels too specific). Introduction to DevOps for experimental research 2023-09-18T00:00:00+00:00 I recently gave a full-day training session called Introduction to DevOps for experimental research with several colleagues. Why DevOps in research?</h2> At first glance, DevOps and experimental research in computer science have widely different goals: most of the time, researchers perform experiments using software but they don't have to maintain production systems in the long run! However, we do have to build, deploy and configure complex software ecosystems to be able to do experimental research. In this process, we often need very fine control over the deployment process, and we need to be extra sure about our results. Imagine: you measure the impact of a specific parameter of a PostgreSQL database on the performance of various applications, and you happily write and publish a paper about it. Two years later, somebody writes you an email stating that they obtain completely different performance results with this parameter. What happened? Are you sure that you didn't mess up your deployment or forgot to restart the database after changing its configuration? Do you still have your deployment scripts around, and can you still get them to run in the same conditions? Maybe a newer version of PostgreSQL changes the performance results completely? I believe that these kind of problems in research can (partially) be solved by adapting ideas and tools from the DevOps ecosystem. In particular, automation is a core concept in DevOps and is definitely 100% necessary in research: this is the first step to improve the quality of research works that are based on deploying complex software. DevOps tools in research</h2> Researchers are more and more using classical DevOps tools: Git, Docker, Terraform, Kubernetes, Ansible... For general automation, Gitlab CI is particularly appreciated in research because it can be run on independant on-premise infrastucture, which is often a requirement for publicly-funded research (see this Gitlab CI gallery</a> for many examples) But researchers also develop tools that are specific to research problems. To give just a few examples: Kameleon</a>: a very flexible tool to build customized software appliance (VM images, Docker images, bare-metal images...). Similar to Packer but much more flexible.</li> EnOSlib</a>: a Python library allowing to easily provision hardware resources on research platforms, and then configure them through Ansible</li> scientific workflow engines such as Nextflow</a>, Pydra</a>, SnakeMake</a> and many others</a></li> </ul> Focus on reproducibility</h2> A specific concept we need in research is reproducibility. Reproducibility means that somebody else should be able to take your code and your data and use it to reproduce your research results. It looks simple, but this is very hard to achieve in practice: your results also depend on the software environment (OS, libraries, compiler, interpreter...) and the hardware you run it on. In addition, this environment evolves very rapidly: if you write moderately complex Python code today, there is a good chance that you won't be able to run it again five years from now. The language itself and its interpreter will have changed, your OS will have new versions of everything, your Python dependencies will no longer exist or will have broken compatibility, etc. DevOps tools are not specifically designed for reproducibility: sure, you can build a Docker image with your code, its dependencies, and the right version of the Python interpreter. This image will probably still run fine in five years. But what if you or somebody else wants to extend this work, for instance changing a small parameter or updating a single dependency? In a production system, how would you update dependencies to fix known security issues that were uncovered in the last five years? For both problems, you would need to rebuild the Docker image: hopefully, you still have a Dockerfile around, but it will most likely fail to build (see the examples in the training material below). If you think about it, DevOps is all about deploying and iterating as fast as possible, and this does not particularly encourage reproduciblity. In a modern production systems where a new version is deployed every day, why would you even try to rebuild your software stack from two years ago? However, the DevOps tooling situation is improving: tools that focus on full reproducibility such as Nix and Guix are getting mainstream, and initiatives around the "Software Supply Chain" such as OpenSSF</a> are targeting the provenance of software dependencies, which helps improve reproducibility. Training material</h2> This blog article is purposefully short: you can find all material used for the training here</a>. This includes slides but also example code for several of the tools that were demonstrated: Docker, Packer, Nix, Guix, Ansible, Terraform and EnOSlib. Many thanks to the colleagues that contributed to the training and the associated material. The training does not go a lot in-depth into each tool: the goal is to give an overview of the ecosystem and understand what each DevOps tool can and cannot do. It is then up to you to choose the right tool for your specific research problem, and then learn to use this tool. Benchmarking RISC-V: VisionFive 2 vs the world 2023-01-21T00:00:00+00:00 This post is part of the Upstream RISC-V serie: The upstream RISC-V experience: running RISC-V hardware with upstream distros</a></li> VisionFive 2 Benchmarking RISC-V: VisionFive 2 vs the world</a></li> VisionFive 2: the hardware</a></li> </ul> </li> BananaPi BPI-F3 BananaPi BPI-F3: hardware and software overview</a></li> BananaPi BPI-F3: bootstraping upstream Debian</a></li> </ul> </li> </ul> Introduction</h2> I recently got my "super early bird" version of the VisionFive 2 RISC-V board</a>. As the documentation says, it is supposed to be "the world’s first high-performance RISC-V single board computer (SBC) with an integrated GPU". Since I have access to several kind of RISC-V, ARM and x86 boards, let's see if the claim about performance is true! We will look both at processing performance and energy efficiency. Updated 2023-01-22: added Kobol Helios64 performance results from Max Updated 2023-01-23: added results (performance and power) for Raspberry Pi 3B+ Updated 2023-01-24: re-done all SBC power measurements, significant changes for VisionFive 2</li> fixed completely incorrect performance measurements for Raspberry Pi 3B+ caused by a faulty USB cable (causing a huge 2.5x drop in performance!)</li> re-done measurements for Raspberry Pi 1 with Debian 11 and without the faulty USB cable</li> added results for Raspberry Pi 3B, it was not fried after all, it was also the faulty USB cable</li> re-done Raspberry Pi 4 power measurements to be more comparable (avoid POE)</li> </ul> Updated 2024-05-15: added Raspberry Pi 5 performance results from Denis </a> Power measurement setup. The VisionFive 2 is visible at the bottom with its serial cable, the wattmeter is on the left. The other visible boards are Raspberry Pis. A disclaimer on methodology</h2> Benchmarking CPU performance correctly requires a huge software and hardware expertise, and I can certainly not claim to have such an expertise. I have chosen two basic computing primitives, hoping that they are representative enough: crypto (sha1 and chacha20-poly1305 using openssl) and decompression (xz</code>). All numbers shown in this article are very "unscientific": I made no formal repetition to account for variability, and there are many factors that I purposefully ignore (kernel version, software version, compiler...). That being said, I tried to document these parameters as much as possible to help further analysis. Overall, the goal is to give a rough idea of the CPU performance and power efficiency you can expect from RISC-V hardware. Hardware and software environment</h2> The VisionFive 2 has a StarFive JH7110 SoC, with 4 SiFive U74 cores at 1.5 GHz. The original VisionFive had a StarFive JH7100 SoC with 2 SiFive U74 cores at 1.2 GHz. It had known hardware design issues: frequent L2 cache flushing needed because of a non-coherent bus</a> and a slow RAM controller</a>. So, the new SoC should be significantly faster. Software-wise, I built a Linux kernel using the non-upstream repository</a> (5.18-based for VisionFive 1, and 5.15-based for VisionFive 2). I built a Debian rootfs using the Debian guide for VisionFive</a>. That guide works almost the same way for VisionFive 2, but that (as well as the upstream status for kernel support) will be for another article. For other systems used in the comparison, they mostly run Debian or Ubuntu, with a few exceptions (NixOS, Armbian). CPU performance</h2> Here are the three benchmarks I will use: openssl speed -evp sha1 openssl speed -evp chacha20-poly1305 # https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.tar.xz (116606704 bytes) time xz -d < /dev/shm/linux-5.10.tar.xz > /dev/null</code></pre> The xz</code> benchmark uses decompression of a known file to ease reproducibility, and this file is stored in memory (/dev/shm</code>) to make sure we have no disk I/O. All benchmarks are using a single CPU core. I converted all results into MB/s for easier comparison, taking the largest block size for the openssl results (16 KiB). As a reminder, one MB equals 1000000 bytes. For the xz</code> benchmark, the real elapsed time is used. To ease comparisons with other hardware, I computed a "speedup" of each board compared to the VisionFive 2: 1x</code> means the same performance, 2x</code> means twice as fast, 0.33x</code> means three times as slow, etc. You can find the full output of the benchmarks for each machine here</a>. Most hardware is either running locally, from the Compile Farm</a>, or from Grid'5000</a>. The Celeron G1840T system belongs to Deuxfleurs</a>. The Kobol Helios64 result is from Max. Raspberry Pi 3B+ and 4 are courtesy of $DAYJOB. Raspberry Pi 5 result is from Denis. Hardware</th> sha1</th> chacha20</th> xz</code> decompr.</th></tr></thead> RISC-V</td> </td> </td> </td></tr> VisionFive 2 Debian unstable</td> 97.5 MB/s</td> 50.8 MB/s</td> 4.66 MB/s</td></tr> VisionFive 1 (gcc91) Debian unstable</td> 64.1 MB/s (0.66x)</td> 33.0 MB/s (0.65x)</td> 2.68 MB/s (0.58x)</td></tr> HiFive Unmatched gcc92, Ubuntu 22.04</td> 34.7 MB/s (0.36x)</td> 40.6 MB/s (0.80x)</td> 3.12 MB/s (0.67x)</td></tr> ARM / ARM64</td> </td> </td> </td></tr> Raspberry Pi 1 Debian 11, armv6l</td> 27.3 MB/s (0.28x)</td> 22.9 MB/s (0.45x)</td> 0.928 MB/s (0.20x)</td></tr> Raspberry Pi 3B Debian 11</td> 149 MB/s (1.53x)</td> 188 MB/s (3.70x)</td> 4.35 MB/s (0.93x)</td></tr> Raspberry Pi 3B+ Debian 11</td> 174 MB/s (1.79x)</td> 225 MB/s (4.44x)</td> 5.02 MB/s (1.08x)</td></tr> Raspberry Pi 4B Debian 11</td> 192 MB/s (1.97x)</td> 266 MB/s (5.24x)</td> 6.82 MB/s (1.46x)</td></tr> Raspberry Pi 5 RaspiOS, Debian 12</td> 1437 MB/s (14.7x)</td> 703 MB/s (13.9x)</td> 14.4 MB/s (3.09x)</td></tr> Kobol Helios64 Armbian 22.02.1</td> 979 MB/s (10x)</td> 323 MB/s (6.36x)</td> 7.64 MB/s (1.64x)</td></tr> Ampere eMAG gcc185, CentOS 8</td> 903 MB/s (9.26x)</td> 296 MB/s (5.83x)</td> 12.3 MB/s (2.64x)</td></tr> Mac M1 gcc103, Debian 12</td> 2244 MB/s (23x)</td> 1710 MB/s (33.7x)</td> 21.0 MB/s (4.51x)</td></tr> x86_64</td> </td> </td> </td></tr> Celeron G1840T NixOS 22.11</td> 599 MB/s (6.14x)</td> 678 MB/s (13.3x)</td> 11.2 MB/s (2.40x)</td></tr> Xeon Gold 6130 dahu.g5k, Deb. 11</td> 1045 MB/s (10.7x)</td> 2611 MB/s (51.4x)</td> 17.1 MB/s (3.67x)</td></tr> i7-8086K Ubuntu 20.04</td> 1414 MB/s (14.5x)</td> 2971 MB/s (58.5x)</td> 22.6 MB/s (4.85x)</td></tr> AMD EPYC 7642 neowise.g5k, Deb. 11</td> 1706 MB/s (17.5x)</td> 1796 MB/s (35.4x)</td> 17.0 MB/s (3.65x)</td></tr> AMD EPYC 7513 grat.g5k, Deb. 11</td> 1875 MB/s (19.2x)</td> 2460 MB/s (48.4x)</td> 21.3 MB/s (4.57x)</td></tr> </tbody></table> SHA1 and Chacha20-poly1305 results are very variable, which may be due to optimizations in certain versions of OpenSSL (vectorisation, assembly implementation) or even hardware acceleration for SHA1. They also seem to be sensitive to memory bandwidth: the Raspberry Pis have much better memory bandwidth than the RISC-V boards. In contrast, xz</code> results seem much more representative of raw CPU performance (clock frequency, CPU cache, out-of-order execution, memory access patterns...) To get clock frequency out of the equation, I am now showing xz</code> results normalized by the clock frequency, measured in "CPU cycles per processed byte" (basically dividing clock frequency by xz</code> performance). It should give an idea of the overall performance of the CPU architecture for this specific decompression task. Beware, lower values are now better! Hardware</th> Max clock frequency</th> xz -d</code> cycles/byte (lower is better)</th></tr></thead> VisionFive 2</td> 1.50 GHz</td> 322</td></tr> VisionFive 1</td> 1.20 GHz</td> 448 (0.72x)</td></tr> HiFive Unmatched</td> 1.20 GHz</td> 385 (0.84x)</td></tr> Raspberry Pi 1</td> 0.70 GHz</td> 754 (0.43x)</td></tr> Raspberry Pi 3B</td> 1.20 GHz</td> 276 (1.17x)</td></tr> Raspberry Pi 3B+</td> 1.40 GHz</td> 279 (1.15x)</td></tr> Raspberry Pi 4B</td> 1.50 GHz</td> 220 (1.46x)</td></tr> Raspberry Pi 5</td> 2.40 GHz</td> 167 (1.93x)</td></tr> Kobol Helios64</td> 1.80 GHz</td> 236 (1.37x)</td></tr> Ampere eMAG</td> 3.00 GHz</td> 244 (1.32x)</td></tr> Mac M1</td> 3.00 GHz</td> 143 (2.25x)</td></tr> Celeron G1840T</td> 2.50 GHz</td> 223 (1.44x)</td></tr> Xeon Gold 6130</td> 3.70 GHz</td> 216 (1.49x)</td></tr> i7-8086K</td> 5.00 GHz</td> 221 (1.45x)</td></tr> AMD EPYC 7642</td> 3.30 GHz</td> 194 (1.66x)</td></tr> AMD EPYC 7513</td> 3.65 GHz</td> 171 (1.88x)</td></tr> </tbody></table> Here are some of the main highlights of these results: VisionFive 2 single-core performance is 52% to 74% higher than VisionFive 1. This is very good compared to the 25% clock frequency improvement. When normalizing by the clock frequency, the VisionFive 2 is 39% faster per MHz compared to the VisionFive 1</li> VisionFive 2 is also 25% to 50% faster than the HiFive Unmatched. When normalizing by the clock frequency, the VisionFive 2 is 20% faster per MHz compared to the Unmatched. The Unmatched was itself slightly faster than the VisionFive 1 on a single-core basis.</li> VisionFive 2 is roughly as fast as a Raspberry Pi 3B/3B+ on the xz</code> benchmark, but much slower for SHA1 and Chacha20.</li> VisionFive 2 is still around 1.5 slower than a Raspberry Pi 4 (and 5 times slower on Chacha20)</li> </ul> Here are other interesting insights: The Raspberry Pi 1 always felt really slow. Well, now I know it's objectively really slow. Even when taking into account its low clock frequency of 700 MHz, performance per MHz is still really poor.</li> The Intel CPUs (from 2014, 2017 and 2018 respectively) have very similar performance per MHz for this task, despite being very different in terms of frequency, number of cores and price. This indicate that they basically share the same kind of architectural design.</li> The Raspberry Pi 4 and the Helios64 have good performance per MHz for a SoC, even comparable to an $1900 Intel CPU from 2017! Of course, the Intel CPU has much more cores, and there may be other workloads where Intel CPUs are much better.</li> The AMD EPYC CPUs (Zen 2 and Zen 3) have very good performance per MHz for this workload, and there is a clear improvement from Zen 2 to Zen 3.</li> The Raspberry Pi 5 has roughly the same performance per MHz as a Zen 3 CPU, which is really impressive.</li> As always, despite being already a few years old, the Mac M1 is even more impressive and easily smashes all other processors I could test on a per-MHz basis.</li> </ul> As a final note: remember that this is a single benchmark and is not representative of all kind of computing workloads. I suspect xz</code> to be quite sensitive to the amount of CPU cache and to memory latency. Energy consumption</h2> Now that we have an idea of CPU performance, the other important criteria is energy consumption. Here, I am interested in whole-system energy consumption. I could only measure it for systems I have locally, so only a subset of the previous machines are tested here. Technically, I could have used wattmeters available on Grid'5000, but it makes little sense to compare the power consumption of a big server with that of a small embedded board. All figures below are taken using a basic Perel plug-in wattmeter on 230V. The wattmeter gives the "active" (or real) power in Watts, as well as the power factor</a>. All figures include the power transformer, which is either: an Akashi ALT2USBACCH USB transformer designed for up to 2.4A (VisionFive 1 & 2, Raspberry Pis) ; the stock Lenovo power transformer (Celeron G1840T) ; or an ATX power supply (HiFive Unmatched, i7-8086K). For each system, I measure power consumption in the following situations: idle ; 1 CPU core at 100% ; half of CPU cores at 100% ; all CPU cores at 100% (ignoring hyper-threads). Each measurement is run for only a few seconds (still waiting for a steady-state) to avoid thermal throttling. The workload is a simple infinite loop in bash: while :; do :; done</code>. All systems run Linux (various versions and distributions), have one NIC up, and no screen or other peripheral attached. Note: I am not very confident in the absolute power values shown below (because I don't really trust the wattmeter or the USB transformer). However, since I did all measurements in the same conditions, the values are comparable with each other. Hardware</th> Idle</th> 1 core</th> Half cores</th> All cores</th></tr></thead> VisionFive 2 4 cores, 8 GB RAM</td> 7.4 W</td> 10.4 W</td> 11.2 W</td> 13.1 W</td></tr> VisionFive 1 2 cores, 8 GB RAM</td> 10.6 W</td> 11.1 W</td> -</td> 11.6 W</td></tr> HiFive Unmatched 4 cores, 16 GB RAM</td> 56.8 W</td> 57.7 W</td> 58.6 W</td> 60.7 W</td></tr> Raspberry Pi 1 1 core, 512 MB RAM</td> 5.9 W</td> 6.2 W</td> -</td> -</td></tr> Raspberry Pi 3B rev 1.2 4 cores, 1 GB RAM</td> 4.5 W</td> 6.5 W</td> 8.7 W</td> 13.8 W</td></tr> Raspberry Pi 3B+ 4 cores, 1 GB RAM</td> 7.0 W</td> 9.7 W</td> 12.2 W</td> 18.0 W</td></tr> Raspberry Pi 4B rev 1.5 4 cores, 2 GB RAM</td> 4.6 W</td> 6.8 W</td> 8.3 W</td> 11.1 W</td></tr> Celeron G1840T 2 cores</td> 12 W</td> 18 W</td> -</td> 23.5 W</td></tr> i7-8086K 6 c. / 12 threads</td> 23.4 W</td> 61.5 W</td> 79.5 W</td> 112.7 W</td></tr> </tbody></table> Clearly, the VisionFive 2 is quite power-efficient compared to the older RISC-V boards. According to its documentation, it can run without any headsink or fan for bursty loads (e.g. web browsing), but a fan is recommended for long computations. This is consistent with my power consumption measurements. Interestingly, the Raspberry Pi 3B+ has a similar power profile as the VisionFive 2. This makes sense because they are in the same class of devices: same amount of cores, similar maximum clock frequency, similar performance. But it's still noteworthy that the relatively young SoC found on the VisionFive 2 has a power consumption that is so similar to that of the more mature SoC found on the Raspberry Pi 3B+. We can also observe that Intel is much better at dynamic frequency scaling, which helps to achieve low power usage when the CPU is idle. As far as I know, the SoC in the VisionFive 1 and the HiFive Unmatched have no frequency scaling, which explains their near-constant power usage. The VisionFive 2 does have frequency scaling, so it's already much better (-45% power usage when idle compared to fully loaded). Here are some details about the hardware to put these numbers into context: VisionFive 1: no fan, kernel 5.18 (Debian). Power consumption changes significantly with die temperature (9 W idle at 36 °C, 10.4 W idle at 50 °C)</li> VisionFive 2: no fan, kernel 5.15 (Debian), no NVMe, 100M NIC. Using the gigabit NIC would add 0.5 W of power usage.</li> Raspberry Pi 3B: rev 1.2, 1 GB RAM, no fan, kernel 5.10 (Debian 10). 600 MHz idle frequency, 1.20 GHz max frequency.</li> Raspberry Pi 3B+: 1 GB RAM, no fan, kernel 5.10 (Debian 10). 600 MHz idle frequency, 1.40 GHz max frequency.</li> Raspberry Pi 4B: rev 1.5, 2 GB RAM, no fan, kernel 5.10 (Debian 10). 600 MHz idle frequency, 1.50 GHz max frequency.</li> Celeron G1840T: 800 MHz idle frequency, 2.5 GHz max frequency. Lenovo ThinkCentre M73, 4 GB DDR3, ST500LM021-1KJ15 disk.</li> i7-8086K: 800 MHz idle frequency, 4 GHz max frequency, 5 GHz turbo frequency. ASRock H310CM-HDV/M.2 motherboard, 16 GB + 8 GB DDR4, Samsung 980 500GB NVMe, ATX power supply, 2 case fans</li> </ul> Note: earlier versions of this article used some POE power measurements from a switch (for the Raspberry Pis, with the POE hat). After re-doing the measurements with the USB power supply and plug-in wattmeter, it turns out that power measurements given by the POE switch were substantially lower than the wattmeter (probably because the POE switch measurements do not include the AC-to-DC power converter). Moreover, POE values were not stable. In the end, I decided to remove these POE values and only use the USB power supply to enable a fair comparison. CPU performance vs. energy</h2> Now that we have both CPU performance and energy consumption, we can mix the two results to look at energy efficiency. The most reliable figure in the table below is single-core efficiency: it is obtained by simply dividing the result of the single-core performance benchmark for xz</code> by the single-core power consumption. I also extrapolate some figures for all-cores efficiency, but this value should be taken with a grain of salt: it is obtained by multiplying single-core performance by the number of cores (excluding hyper-threads) and dividing the total by the measured all-cores power consumption. Many effects such as thermal throttling, frequency boost for single-core load, and shared cache between cores may decrease the actual all-cores performance and thus decrease the actual all-cores efficiency compared to the figures below. Hardware</th> Single-core efficiency</th> All-cores efficiency (extrapolated)</th></tr></thead> VisionFive 2 (4 cores)</td> 0.448 MB/s/W</td> 1.42 MB/s/W</td></tr> VisionFive 1 (2 cores)</td> 0.241 MB/s/W</td> 0.462 MB/s/W</td></tr> HiFive Unmatched (4 cores)</td> 0.0541 MB/s/W</td> 0.206 MB/s/W</td></tr> Raspberry Pi 1 (1 core)</td> 0.150 MB/s/W</td> </td></tr> Raspberry Pi 3B (4 cores)</td> 0.670 MB/s/W</td> 1.26 MB/s/W</td></tr> Raspberry Pi 3B+ (4 cores)</td> 0.517 MB/s/W</td> 1.12 MB/s/W</td></tr> Raspberry Pi 4B (4 cores)</td> 1.00 MB/s/w</td> 2.46 MB/s/W</td></tr> Celeron G1840T (2 cores)</td> 0.622 MB/s/W</td> 0.953 MB/s/W</td></tr> i7-8086K (6 c. / 12 threads)</td> 0.367 MB/s/W</td> 1.20 MB/s/W</td></tr> </tbody></table> Overall, the VisionFive 2 is much more energy-efficient than existing RISC-V boards: it is 2 to 3 times more energy-efficient than the VisionFive 1, and 7 to 8 times more energy-efficient than the Unmatched. It may seem counter-intuitive that the Unmatched is so inefficient, but that's probably because of its larger form factor, power-hungry PCIe and DDR4, and the need for an ATX power supply that may not be super efficient at low power load. Similarly, even though x86_64 hardware is much faster than the VisionFive 2 (2.4 times to 4.8 times faster), it has roughly the same energy efficiency! If you have moderate computing needs, the VisionFive 2 is an efficient alternative to bigger systems. Compared to the Raspberry Pi 3B and 3B+, the VisionFive 2 again has similar energy-efficiency. This makes sense because it has roughly the same performance and the same power consumption. Finally, the Raspberry Pi 4 is the real winner on the efficiency metric: the VisionFive 2 is half as energy-efficiency as a Raspberry Pi 4. Conclusion</h2> When looking at single-core CPU performance, the VisionFive 2 is roughly 75% faster than the original VisionFive. Since it has twice the core count, that means a +150% total performance increase. And since it has a similar power consumption, it is also 2 to 3 times more energy-efficient. So that's definitely a very big improvement. Compared to the HiFive Unmatched (which is not even technically a SBC), the VisionFive 2 still outperforms it by 50%, and is 7 to 8 times more energy-efficient. So, as far as I can tell, the claim about it being a "high-performance RISC-V SBC" is true. When comparing with Raspberry Pis, the VisionFive 2 is about as fast as a Raspberry Pi 3B+, although much slower on memory-heavy benchmarks, and also as energy-efficient. However, it is still 46% slower than a Raspberry Pi 4, and two times less energy-efficient. As far as I can tell, both SoC are 28 nm, so we would ideally expect the same energy-efficiency. Compared to low-power x86_64 systems, the VisionFive 2 is of course slower when looking at raw performance, but at the same time it is as energy-efficient. This is a general advantage that SBCs have over more complete systems: they have much less peripherals, are less extensible and have generally lower performance, but they are much more energy-efficient. Again, remember that all figures discussed here are approximate, and specific benchmark results cannot be extrapolated to generic performance results for all applications. Overall, the VisionFive 2 is a big step in the right direction, and this kind of RISC-V hardware can definitely compete with recent ARM boards since they have very similar performance-energy tradeoffs. More pictures</h2> </a> VisionFive 2 in its box (I removed the antistatic wrapping) </a> Front with audio, 4xUSB, HDMI, 2xNIC (with one being a 100M NIC, specific to the super early bird version) </a> Rear with USB-C power input, reset button, GPIOs </a> Back with NVMe M.2 slot, micro-SD card slot Debugging eBPF-enabled programs in Docker 2022-08-13T00:00:00+00:00 These days, I'm adding XDP offloading to l2tpns</a>, a L2TP server used in production by several non-profit ISPs in France. While doing that, I need to test if l2tpns can successfully load XDP programs into the kernel. But I don't want to run that directly on my Debian host: it might break network connectivity, and in addition l2tpns is updating the routing table of the kernel. So, let's just run l2tpns in Docker and allow it to break things! It turns out to be not so easy. eBPF and XDP</h3> As a reminder, XDP is a kernel mechanism that allows you to load custom eBPF programs that will execute right in the network device driver. You write your eBPF program in C, load it in the kernel from userspace with a simple system call, and from that point on, your program can process network packets in the kernel, before the rest of the kernel has even started parsing the packets! For a project like l2tpns, this is extremely powerful, fast and flexible, because we should be able to offload the bulk of encapsulation/desencapsulation work to the kernel while keeping a lot of flexibility. That being said, the eBPF ecosystem is still young and is moving fast, and the whole software architecture to make this work is actually very complex. In the end, you always end up with weird errors that can be hard to track down, and especially when trying to run XDP in Docker! What I want to debug</h3> In this case, I'm extending l2tpns so that it loads XDP programs on network interfaces when it starts. The basic process looks like this with libbpf</a> (error handling omitted): char[] xdp_filename = "/path/to/xdp_prog.o"; char[] if_name = "eth0"; __u32 ifindex; int prog_fd = -1; struct bpf_object *obj; __u32 xdp_flags = 0; // Load XDP program into the kernel bpf_prog_load(xdp_filename, BPF_PROG_TYPE_XDP, &obj, &prog_fd); // Find network interface by name ifindex = if_nametoindex(if_name); // Attach XDP program to network interface bpf_set_link_xdp_fd(ifindex, prog_fd, xdp_flags);</code></pre> See the xdp-tutorial repository</a> for more complete examples, but as a starting point this is the basic functionality I want to debug in Docker. Most programs manipulating eBPF are leveraging libbpf</a> to do the hard work. As such, the debugging steps below can be generalized to any eBPF-enabled userspace program. Basic Docker setup</h3> To keep things simple, I only want to run l2tpns in a container. I will keep developing and building on my Debian host. So, let's get started with a simple Dockerfile that installs the required libraries and creates a minimum config to make l2tpns happy: # Dockerfile used to test l2tpns during development. # Do not use in production! FROM debian:bullseye RUN mkdir -p /etc/l2tpns; echo "10.10.10.0/24" > /etc/l2tpns/ip_pool RUN apt update && apt install -y libbpf0 libcli1.10 iproute2 WORKDIR /src VOLUME /src ENTRYPOINT ["/src/l2tpns"]</code></pre> My Debian host is running Bullseye, so I use the same distro in the container to make sure I have the same libraries. Build the image from the Dockerfile: $ docker build - -t l2tpns:latest < Dockerfile</code></pre> Then give it a try (from the host, in the l2tpns git repository): $ make -j4 # To send all logs to stderr $ sed -i -e 's/set log_file/#set log_file/' etc/startup-config.default # Run docker image with parameters $ docker run -it --rm -v $PWD:/src l2tpns:latest -c etc/startup-config.default</code></pre> This yields an error: Can't open /dev/net/tun: No such file or directory</code></pre> Ok, this first error is unrelated to XDP: l2tpns needs to create a tun</code> interface and it cannot. Let's fix this: $ docker run -it --rm -v $PWD:/src --cap-add=NET_ADMIN --device=/dev/net/tun l2tpns:latest -c etc/startup-config.default</code></pre> Now we start seeing the interesting stuff: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.</code></pre> From this point on, I will omit the tun-related options from the examples, but for the specific case of l2tpns they are still needed Allowing the BPF syscall</h3> Obviously, to load a eBPF program into the kernel, you need to do a syscall at some point. This is role of the BPF syscall, that is also used for other eBPF-related functionalities. There is a new CAP_BPF</code> capability that enables the BPF syscal for unprivileged users. This was introduced in Linux 5.8 according to capabilities(7)</code>, which is good because Debian bullseye runs a 5.10 kernel. Let's try: $ docker run -it --rm -v $PWD:/src --cap-add=BPF l2tpns:latest -c etc/startup-config.default</code></pre> Result: docker: Error response from daemon: invalid CapAdd: unknown capability: "CAP_BPF".</code></pre> Crap. Maybe my Docker version is too old to know about this capability. Let's just use a bigger hammer and settle for CAP_SYS_ADMIN</code>, which gives a lot of priviledges, including BPF: $ docker run -it --rm -v $PWD:/src --cap-add=SYS_ADMIN l2tpns:latest -c etc/startup-config.default</code></pre> Result: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.</code></pre> Well, this is the exact same error as before! Configuring limits in the container</h3> Helpfully, the error message mentions something about the "memlock" limit. Let's have a look at the limits in a simple Debian bullseye container: $ docker run -it --rm debian:bullseye /bin/sh -c "ulimit -a"</code></pre> Since ulimit</code> is a shell builtin, we cannot run it directly as the command from Docker. Result: time(seconds) unlimited file(blocks) unlimited data(kbytes) unlimited stack(kbytes) 8192 coredump(blocks) unlimited memory(kbytes) unlimited locked memory(kbytes) 64 process unlimited nofiles 1048576 vmemory(kbytes) unlimited locks unlimited rtprio 0</code></pre> We are interested in the "locked memory" limit. 64 KB is indeed on the low side (try comparing this value with your host system). Looking at the relevant Docker documentation</a>, we find there's an option we can pass to Docker to raise this limit: $ docker run -it --rm --ulimit memlock=1073741824 debian:bullseye /bin/sh -c "ulimit -l" 1048576</code></pre> That looks much better! Now on the real container: $ docker run -it --rm -v $PWD:/src --ulimit memlock=1073741824 --cap-add=SYS_ADMIN l2tpns:latest -c etc/startup-config.default libbpf: map 'sessions_table': failed to create: Invalid argument(-22)</code></pre> Ok, we still have an error, but it looks application-specific (libbpf fails to create a map that is defined in the l2tpns code). EDIT 2022-08-15: it turned out to be indeed a programming error: BPF array maps MUST have a 32-bits key size and I was trying to create a map with a 16-bits key size. It's hard to debug because there is no detailed error reporting, the syscall simply fails with EINVAL. Here is what strace</code> is seeing, not really helpful: bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=2, value_size=20, max_entries=60000, map_flags=0, inner_map_fd=0, map_name="sessions_table", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 72) = -1 EINVAL (Invalid argument)</code></pre> After fixing this bug, libbpf happily creates the map in the kernel: libbpf: map 'sessions_table': created successfully, fd=8</code></pre>Conclusion</h3> So far, after a bit of efforts, I could get basic BPF functionalities to work in a Docker container for debugging purposes! Of course, for further debugging, you would need tools such as bpftools</code> to dump the XDP programs, observe the behaviour of the program by sending packets to the interface, and so on. But this part of the work should be quite similar whether using Docker or not. If this turns out to be more difficult than expected, I will update the article! Teaching routing and network security — Daily log 2019-02-13T00:00:00+00:00 Today I have been mostly teaching or preparing upcoming courses. I also had a nice lunch discussion with colleagues</a> on DNS and the role of transaction IDs, but that story will have to wait until tomorrow! Teaching routing</h3> I gave another networking course for first year's students today. This was the first practical session where they actually had to plug some cables around: you can imagine the excitement but also the mess! To make things even easier, the course was in a new networking lab I had never been before, so I had to improvise with the hardware lying around. The students learnt how to configure network interfaces (ifconfig</code>, route</code> & netstat</code> on FreeBSD), and they had to use their prior knowledge of packet capture and ping</code> to troubleshoot when things didn't work as expected. They had to form a simple "chain" topology (shown below) with two subnets, and the computer in the middle needed to be configured as a router. They needed to figure out that static routes were required on both edge computers, so that they knew how to reach the remote subnet through the router. Finally, they looked in details at the behaviour of ARP and the scope of MAC addresses. </a> Network security course</h3> I then prepared an upcoming practical session on network security with a colleague working for Quarkslab</a>. I already have a good part of the course ready from last year on firewalling and advanced uses of iptables</code> (including compiling custom BPF programs!). My colleague wants to add a part where students will practice ARP spoofing, so we looked at how to integrate that with the existing content. Interestingly, he showed me how to automate virtual machine generation using Packer</a>. This should be really helpful for future teachers in this course: they will be able to easily customize and rebuild the virtual machine images used by the students! Last year, I installed and configured the virtual machine manually, which makes it hard to update it or apply the same modifications to a new VM image. Multipath scheduling simulation — Daily log 2019-02-12T00:00:00+00:00 Multipath scheduling</h3> Today I mostly worked on my current research project, a simulator of multipath multistream scheduling algorithms. Multipath scheduling is needed when you want to transmit data over several concurrent paths: which piece of data should be sent on which path? This problem has been made visible by Multipath TCP</a>, since the Linux implementation includes several schedulers</a> that can be changed at runtime. Several new schedulers for MPTCP are being proposed in the academic literature every year: the original LowRTT</a> and its evaluation</a>, Delay-Aware Packet Scheduling</a>, BLocking ESTimation</a>, Earliest Completion First</a>, and many</a> others</a>. All these algorithms mostly differ in the objective function they try to optimize or in assumptions that can be made for specific data flows (e.g. video streaming traffic). However, they all adopt the semantic of TCP, which transports a single flow of data. I am interested in extending the problem for several streams (have you heard of QUIC?) that need to be scheduled on multiple paths. Instead of a single optimisation problem, you now end up with several concurrent streams, where each stream wants to complete as soon as possible! Writing a simulator</h3> The goal of my simulator is to quickly obtain an intuition on the behaviour of scheduling algorithms: it provides a graphical and animated visualisation of what's going on over time. The simulator also allows for more in-depth exploration, for instance comparing the completion times of streams for different scheduling algorithms. Below is a screenshot of the current simulator: it is not very pretty because that's not its goal! The streams are represented by vertical bars whose size equals the amount of data remaining to be transmitted, and the darker parts represent in-flight data. Below are two paths, each modelled as a packet queue with a constant-rate service (link capacity) and a fixed propagation time. </a> I am writing this simulator in Python thanks to salabim</a>: this is a really well designed, easy-to-use and well-documented simulation framework. I had an initial simulation prototype working in less than one day, and it took only an additional day to add graphical visualisation. One of the reasons it's so easy to use is thanks to Python: I didn't want to spend days implementing complex algorithms in NS-3, even though it would be much more realistic. At the same time, salabim is reasonably fast once you disable logging and visualisation. After working with salabim some more, I did find some limitations: the programming style around salabim is fine for small simulations, but quickly becomes a mess for larger projects. All examples use lots of global variables</a>, which encourages you to write all your code in one file (after all, this is how salabim itself is developed</a> with its 15k lines in a single file...) Introducing daily logs 2019-02-12T00:00:00+00:00 As you may have seen, I am not very good at writing regular articles here! I often get ideas for an article; sometimes I am motivated enough to actually start writing it; but then, most of the time, I never finish the article. With my third and (hopefully) last year of PhD going full steam, and still lots of involvement in community networks, I decided to change my writing approach and start publishing a daily log of what I do. What should you expect?</h3> Content-wise, I will mostly talk about networking, of course! More precisely, I will cover the following activities: my research activities: what I'm currently working on, interesting discussions with colleagues, conferences I attend, etc;</li> my teaching activities, also mostly related to networking;</li> my non-profit activities in Grenode</a>, Rézine</a>, Fédération FDN</a> and other organisations related to community networks.</li> </ul> I may also cover other activities that are not directly related to networking, for instance Openstreetmap</a>, the GCC compile farm</a>, contribution to various free software, and so on. Feedback</h3> Since this is a new exercise for me, I welcome all kind of feedback! You can reach me on Mastodon</a> or by email (root at <the current domain name minus blog></code>). Etat des points d'échange Internet en France 2018-02-11T00:00:00+00:00 Qu'est-ce qu'un point d'échange Internet ?</h3> Un point d'échange Internet</a>, ou IXP (Internet eXchange Point), c'est un endroit où plusieurs opérateurs réseau s'interconnectent pour échanger du trafic. De façon simplifiée, il faut voir ça comme un gros switch Ethernet sur lequel chaque opérateur réseau va se brancher, à l'aide d'un câble RJ45 ou une fibre optique. Oui oui, on parle bien du même genre de switch Ethernet que vous avez sûrement chez vous pour brancher vos ordinateurs, juste un peu plus rapide et fiable (et donc plus cher). Dans la réalité, la plupart des IXP ont une architecture plus complexe, avec plusieurs switches dans différentes baies d'un datacenter, et des points de présence ("PoP" pour "Point of Presence") dans plusieurs datacenters, reliés entre eux avec des fibres optiques. Mais le principe reste le même. Analyse des points d'échange Internet</h3> Récemment, le centre de recherche CAIDA</a> a publié un jeu de données sur les IXP</a>. Du coup je me suis dit que j'allais regarder ce qu'il y a dedans ! C'est intéressant d'avoir une vision globale du paysage des IXPs, parce qu'ils dessinent une grande partie de l'architecture physique d'Internet (qui est, je le rappelle, justement une interconnexion de réseaux). Regardons d'abord quelles informations sont disponibles : $ tail -n +2 ixs_201712.jsonl | jq 'select(.name == "France-IX")'</code></pre> ce qui donne : { "name": "France-IX", "city": "Paris", "country": "FR", "sources": [ "pdb", "wiki", "pch", "looking" ], "alternatenames": [ "Mix Internet Exchange and Transit", "FNIX6", "France Internet Exchange " ], "geo_id": 2988507, "region": "Paris", "pch_id": 74, "url": [ "http://www.mixt.net/", "http://www.fnix6.net/" ], "pdb_id": 68, "pdb_org_id": 147, "alternativenames": [ "French National Internet Exchange IPv6" ], "ix_id": 377, "org_id": 23 }</code></pre> On peut déjà remarquer plusieurs choses : les données viennent de différentes sources, comme indiqué ici</a> : PeeringDB</a>, la page Wikipédia sur les points d'échanges</a>, PCH</a>, et bgplookingglass.com</a>. </li> c'est un peu le bazar... En recoupant les différentes sources, CAIDA a associé des points d'échange qui n'ont rien à voir les uns avec les autres (FranceIX, MIXT, FNIX6) ! Là encore, la méthode utilisée est décrite sur la page du jeu de données</a>. </li> </ol> Ensuite, on voit qu'on peut facilement filtrer par pays : $ tail -n +2 ixs_201712.jsonl | jq 'select(.country == "FR") | .name' | wc -l 42</code></pre> Il semble donc y avoir 42 IXPs en France (modulo les doublons et erreurs), ça ne s'invente pas :) Les points d'échange en France</h3> Après nettoyage manuel des doublons et des vieux IXPs qui ont disparus, il reste environ 21 points d'échange en France, ce qui reste un nombre conséquent ! Auvernix</a> (PeeringDB</a>)</li> Breizhix</a></li> DE-CIX Marseille</a> (PeeringDB</a>)</li> Equinix Paris</a> (PeeringDB</a>)</li> EuroGIX</a> (PeeringDB</a>)</li> FNIX6</a> (PeeringDB</a>) : n'a plus l'air très actif, mais le site web marche toujours et il y a quelques membres connectés d'après PeeringDB</li> FranceIX</a> (PeeringDB</a>)</li> FranceIX Marseille</a> (PeeringDB</a>)</li> FR-IX</a> (PeeringDB</a>)</li> GrenoblIX</a> (PeeringDB</a>)</li> LyonIX</a> (PeeringDB</a>)</li> Guyanix</a> (PeeringDB</a>)</li> Hopus</a> (PeeringDB</a>) : pas vraiment un IXP, mais présent dans les données CAIDA et PeeringDB</li> Lillix</a> (PeeringDB</a>)</li> NetIX</a> (PeeringDB</a>)</li> NormandIX</a> : encore en projet ?</li> OuestIX</a> (PeeringDB</a>)</li> PARIX</a> : n'a plus l'air très actif, le site web est down, mais il y encore des membres connectés d'après PeeringDB</li> REUNIX</a> (PeeringDB</a>)</li> SFINX</a> (PeeringDB</a>)</li> TouIX</a> (PeeringDB</a>)</li> </ul> Contrairement à ce qu'on pourrait penser, il n'y a que 4 points d'échanges actifs à Paris : FranceIX, FR-IX, Equinix et SFINX (ainsi que Hopus, qui n'est pas vraiment un IXP). Tous les autres (environ 13 actifs) sont donc soit en région, soit en outre-mer. C'est étonnant quand on sait que l'Internet français est ultra-centralisé sur Paris (heureusement, ça s'améliore depuis quelques années, grâce notamment à Rézopole</a>). D'ailleurs, il y a historiquement eu beaucoup de points d'échange à Paris, mais la plupart sont morts ou ont été absorbés. On peut analyser cette rareté relative des points d'échanges à Paris, ainsi que leur prolifération en région et en outre-mer, de plusieurs manières : Les IXP permettent de développer le territoire local</h4> Les points d'échange sont importants pour développer le réseau sur le territoire local, puisqu'ils permettent aux opérateurs locaux de s'échanger du trafic directement, sans passer par les gros noeuds d'interconnexion comme Paris, Londres ou Amsterdam. Ça permet de réduire la latence et le coût, et de moins dépendre d'infrastructures qui deviennent critiques de par leur concentration (par exemple TH2</a> à Paris concentre une grosse partie des interconnexions de l'Internet français...). En somme, décentraliser et relocaliser le réseau, ce qui a des vertus non seulement techniques et économiques, mais également humaines : cela permet aussi de relocaliser les compétences techniques. C'est d'autant plus important en outre-mer ! Imaginons un abonné à La Réunion qui veut accéder à un serveur hébergé également à La Réunion. Sans point d'échange local, sa requête passera par une fibre sous-marine, parcourera probablement quelques centaines ou milliers de kilomètres, et reviendra par le même chemin... Du coup, il paraît logique que de plus en plus de régions développent des points d'échanges Internet en local. Par exemple, Rézopole</a> est financé en partie par la Région Rhône-Alpes pour s'occuper de LyonIX</a> et GrenoblIX</a>. Deux IXP sur un même territoire se font concurrence</h4> Une autre explication, c'est qu'il y a peu de place pour plusieurs points d'échanges sur un même territoire. En effet : pour un opérateur, se connecter à un point d'échange représente un coût majoritairement fixe, qui ne dépend que très peu de la quantité de trafic échangée (contrairement à du transit).</li> </ol> Il faut payer le cablage dans le datacenter, puis le port sur le switch du point d'échange : ce dernier coût est souvent lié à la capacité du port (1 Gbit/s, 10 Gbit/s, etc) et non à son utilisation réelle. Du coup, si un opérateur a le choix entre 5 petits points d'échanges qui permettront au total d'échanger 400 Mbit/s, et un seul point d'échange plus gros sur lequel il pourra écouler ses 400 Mbit/s, il aura tendance à privilégier le plus gros. Bien sûr, il y a d'autres critères de choix (redondance, présence dans plusieurs datacenters, tarifs, qualité du service) qui font que quelques points d'échange peuvent cohabiter sur le même territoire, mais ça limite quand même fortement le potentiel d'avoir des dizaines d'IXP au même endroit. l'effet de réseau joue : comme pour beaucoup de systèmes en réseau, plus un point d'échange possède de membres, plus il devient intéressant de s'y connecter. En effet, plus de membres présents signifie d'avantage de trafic échangé potentiel, pour le même coût fixe.</li> </ol> Cet effet a naturellement tendance à faire grossir les gros IXP et à faire disparaître les petits, et finit généralement par converger vers un unique IXP sur un territoire donné (sauf à Paris, où la demande est suffisamment forte et les datacenters suffisamment nombreux pour permettre à quelques IXP de co-exister ; on peut également voir des IXP avec des politiques très différentes co-exister, par exemple un IXP académique et un IXP commercial) Notons qu'il est quand même possible d'aller à contre-courant de cet effet de réseau. Par exemple, le point d'échange SIX à Seattle</a> a un modèle financier particulier : les opérateurs payent uniquement des frais d'accès au service, et ensuite ils peuvent échanger du trafic sur le point d'échange sans frais récurrents ! Le MINAP à Milan</a> a un modèle similaire, à une plus petite échelle, où même les frais d'accès sont offerts (mais pas les frais de raccordement). Plus généralement, pas mal de points d'échange (notamment les petits) sont sponsorisés par des acteurs du marché télécom local, qui se rendent bien compte des intérêts techniques et politiques de l'interconnexion locale : faible latence, contrôle de l'infrastructure, indépendance. En outre, les membres du point d'échange représentent des clients potentiels, à qui les sponsors du point d'échange pourront ensuite vendre de l'hébergement ou du transit ! La qualité de service d'un IXP doit être irréprochable</h4> Lorsqu'un point d'échange commence à grossir, il se pose forcément la question de la qualité du service. Tant que le point d'échange connecte le FAI associatif de la ville d'à côté et les 2 petites boîtes du coin, les coupures n'ont pas un impact énorme. Mais lorsque des centaines de membres sont connectés, certains de grosse taille, la moindre panne peut impacter des millions d'utilisateurs finals. Par ailleurs, pour les opérateurs, cela représente du travail de maintenance et de suivi, qui peut s'avérer plus lourd et coûteux que le bénéfice d'être connecté au point d'échange. Les opérateurs ont donc naturellement tendance à privilégier les points d'échange bien gérés et fiables. En réponse, les points d'échanges qui veulent subsister et grossir se donnent les moyens d'assurer un service fiable : astreinte 24/24, architecture technique redondée, matériel de pointe, etc. Soyons clair : gérer un point d'échange de taille raisonnable n'est pas facile, puisque cela demande à la fois une forte expertise technique (matériel spécialisé, architecture distribuée sur plusieurs sites) mais il y a aussi une forte composante relationnelle : la structure opérant le point d'échange doit interagir avec des centaines de structures hétérogènes, qui souhaitent toutes avoir un service fonctionnel sans que ça ne leur demande trop de temps de gestion et d'entretien. On assiste donc à la fois à un regroupement des compétences, via des structures comme Rézopole</a> pour éviter de tout réinventer de zéro à chaque IXP, mais aussi à un fort partage de connaissance et d'expérience à plus large échelle, avec le RIPE</a> et EuroIX</a>. Conclusion</h3> L'ecosystème des points d'échange n'est pas un sujet nouveau, mais il reste fascinant parce qu'il entrelace des problématiques techniques et des relations entre structures parfois très différentes. Il illustre bien le modèle distribué et pair-à-pair qui a fait d'Internet un succès. On peut par ailleurs constater que certains points d'échange sont gérés comme un bien commun ! Si le sujet vous intéresse, le RIPE NCC maintient un blog collaboratif</a> très actif sur des sujets liés à Internet en Europe, notamment les IXP</a> et le peering</a>. Toujours sur RIPE labs, Uta Meier-Hahn écrit régulièrement des articles passionnants</a> sur les enjeux des interconnexions entre opérateurs. OVH's OverTheBox: Internet access link aggregation using Multipath TCP 2015-09-24T00:00:00+00:00 OVH announced today its OverTheBox</a> project, which is basically a link-aggregation solution for Internet access links. Analysis of the technology</h3> Foreword on link aggregation</h4> First of all, aggregating Internet access links has nothing to do with classical link aggregation</a> (also called bonding or trunking). This is a much harder problem, because the access links typically have very diverse characteristics, in terms of latency, capacity, and packet loss. Think of aggregating a DSL line, a FTTH line and a satellite connection. If you simply send packets in a round-robin fashion, you will basically get the worst out of each link: packets will be heavily reordered, causing TCP to fall apart. The latency of a flow will basically be the latency of the worst link. Additionally, packet loss on any of the links will heavily impact the whole flow. Technology used in OverTheBox</h4> For OverTheBox, the main technology used by OVH is Multipath TCP</a>, often abbreviated as MPTCP. Multipath TCP basically allows to split a TCP flow across multiple paths, providing redundancy and increased throughput. It does so in a clever way: each subflow runs TCP independently, providing congestion control and packet loss recovery independently for each path. A scheduler decides on which path to send data, based first on the RTT of each path (lower RTT is preferred) and moving to the next path when the congestion window is filled. While Multipath TCP was not initially designed for link aggregation, it implements all necessary ingredients to do this efficiently. However, it only works for TCP traffic, and requires that both ends of a TCP connection know how to speak Multipath TCP. This is actually by design: end hosts are in the best position to discover paths and their associated characteristics (the typical use-case being a smartphone with both 4G and Wi-Fi</a>). OVH used the Linux implementation of Multipath TCP, and based its distribution on OpenWRT</a>, using an existing patch. Since Multipath TCP is not yet widely deployed in end-hosts, a link-aggregation solution based on Multipath TCP must be transparent for the devices behind the aggregation point. To do this, OVH used a classical solution based on a VPN. The idea is to run a VPN protocol able to tunnel data over TCP, such as OpenVPN. This way, provided both the VPN client and servers and MPTCP-compatible, the VPN will automatically use all available paths, with associated load-balancing and failover benefits. OVH apparently decided</a> to use vtun</a>, which I had never heard of before. That being said, there are also references to OpenVPN</a> in the code, so I am not sure which one they use. In addition to that, OVH seems to use a transparent SOCKS proxy, shadowsocks</a>. The goal is to avoid TCP over TCP encapsulation, which is notoriously a bad idea</a>. Thanks to the SOCKS proxy, TCP connections from local clients are terminated locally, and new TCP connections are established from the other end of the tunnel towards the destination. This way, any packet loss on the path towards the destination does not trigger retransmissions inside the VPN. For UDP traffic, I am not sure whether it also goes through the SOCKS proxy (this is possible with SOCKS5, but would be somewhat useless in this case) or travels directly on the VPN. Finally, as a last note, OVH decided to shoot IPv6 in the head by completely ignoring AAAA DNS requests</a> in their local DNS resolver. This is a ugly hack, and sounds like a quick and dirty fix for an issue discovered just before the initial release. My guess is that either shadowsocks does not support IPv6, or the IPv6 connectivity provided by some of the access links interferes with the operation of the OverTheBox box. I do hope that this is a temporary fix, because crippling IPv6 like this will certainly not help its deployment. By the way, Multipath TCP of course fully supports IPv6. By the way, this analysis is based on a rather quick look at the source code, and my own experience. If you think I made a mistake, feel free to send me an email (contact at the domain name of this blog). Impact of OverThebox</h3> As such, this project from OVH merely assembles existing components. It introduces nothing new, except maybe a nice web interface (which is actually non-negligible in terms of user impact). And indeed, technically speaking, people have already been doing the exact same thing for a while: Multipath TCP for link aggregation, a VPN such as OpenVPN for encapsulation, and a transparent SOCKS proxy to terminate client TCP connections before entering the tunnel. See for instance this mail on the mptcp-dev mailing list</a>. But this is, to my knowledge, the first open off-the-shelf solution providing an easy-to-use interface. What's more, OVH released the code</a>, and the solution should work just fine with your own VPN server: it does not force you to use OVH services, which is extremely nice. This is in huge contrast with existing proprietary solutions for the same problem, such as the products sold by Peplink</a>. Their business model is to sell you the hardware and the service, with associated licensing fees. Since the protocol is proprietary, you are forced to use the Peplink VPN servers (even though they seem to offer to deploy VPN servers in the cloud, that you can manage through their provided interface). OverTheBox is likely to have an effect on this kind of proprietary businesses. On the other hand, providers like Peplink can (and probably should) make a difference by providing custom support for companies, something that OVH probably won't do. Finally, let us note that there are other solutions to the original problem, such as MLVPN</a> (which is not based on Multipath TCP). But OVH clearly has enough weight to make a huge impact with its nice, integrated solution. Utiliser IPv6 chez SFR, sans la Neufbox 2014-08-22T00:00:00+00:00 Cet article est en français, puisqu'il est susceptible d'intéresser principalement des lecteurs français. En France, SFR fournit de l'IPv6 sur ses accès ADSL et fibre : c'est très bien ! En revanche, il ne s'agit pas d'une connectivité IPv6 native (probablement parce que le réseau d'accès ne fonctionne qu'en IPv4 pour le moment). La connectivité IPv6 est fournie par un tunnel monté au-dessus d'IPv4. Lorsqu'on utilise la box de SFR, c'est transparent : la box monte le tunnel elle-même, et on ne voit rien (à part une MTU un peu faible). En revanche, si on remplace la box par un routeur à soi (par exemple, sous OpenWRT</a>), il faut monter le tunnel soi-même si on veut profiter de l'IPv6. Ce n'est pas si évident à mettre en place (techniquement, c'est de l'IPv6 sur L2TP sur UDP sur IPv4, avec du PPP et du DHCPv6 pour faire bonne mesure). Le but de cet article est de détailler la mise en place du tunnel IPv6 sous OpenWRT, sachant que la configuration est adaptable pour d'autre systèmes GNU/Linux ou BSD. Configuration IPv4</h3> Obtenir une adresse IPv4 avec un routeur branché sur l'ONT SFR a déjà été beaucoup</a> documenté</a> : il suffit de faire du DHCP avec un vendor-id spécifique. Sous OpenWRT, ça se traduit par : # /etc/config/network config interface 'wan' option ifname 'eth0' # à adapter option proto 'dhcp' option vendorid "neufbox-BypassedNeufBox-DirectConnectionToFTTH-toto@nowhere.xxx"</code></pre> Le vendorid doit en fait simplement commencer par "neufbox", mais indiquer que ce n'est pas une Neufbox semble recommandé, des fois que le support technique passe par là (même si en pratique, le support de SFR est plutôt du genre « C'est bizarre, votre connexion Internet n'a pas l'air de fonctionner. » « Si si, je vous assure, ça marche très bien. » « Ah bon. »). Analyse du tunnel</h3> La première étape est de déterminer l'adresse du LNS (L2TP Network Server), qui est le routeur avec lequel le tunnel L2TP est monté. Pour monter le tunnel, il y a ensuite deux niveaux d'authentification : une authentification pour monter le tunnel L2TP lui-même. C'est simplement un mot de passe codé en dur, le même pour toutes les Neufbox : 6pe</li> une authentification PPP, dont le couple login/mot de passe est spécifique à chaque client SFR.</li> </ul> Il faut donc connaître le login et le mot de passe PPP. Fort heureusement, la Neufbox envoie ceux-ci en clair lorsqu'elle établit le tunnel. Pour récupérer toutes ces informations, il suffit donc d'écouter le trafic de la Neufbox juste après son démarrage. Écouter le trafic de la Neufbox</h4> Pour une connexion fibre, c'est très simple, il suffit de se mettre entre la Neufbox et l'ONT. Voir cet article</a> pour plus de détails. Le plus simple est probablement d'utiliser une machine sous Linux avec deux interfaces réseau (par exemple, un laptop avec une carte Ethernet en USB). Une interface est branchée sur le port WAN de la Neufbox, l'autre est branchée sur l'ONT. Ensuite, on bridge les deux interfaces : $ sudo brctl addbr br0 $ sudo brctl addif br0 eth0 $ sudo brctl addif br0 eth1 $ sudo ip link set eth0 up $ sudo ip link set eth1 up $ sudo ip link set br0 up $ sudo sysctl -w net.ipv4.ip_forward=1</code></pre> Il faut aussi s'assurer que le firewall autorise le forwarding de paquet. En cas de doute : $ sudo iptables -P FORWARD ACCEPT $ sudo iptables -F FORWARD</code></pre> Il ne reste plus qu'à regarder le trafic qui passe : $ sudo tcpdump -n -v -i eth0</code></pre>Adresse du LNS</h4> Sur ma connexion fibre, l'adresse du LNS avec lequel la Neufbox établit le tunnel est 109.6.3.95</code>. Il se peut que le serveur soit différent selon la région, ou d'autres critères. Par exemple, dans la capture faite par Marin</a>, le LNS est 109.6.1.72</code>. Un autre utilisateur indique</a> que chez lui, le LNS est 109.6.4.36</code>. Le plus simple est donc d'écouter le trafic et d'utiliser le même LNS que votre Neufbox. Login et mot de passe PPP</h4> Le login PPP est manifestement de la forme dhcp/XX.XX.XX.XX@YYYYYYYYYYYY, où XX.XX.XX.XX</code> est l'IPv4 publique de l'accès Internet, et YYYYYYYYYYYY</code> est l'adresse MAC du port WAN de la Neufbox, sans les :</code>. Pour le mot de passe PPP, il ne semble pas y avoir de logique particulière. Il s'agit visiblement d'une chaîne de 16 caractères dans l'alphabet [A-Z0-9]</code> (alphanumérique avec uniquement des lettres en majuscule). Configuration sous OpenWRT Barrier Breaker</h3> Tout est dans la documentation d'OpenWRT</a>. En adaptant pour SFR, ça donne donc : # /etc/config/network config interface 6pe option proto l2tp option server '109.6.3.95' # à adapter option username 'dhcp/XX.XX.XX.XX@YYYYYYYYYYYY' option password 'ZZZZZZZZZZZZZZZZ' option keepalive '6' option ipv6 '1' config interface 'wan6' option ifname '@6pe' option proto 'dhcpv6'</code></pre> Ainsi que : # /etc/xl2tpd/xl2tp-secrets * * 6pe</code></pre> Et hop, ça juste marche (autoconfiguration sur le LAN, règles de firewall, etc). Magique, OpenWRT, non ? :) Configuration pour d'autres OS (GNU/Linux, BSD)</h3> La méthode pédestre, en configurant xl2tpd puis pppd, est également documentée sur le wiki OpenWRT</a>. Quelqu'un a également essayé avec un EdgeRouter Lite</a>, et a fini par obtenir une config</a> pas super propre, mais qui marche. Notons que le firmware des Neufbox est disponible : http://neufbox.alwaysdata.net/</a>. Notamment, il est possible de récupérer la configuration de xl2tpd</code> et pppd</code> pour être sûr d'avoir la même. Performance</h3> Le tunnel est monté sur un routeur Netgear WNDR3800</a>, sous OpenWRT Barrier Breaker rc3. La connexion est une fibre SFR 1G/200M. Les tests sont fait depuis un laptop branché en filaire, vers ipv6.intuxication.testdebit.info</code>, qui est à environ 10 ms, avec les commandes suivantes : # Download wget -O /dev/null http://ipv6.intuxication.testdebit.info/fichiers/1000Mo.dat # Upload curl -o /dev/null -F 'filecontent=@1000Mo.dat' http://ipv6.intuxication.testdebit.info</code></pre> Chaque commande est lancé plusieurs fois en parallèle si nécessaire (pour remplir le tuyau), et le débit instantané est relevé sur le routeur. J'obtiens les débits IP maximaux suivants : 80 Mbits en upload</li> 105 Mbits en download</li> </ul> En upload, un top</code> sur le routeur montre que le CPU passe 100% du temps à traiter des interruptions logicielles (sirq). En download, l'utilisation CPU est plutôt de l'ordre de 90%. Pour comparer, en IPv4 dans les mêmes conditions, j'obtiens 180 Mbps avec 85% d'utilisation CPU sur le routeur, le résultat étant identique en upload et en download. Visiblement, encapsuler des paquets L2TP est donc plus coûteux que de les décapsuler. Conclusion</h3> Cet article décrit comment utiliser la connectivité IPv6 fournie par SFR lorsqu'on remplace la Neufbox par un routeur à soi. On peut avoir envie d'utiliser d'autres services (téléphone, télévision, etc), mais plein de gens ont documenté comment faire : voir les liens ci-dessous. Liens</h4> Configuration IPv6 over L2TP pour OpenWRT</a></li> OpenWrt native IPv6-stack</a> (pour OpenWRT Barrier Breaker 14.07)</li> Captures réseau du démarrage et fonctionnement de la Neufbox 6</a> fort intéressant</li> Bypasser sa neufbox</a> article de référence, même si pas forcément à jour</li> Bypasser la Neufbox Fibre avec un Routeur Linux</a></li> Bypass neufbox 6 avec NetBSD</a></li> </ul> Back to business 2014-07-23T00:00:00+00:00 So, writing a blog again. I had one, years ago. Hosted at home, like this one (though it was behind DSL at the time, FTTH wasn't as widespread as it is now). It was about free software, programming languages (especially functional), and maybe already some bits of networks. I remember writing a long post after discovering network neutrality for the first time, thanks to a talk by La Quadrature du Net</a>. This new blog will be mostly about networks, and how you can use them for fun and for saving the world (yes, this is overly ambitious). Overlay networks, wireless mesh networks, routing protocols, free software, free hardware, Do-It-Yourself ISPs, community-owned networks... And probably other stuff I forgot. As a general rule, ideas and principles behind networks will be discussed, and not only "how to do this particular thing with that particular software". I strongly believe that networking is not hard, provided you understand what you are doing, which is often the most difficult part. Once you know what you are doing, it is relatively easy to use the available networking tools, or to create new ones. That being said, I will definitely provide configuration examples when they are non-obvious and/or use some obscure functionalities. By the way, this blog will not show photos of big Cisco routers</a>, or explain how to do X with insert your favourite prioprietary router OS here. Hopefully, this blog will stay up longer than the previous one. Enjoy reading, and happy hacking!

Debian installation</h2>

Long-term plans</h2>
In the long term, we would like to work on vector instructions and virtualisation instructions support, but this is still a long way to go given the lack of hardware supporting these instructions.</p>

Debugging eBPF-enabled programs in Docker

Teaching routing and network security — Daily log

Multipath scheduling simulation — Daily log

Introducing daily logs

Feedback</h3>
Since this is a new exercise for me, I welcome all kind of feedback! You can reach me on Mastodon</a> or by email (`root at <the current domain name minus blog></code>).</p>`

Etat des points d'échange Internet en France

OVH's OverTheBox: Internet access link aggregation using Multipath TCP

Utiliser IPv6 chez SFR, sans la Neufbox

Back to business

Bits of networks

Patching OpenSSH in Debian

BananaPi BPI-F3: bootstraping upstream Debian

BananaPi BPI-F3: hardware and software overview

VisionFive 2: the hardware

The upstream RISC-V experience: running RISC-V hardware with upstream distros

Moving from Pelican to Zola

Introduction to DevOps for experimental research

Benchmarking RISC-V: VisionFive 2 vs the world

Bits of networks

Patching OpenSSH in Debian

BananaPi BPI-F3: bootstraping upstream Debian

Debian installation</h2>

BananaPi BPI-F3: hardware and software overview

VisionFive 2: the hardware

The VisionFive 2 hardware: TL;DR</h2> The board comes with a cute transparent case</strong></li> The hardware is relatively fast and stable</strong></li>

The upstream RISC-V experience: running RISC-V hardware with upstream distros

Moving from Pelican to Zola

Introduction to DevOps for experimental research

Benchmarking RISC-V: VisionFive 2 vs the world

Debugging eBPF-enabled programs in Docker

Teaching routing and network security — Daily log

Multipath scheduling simulation — Daily log

Multipath scheduling</h3> Today I mostly worked on my current research project, a simulator of multipath multistream scheduling algorithms</strong>.</p>

Introducing daily logs

What should you expect?</h3> Content-wise, I will mostly talk about networking, of course!</p> More precisely, I will cover the following activities:</p>

Etat des points d'échange Internet en France

Les points d'échange en France</h3> Après nettoyage manuel des doublons et des vieux IXPs qui ont disparus, il reste environ 21 points d'échange en France, ce qui reste un nombre conséquent !</p>

OVH's OverTheBox: Internet access link aggregation using Multipath TCP

Utiliser IPv6 chez SFR, sans la Neufbox

Analyse du tunnel</h3> La première étape est de déterminer l'adresse du LNS</strong> (L2TP Network Server), qui est le routeur avec lequel le tunnel L2TP est monté. Pour monter le tunnel, il y a ensuite deux niveaux d'authentification :</p>

Back to business