Fix Beszel Agent Root Disk Reporting: AWS EC2 & Docker

Hey guys, have you ever run into one of those head-scratching moments where your monitoring agent is almost perfect, but then it completely misses something super crucial? Well, if you're using Beszel Agent 0.17.0 on an AWS EC2 instance with Docker, and your root disk usage isn't showing up, then you've landed in the right place! We're talking about that sneaky situation where your main / filesystem, often /dev/nvme1n1p1 on EC2, just refuses to report its precious metrics, even though other disks are happily chatting away. This can be super frustrating, especially when you're relying on Beszel to give you the full picture of your server's health. We've all been there, scratching our heads, wondering why our primary storage is playing hide-and-seek with our monitoring tools. It’s like having a security camera that only shows you the backyard but misses the front door – you're getting some data, but not the critical data. This issue, specifically concerning the Beszel Agent not properly detecting or reporting the root disk usage, is a common snag for folks deploying their agents within Docker containers on environments like AWS EC2, where disk partitioning and device naming conventions (like nvme devices) can add an extra layer of complexity. Our goal here is to dive deep into why this happens, explore the nuances of Docker containerization impacting disk visibility, and most importantly, equip you with the steps and configurations needed to get your root disk reporting like a champ. So, grab your favorite beverage, and let's unravel this mystery together to ensure your Beszel Agent delivers the comprehensive monitoring data you absolutely need for stable and performant systems. We'll make sure no disk goes unmonitored!

Hey Guys, Is Your Beszel Agent Playing Hide-and-Seek with Your Root Disk?

Alright, let's get right into it, guys. The primary issue we're tackling here is the Beszel Agent not properly reporting root disk usage, specifically when deployed within a Docker container on an AWS EC2 instance. Imagine this scenario: you've got your shiny new Beszel Agent version 0.17.0 humming along, meticulously monitoring your system. You've even got a secondary EBS volume, perhaps mounted at /var/lib/docker (like /dev/nvme0n1), and Beszel is happily reporting its usage. But then you glance at your main / partition, often dev/nvme1n1p1 on EC2, and boom – nothing. It's like your most critical disk has gone invisible! This is exactly the kind of situation that prompted this deep dive. The logs for the agent, while showing network activity and data directory paths, remain silent on the root disk's metrics: 2025/12/03 04:57:18 INFO Data directory path=/var/lib/beszel-agent, 2025/12/03 04:57:18 INFO Detected network interface name=ens5 sent=95919684 recv=89264996, 2025/12/03 04:57:18 INFO WebSocket connected host=127.0.0.1:8090. Notice, no disk usage information for the root. This isn't just an annoying glitch; it's a critical monitoring gap. Your root disk hosts your operating system, essential binaries, and often crucial application components. Without accurate and continuous monitoring of its usage, you're essentially flying blind. You won't know if your OS logs are filling up the disk, if temporary files are accumulating dangerously, or if an application is silently consuming all available space. This can lead to unexpected outages, performance degradation, and a frantic scramble to diagnose the issue when it finally hits. The user in our case had a very specific setup: an AWS EC2 server with two EBS volumes, /dev/nvme1n1p1 mounted as / and /dev/nvme0n1 mounted as /var/lib/docker. The latter was explicitly mounted into the Docker container, leading to the obvious question: do we need to explicitly mount the root disk into Docker Compose as well? This problem highlights a fundamental challenge when containerizing monitoring agents: how do you give a containerized application sufficient visibility into the host system's resources, especially when those resources aren't directly part of the container's own filesystem? The expected behavior, naturally, is that the root disk /dev/nvme1n1p1 => / should absolutely report its usage stats, just like any other monitored resource. The lack of this vital metric compromises the entire purpose of having a monitoring agent in the first place. Understanding this problem is the first step toward a robust and reliable monitoring setup. We're going to ensure your Beszel Agent gets full sight of all your important disks!

Diving Deeper: Understanding the Beszel Agent's Disk Detection Quirks

Now that we've established the problem – our Beszel Agent stubbornly ignoring the root disk – let's pull back the curtain a bit and understand why this might be happening. When you run an application like the Beszel Agent inside a Docker container, it operates within its own isolated environment. This isolation is fantastic for portability and dependency management, but it also creates a challenge for processes that need to inspect the host system's resources, like disk usage. Typically, a monitoring agent needs to access specific host-level directories to gather disk statistics. On Linux, this often involves reading from /proc/mounts, /etc/fstab, or inspecting files within the /sys filesystem. These paths provide crucial information about mounted filesystems, their types, and their current usage. However, by default, a Docker container doesn't have direct access to these host-level paths. Its view of the filesystem is confined to what's inside its own container image and any volumes you explicitly mount. In our specific case, the secondary disk /dev/nvme0n1, mounted on the host at /var/lib/docker, was explicitly mapped into the Beszel Agent container via the Docker Compose configuration: - /var/lib/docker:/extra-filesystems/nvme0n1__portainer1-docker:ro. This tells Docker to take the host path /var/lib/docker and make it available inside the container at /extra-filesystems/nvme0n1__portainer1-docker in read-only mode. Because this mount was explicitly provided, the Beszel Agent could 'see' and report on it. But what about the root disk, /dev/nvme1n1p1 => /? It wasn't given the same VIP treatment. The agent, from within its container, simply doesn't have a clear path to observe the host's root filesystem's metrics. While the cap_add: SYS_ADMIN capability and network_mode: host are excellent for giving the container elevated network privileges and some system administration capabilities, they don't automatically expose host filesystems or allow the container to arbitrarily snoop on host /proc or /sys directories for disk information. SYS_ADMIN grants a broad set of administrative powers, which can sometimes include operations relevant to filesystem management, but it doesn't bypass the fundamental filesystem isolation provided by Docker's namespaces unless you specifically bind mount the relevant host directories. The network_mode: host primarily impacts how the container's networking stack is configured, allowing it to share the host's network namespace, but again, it doesn't directly address filesystem visibility. The core of the problem lies in the fact that Beszel Agent, when trying to detect disks, is looking at the mount points and device information it can see from within its own container. If the host's root filesystem isn't presented to it as an accessible path, it simply won't know it exists for the purpose of reporting usage. This is a common pitfall with containerized monitoring, and understanding this isolation boundary is key to bridging the gap and getting all your essential metrics reported accurately. We're going to fix this by giving our Beszel Agent a proper window into the host's root filesystem.

The Docker Compose Conundrum: Is Your Root Disk Properly Exposed?

Alright, let's zero in on the heart of the matter for our root disk reporting problem: the Docker Compose configuration for the beszel-agent service. This YAML file is the blueprint for how your container runs, and every little detail matters, especially when it comes to host resource access. We saw that the secondary disk, /dev/nvme0n1 mounted at /var/lib/docker on the host, was successfully reporting its metrics. Why? Because it had this specific line in the volumes section of the beszel-agent service: - /var/lib/docker:/extra-filesystems/nvme0n1__portainer1-docker:ro. This line is crucial! It tells Docker to create a bind mount. It literally takes the directory /var/lib/docker from your host machine and makes it available inside the beszel-agent container at /extra-filesystems/nvme0n1__portainer1-docker. The :ro at the end means it's mounted as read-only, which is a smart security practice for monitoring agents. Now, if we look at the configuration without this line for the root disk, it becomes clear why it's missing. The root disk / (i.e., /dev/nvme1n1p1) doesn't have a corresponding entry! The Beszel Agent, running inside its isolated environment, simply doesn't have a path to observe the host's root filesystem. Think of it like this: the agent is in a house, and you've given it a map to the basement (the secondary disk), but you haven't given it a map to the living room (the root disk). It can only report what it can 'see' and access based on the maps you provide. For an agent to report on host-level disk usage, it generally needs access to the host's / (root) filesystem, or at least the critical parts of it that contain device and mount information, such as /proc and /sys. These directories contain ephemeral filesystem and kernel information that the agent uses to discover disks and their statistics. Without binding these host paths into the container, the agent sees only the container's own (often very small) root filesystem, or a limited view of proc and sys that reflects the container's environment, not the host's. So, the direct answer to the question, "Do I need to explicitly mount the root disk in Docker compose as well?", is a resounding yes, absolutely! To allow the Beszel Agent to see and report on the host's root disk, you need to explicitly bind mount the host's root filesystem into the container. This bridges the isolation gap, giving the agent the necessary visibility. It's a critical step in ensuring comprehensive monitoring and avoiding those frustrating blind spots. Without this explicit mount, the agent is inherently limited in what it can report about your host's storage, leaving you in the dark about your most critical disk's health. We need to give our Beszel Agent an 'all-access pass' to the host's filesystems, but in a secure, read-only manner, to ensure it can do its job properly. This adjustment in your Docker Compose is going to be the game-changer you're looking for.

Troubleshooting Steps for a Stubborn Root Disk

Alright, guys, let's get hands-on and troubleshoot this stubborn root disk reporting issue with the Beszel Agent. We've identified the root cause: the agent in the Docker container doesn't have sufficient visibility into the host's root filesystem. Now, let's walk through the practical steps to fix this and ensure your Beszel Agent reports all disk usage accurately. Trust me, these steps are tried and true for bridging that container-to-host visibility gap!

Solution 1: Explicitly Mount the Host's Root Filesystem

This is often the most direct and effective solution. You need to modify your docker-compose.yaml file to explicitly bind mount the host's root filesystem (/) into your beszel-agent container. This gives the agent the necessary access to discover and monitor the root disk.

Here's how you'll adjust the volumes section for your beszel-agent service:

services:
  beszel-agent:
    image: henrygd/beszel-agent:0.17.0
    container_name: beszel-agent
    hostname: ${HOSTNAME}
    cap_add:
      - SYS_ADMIN
    environment:
      - LOG_LEVEL=info
      - LISTEN=45876
      - KEY=${KEY}
      - TOKEN=${TOKEN}
      - HUB_URL=http://127.0.0.1:8090
    volumes:
      # Existing secondary disk mount
      - /var/lib/docker:/extra-filesystems/nvme0n1__portainer1-docker:ro
      # *** ADD THIS LINE FOR ROOT DISK MONITORING ***
      - /:/host:ro  # Mount the host's root filesystem into /host inside the container
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: always
    network_mode: host
    deploy:
      resources:
        limits:
          cpus: 0.5

By adding - /:/host:ro, you're telling Docker: "Take the root directory of my host machine (/) and make it available inside the beszel-agent container at a new path called /host. And, importantly, make it read-only for security reasons." The Beszel Agent, once restarted, should now be able to traverse /host and discover the underlying disks and their usage stats. This is often the magic bullet! While /host is a common convention, you could theoretically pick any path inside the container (e.g., /mnt/host_root), as long as it's consistent. Just remember, always use :ro for monitoring data to prevent accidental writes or modifications to your host system from within the container. After making this change, remember to restart your Docker Compose services using docker-compose up -d --force-recreate to apply the new volume mounts.

Solution 2: Double-Check Permissions and cap_add

While the primary issue is likely the missing bind mount, it's always good practice to ensure your container has the necessary capabilities. You've already included cap_add: - SYS_ADMIN in your configuration, which is generally a good move for monitoring agents that need deeper system insights. SYS_ADMIN grants a broad range of system administration privileges. Also, network_mode: host allows the container to share the host's network stack, which is vital for network metrics and direct communication with the Beszel Hub without port mapping. Confirming these are present is a quick sanity check, but they usually don't solve the filesystem visibility problem directly.

Solution 3: Inspecting Beszel Agent Logs (with verbose logging)

Your initial logs were quite lean. If the issue persists, consider temporarily increasing the LOG_LEVEL for the Beszel Agent. While info is good, a debug level might offer more granular details about what filesystems the agent is detecting and which ones it's ignoring, potentially giving you hints. You can change LOG_LEVEL=info to LOG_LEVEL=debug in your environment variables. Then, after restarting, examine the agent logs closely for any messages related to disk enumeration or errors accessing paths. This can sometimes reveal specific permission issues or unexpected behaviors.

Solution 4: Verify Host Mounts and Device Naming

Sometimes, the problem isn't with Docker or the agent, but with the host itself. Connect to your AWS EC2 instance directly via SSH and run commands like df -h and lsblk. Confirm that /dev/nvme1n1p1 is indeed mounted at / and that it's healthy and reporting usage correctly on the host. Also, ensure the device names (e.g., nvme1n1p1) are consistent with what you expect. While unlikely to be the primary cause of a missing report, it's a good diagnostic step to rule out host-level issues.

Solution 5: Understanding Beszel Agent's Internal Logic (Advanced)

While you usually don't need to dive into the agent's source code, it's worth understanding that monitoring agents have specific logic for what constitutes a