marschall.systems

Janus: Multi-Tenant Access Control for Observability-Data

Maximilian Marschall — Tue, 03 Feb 2026 18:01:01 +0200

If you’re running a shared observability platform, you’ve probably faced this question: How do I stop Team A from seeing Team B’s data?

Grafana is fantastic for visualization, but the datasources behind it (Prometheus, Loki, Tempo, and their derivatives) weren’t designed with fine-grained multi-tenancy in mind. You typically get one of two options:

Basic authentication: Everyone with access sees everything
Separate instances: Operational overhead multiplied by N teams

Neither scales well. Enterprise solutions exist, but they’re expensive and often locked to specific vendors. Building custom authorization middleware means maintaining bespoke code for each query language. Most organizations end up with some combination of “trust people not to query the wrong namespaces” and “hope for the best.”

We ran into this exact problem at work. Multiple teams sharing a single observability stack, and no good way to enforce boundaries without spinning up separate instances for everyone. So we built Janus.

Together with Erik Kaisler and Irfan Hadzijusufovic, we’ve been working on this for a while, and we’re now releasing it as open source under the AGPLv3 license. A big thank you to our employer, evoila, for giving us the time to work on this.

What Janus Does

Janus sits between Grafana and your datasources, intercepting every query. It extracts identity from OAuth2 tokens, maps that identity to a set of allowed label values, and automatically injects label filters into queries before they reach the backend.

Your users write queries like normal. They don’t need to know about tenant boundaries or remember to add namespace filters. Janus handles it transparently.

The biggest win for us: we can now use a single dashboard for all teams, and everyone only sees data relevant to them. Janus even filters the label values returned by the API, so dropdowns in Grafana are only populated with values you’re actually allowed to query. No more scrolling through namespaces you can’t access anyway.

# User writes:
rate(http_requests_total[5m])
# Janus rewrites to:
rate(http_requests_total{namespace=~"team-a|team-b"}[5m])

The same principle applies across all three query languages:

PromQL for metrics (Prometheus, Thanos, Mimir, VictoriaMetrics)
LogQL for logs (Loki)
TraceQL for traces (Tempo)

How It Works

Janus is a reverse proxy that does three things:

1. Identity Extraction

When a request arrives, Janus extracts the user’s identity from the OAuth2 access token. This can come from JWT claims, token introspection, or userinfo endpoints, whatever your identity provider supports.

2. Policy Resolution

Based on the extracted identity (user, groups, roles, custom claims), Janus determines which label values the user is permitted to query. Policies are defined declaratively:

thanos:
 tenant-header-constraints:
 order-service-team:
 header:
 - X-Scope-OrgID: tenant1
 my-service-team:
 header:
 - X-Scope-OrgID: tenant2
 regex-team:
 header:
 - X-Scope-OrgID: tenant-regex
 multi-group-user:
 header:
 - X-Scope-OrgID: tenant3
 admin:
 header:
 - X-Scope-OrgID: admin-tenant

 user-label-constraints:
 # Regex pattern: starts with "demo" (anchored)
 order-service-team:
 labels:
 - "*"
 namespace:
 - "~^demo.*$"

 # multi-group-user has access to BOTH demo AND observability
 multi-group-user:
 labels:
 - "*"
 namespace:
 - demo
 - observability

3. Query Rewriting

Before forwarding the request, Janus parses the query, injects the appropriate label matchers, and reconstructs it. This happens transparently, the datasource receives a valid, scoped query, the user receives results they’re authorized to see.

Why Label-Based Filtering?

Labels are the universal primitive in modern observability. Kubernetes workloads are labeled by namespace, team, environment. Logs carry metadata about their source. Traces propagate context through service boundaries.

By filtering at the label level, Janus works with your existing data model. No need to restructure your metrics or maintain separate Loki tenants. If you can express a boundary as a label selector, Janus can enforce it.

This also means enforcement happens at query time, not ingestion time. You can retroactively apply access controls without re-ingesting historical data.

Deployment

Janus is designed for Kubernetes-native deployment, though it runs anywhere you can run a container.

Without Janus

In a typical setup, Grafana connects directly to each datasource:

flowchart LR
G[Grafana] --> P[Prometheus/Mimir/Thanos]
G --> L[Loki]
G --> T[Tempo]

Every user with Grafana access can query any data in the backend-no tenant isolation.

With Janus

Janus sits between Grafana and your datasources, enforcing access control transparently:

flowchart LR
G[Grafana] --> J[Janus]
J --> P[Prometheus/Mimir/Thanos]
J --> L[Loki]
J --> T[Tempo]
J <--> IDP[Identity Provider]
IDP <--> G

Query Flow

Here’s what happens when a user runs a query:

sequenceDiagram
participant U as User
participant G as Grafana
participant J as Janus
participant DS as Datasource
U->>G: Execute query
G->>J: Forward query + OAuth2 token
J->>J: Validate token / get claims
J->>J: Resolve policy → allowed labels
J->>J: Rewrite query with label filters
J->>DS: Execute scoped query
DS-->>J: Filtered results
J-->>G: Return results
G-->>U: Display dashboard

Configure Grafana to use Janus as the datasource URL instead of pointing directly at your backends. Janus handles the rest.

Pre-built container images are available from the repository.

The Query Rewriting Challenge

The interesting part was supporting three different query languages. Each has different syntax, different semantics for label matching, and different edge cases.

PromQL label matchers are relatively straightforward, inject {label=~"value1|value2"} and merge with existing selectors. LogQL adds pipeline stages that need to be preserved. TraceQL has its own structural query syntax for spans and traces.

Janus uses dedicated parsers for each language rather than regex manipulation. This ensures queries remain valid after rewriting and handles edge cases like nested expressions, subqueries, and aggregations correctly.

To make sure we don’t break anything on updates, we’ve built extensive end-to-end tests that run against real Prometheus, Loki, and Tempo instances with example data. This way we catch regressions before they ship, not when someone’s dashboard stops working.

What It Doesn’t Do

Rate limiting: Use your existing ingress or API gateway for that
Data masking: Janus filters which data you can query, not which fields are visible within results
Write-path authorization: This is query-side only; ingestion controls are a separate concern

Getting Started

The repository includes a pre-built container image and documentation for the policy syntax.

Run Janus as a container (Docker, Podman, or Kubernetes)
Configure your policies to map users/groups to allowed label values
Point your Grafana datasources at Janus instead of directly at your backends

One requirement: Users need to be authenticated via OAuth2. Janus extracts identity from the access token, so anonymous access won’t work.

Check out the repo for example configs.

Why AGPLv3?

We went with AGPLv3 to make sure improvements stay available to everyone. If you modify Janus and offer it as a service, those modifications need to be shared. We’ve seen too many useful infrastructure tools get absorbed into proprietary platforms and fragment the community.

For most users, running Janus internally to secure their own observability stack, the license has no practical impact beyond ensuring the project stays open.

What’s Next

The initial release covers the core use case: OAuth2 identity, label-based policies, and PromQL/LogQL/TraceQL rewriting. Possible Roadmap features could be :

Additional identity sources (mTLS, API keys)
Configurable Audit logging
Caching layer for policy decisions

Contributions welcome, whether it’s bug reports, docs, or new features, the GitHub issues are open.

Janus is available now at GitHub. If tenant isolation has been sitting on your backlog, give it a try.

Questions, feedback, or war stories about observability access control? We’d love to hear them.

Why “Janus”? Every observability tool seems to be named after a god Prometheus, Loki, Thanos. We needed a gatekeeper, so we went with the Roman god of doorways and gates. Plus, he has two faces, which felt appropriate for a proxy that sees both sides of every request.

MinIO OSS: Another Rug Pull in Open Source?

Maximilian Marschall — Thu, 23 Oct 2025 17:13:01 +0200

MinIO, once a darling of the open-source object storage world, has followed a path we’ve seen before: slowly pulling the rug on its OSS community. It started subtly, features were stripped from the community edition, leaving users with a barebones experience. Now, MinIO has taken it a step further by ceasing distribution of pre-built binaries altogether. The project is now source-only, as confirmed in their GitHub README. This move significantly raises the barrier to entry for users who relied on the simplicity of ready-to-run binaries. One of MinIO’s biggest selling points was its sleek UI. With that gone from the OSS version and enterprise pricing remaining steep, it’s time to consider alternatives.

Enter Ceph

A strong contender is Ceph, especially when paired with Rook for Kubernetes-native deployments. Ceph supports:

Multi-node HA setups
Single-node deployments with replication or erasure coding
CephFS for true RWX support
RBD for block storage

Depending on your architecture, Ceph can scale impressively and offers a rich feature set that MinIO’s OSS version no longer matches.

Personal usecase

Personally, I use Ceph in my homelab as a single-host NAS setup. I’ve configured different pools for:

VM storage
Object storage
File storage
RWX-capable storage for my talos Kubernetes cluster via the Ceph CSI

My hardware setup includes:

count	drive
4	4TB NVMe datacenter-grade drives
21	8TB SAS drives
12	16TB SATA drives (joining the pool soon)

Using CRUSH maps, I assign disks to different use cases:

VMs reside on NVMe with three-way replication
CephFS metadata also uses three-way replicated NVMe
File and object storage use Erasure Coding (EC 18+3) on HDDs, yielding ~86% usable space while tolerating up to 3 disk failures

Performance-wise, the HDD pool delivers around 2.6 GB/s read/write, bottlenecked by my Xeon E2640v4 CPU, as the EC 18+3 config is quite compute-intensive. Replication deliveres native NVMe speeds.

If you’re exploring Ceph or Rook and have questions, feel free to reach out, happy to help!

Setting up Harbor as private Docker registry

Maximilian Marschall — Fri, 28 Mar 2025 00:19:01 +0200

Because I didn’t want to push all my custom images and sourcecode to public repos like hub.docker.com, I was looking for a self-hosted solution. After checking out the features of the official Docker registry, I was a bit underwhelmed by the missing repo based auth and the lack of combining the use as repository and cache.

Why would you need private Docker registry?

First if you don’t feel comfortable in sharing all your images with a third party you’ll need to find a more local way of distributing your images. Also Docker changed their policies and limit the maximum pulls you can do to 10 or 100 pulls per hour (depending if your autheticated or not), and while this may seem like a big number, I can assure you it’s easy to reach it. Imagine you got a kubernetes cluster, and you restart some nodes because of an update. Now every node is pulling every image it needs if you’ve set your imagePullPolicy to Always. I really like to keep my deployments configured that way, because it makes updates via kubectl rollout restart deployment x really easy for uncritical services.

To mitigate the pull limits, we will use a cache in our private repository.

Candidates

I spent quite some time on digging through the different options. While doing so, I checked out the following projects:

Portus (made by SUSE)
JFrog Container Registry
Harbor (CNCF Graduated project)

Portus

After looking into Portus, I quickly realised that this project is quite dead. The last commit dated back to Mar 25, 2020 and was only a change in the CHANGELOG.md, also the last release is from Mar 19, 2019.

JFrog Container Registry

I know JFrog from work and know it works, but it’s closed source, and I think it’s quite overkill for my private use-cases.

Harbor

Checking out Harbor, I quickly noticed the CNCF banner on their website. The graduated state in CNCF means it’s stable and should be quite easy to handle.

Also, they have an official helm3 chart available.

Deploying

To get the helm chart we will use the bitnami chart via oci.

# get default values
helm show values oci://registry-1.docker.io/bitnamicharts/harbor > values.yaml

After fetching the values we need to change some stuff to match our infrastructure. For example I needed to set the ingress to LoadBalancer and change the StorageClass.

You should walk through the default values and adjust everything to your needs, here I’ll share my values file without any secrets (you’ll need to change all those xxx)

registry:
 secret: xxx
 credentials:
 password: xxx
jobservice:
 secret: xxx
core:
 secret: xxx
 xsrfKey: xxx
expose:
 type: ingress
 tls:
 enabled: true
 certSource: secret
 secret:
 secretName: "hub.marschall.systems-tls"
 ingress:
 hosts:
 core: "hub.marschall.systems"
 controller: default
 className: "nginx-dmz"
 annotations:
 ingress.kubernetes.io/ssl-redirect: "true"
 ingress.kubernetes.io/proxy-body-size: "0"
 nginx.ingress.kubernetes.io/ssl-redirect: "true"
 nginx.ingress.kubernetes.io/proxy-body-size: "0"
 cert-manager.io/cluster-issuer: cloudflare-issuer
externalURL: https://hub.marschall.systems
ipFamily:
 ipv6:
 enabled: false
 ipv4:
 enabled: true
persistence:
 enabled: true
 resourcePolicy: "keep"
 persistentVolumeClaim:
 registry:
 storageClass: "vsan-default"
 accessMode: ReadWriteOnce
 size: 200Gi
 jobservice:
 jobLog:
 storageClass: "vsan-default"
 accessMode: ReadWriteOnce
 size: 1Gi
 redis:
 storageClass: "vsan-default"
 accessMode: ReadWriteOnce
 size: 1Gi
 trivy:
 storageClass: "vsan-default"
 accessMode: ReadWriteOnce
 size: 5Gi
 imageChartStorage:
 disableredirect: true
 type: filesystem
 filesystem:
 rootdirectory: /storage
imagePullPolicy: IfNotPresent
updateStrategy:
 type: Recreate
logLevel: info
existingSecretAdminPasswordKey: HARBOR_ADMIN_PASSWORD
harborAdminPassword: "xxx"
secretKey: "xxx"
database:
 type: external
 external:
 host: "db1.marschall.management"
 port: "5432"
 username: "xxx"
 password: "xxx"
 coreDatabase: "harbor"
 sslmode: "require"
 maxIdleConns: 10
 maxOpenConns: 50
metrics:
 enabled: true
 serviceMonitor:
 enabled: true
trace:
 enabled: false

Now we can install the application with

helm install --namespace harbor --name harbor --create-namespace oci://registry-1.docker.io/bitnamicharts/harbor -f values.yaml

Configuring pull through cache for DockerHub

To make use of the millions of public available images on DockerHub without running into the pull limits we no create a new upstream Registry in Harbor.

Now we need to map the Registry to a Project.

Congratulations you now have your own docker registry that can be used as pull through proxy! To do so just substitute docker.io with <yourregistry>/cache in my case docker.io/hugomods/hugo:git becomes hub.marschall.systems/cache/hugomods/hugo:git

Simplify logging into all your VKS Clusters with one Command

Maximilian Marschall — Tue, 25 Mar 2025 22:13:01 +0200

Having to log into all of your VKS clusters one by one sucks as you either have to export your vSphere Password as an environment variable (which is kinda insecure) or enter the Password for each of your Clusters over and over again.

As we got annoyed with that one of my Customers (Ralf Dahmen) and I decided to fix this by developing a little bash script that automatically detects all clusters in your vSphere Namespaces and logs you in to them, to do so you only ned to enter your Password once.

Prequesits

To run the script you’ll need to have some binaries available, those are sed, tail, grep, jq and expect this shouldn’t be a problem for any modern linux distro or MacOS.

Here are some examples on how to get those commands..

Ubuntu/Debian:

apt-get install sed tail grep jq expect

Arch:

pacman -Sy sed grep jq expect

Script

Just save the following script as vks-login.sh file in your PATH and change permissions chmod +x vks-login.sh afterwards you can login with vks-login.sh.

#!/bin/bash
# Ralf Dahmen
# Maximilian Marschall <maximilian.marschall@3xm.at>
# https://msta.cc https://marschall.systems
# Requirements:
# apt-get install sed tail grep jq expect

# BEGIN USER SETTINGS
SERVER=svc.marschall.management
USERNAME=Administrator@vsphere.local
# END USER SETTINGS

# DO NOT MODIFY AFTER THIS LINE IF YOU'RE NOT 100% SURE WHAT YOUR DOING

GREEN='\033[032m'
NC='\033[0m' # No Color

unset -v KUBECTL_VSPHERE_PASSWORD
set +o allexport
IFS= read -rsp "Please enter password for $USERNAME: " KUBECTL_VSPHERE_PASSWORD
echo ""

expect - <<EOF
 log_user 0
 spawn kubectl vsphere login --server=$SERVER --insecure-skip-tls-verify --vsphere-username $USERNAME
 expect Password:
 send -- "$KUBECTL_VSPHERE_PASSWORD\r"
 expect eof
EOF

namespaces=$(kubectl config get-contexts | tr -s " " | sed -e "s/ /,/g" | tail -n +2 | awk -F, '{print $NF}' | grep -v '^$' | sort | uniq)

if [ -z "$namespaces" ]; then
 echo "No namespaces found"
 exit 1
else
 ns_count=$(echo "$namespaces" | wc -l)
 echo "$ns_count Namespaces found"
 for namespace in $namespaces; do
 kubectl config use-context $SERVER > /dev/null 2>&1
 tkcs=$(kubectl get cluster -n $namespace -o json | jq -r .items[].metadata.name)
 if [ -z "$tkcs" ]; then
 echo "no clusters found in $namespace"
 else
 tkc_count=$(echo "$tkcs" | wc -l)
 echo "$tkc_count clusters found in $namespace"
 for tkc in $tkcs; do
 kubectl config use-context $SERVER > /dev/null 2>&1
 echo -e "${GREEN}Login to $tkc in $namespace ${NC}"
 expect - <<EOF
 log_user 0
 spawn kubectl vsphere login --server=$SERVER --insecure-skip-tls-verify --vsphere-username $USERNAME --tanzu-kubernetes-cluster-namespace $namespace --tanzu-kubernetes-cluster-name $tkc
 expect Password:
 send -- "$KUBECTL_VSPHERE_PASSWORD\r"
 expect eof
EOF
 done
 fi
 done
fi

Scaling with NSX-T and ALB beyond vNIC limits

Maximilian Marschall — Thu, 11 Jul 2024 19:15:32 +0200

When using NSX-T for networking in combination with NSX ALB for load balancing in the vSphere IaaS Control Plane, the ALB Service Engines attach to each Tier-1 Router (vSphere Namespace) with one vNIC. Naturally, there’s a limit of 10 vNICs per Service Engine, meaning we can connect up to 8 Tier-1 routers to one Service Engine, as one vNIC is needed for management and another for frontend traffic.

We were curious to see how NSX ALB would handle more than 8 vSphere Namespaces in this setup. My colleague Steven Schramm and I decided to test this in our lab.

Test Setup

Our lab environment was based on the following components:

vCenter 8.0.3 (24022515)
ESXi 8.0.3 (24022510)
NSX ALB 22.1.5 (22.1.5-2p6)
NSX-T 4.1.2.4

For testing, we created 9 vSphere Namespaces, each with one workload cluster using default settings and the default VM-Class xsmall-best-effort.

Observations

We observed the behavior of NSX ALB as we created the ninth vSphere Namespace and the corresponding Workload Cluster. As expected, NSX ALB started to deploy a new Service Engine to accommodate the additional namespace.

To visualize this procedure we created two drawings. The first drawing shows the number of Service Engines and the distribution of the running virtual services with nine vSphere Namespaces and one K8s Cluster per vSphere Namespace. Based on the corresponding Service Engine Group each of those Service Engines is allowed to run 11 virtual services. Further the drawing shows eleven virtual services. One virtual service for the K8s APi of the Supervisor cluster, one for the CSI controller and nine for the running K8s clusters. The virtual services for the Supervisor K8s API and the CSI controller are running in the same VRF and the different K8s clusters are running in dedicated VRFs. One VRF for each vSphere Namespace. As you can see we do have two Service Engines deployed, since the Service Engine Group is running in HA Mode N+M and one additional Service Engine is required to cover the outage of one Service Engines an still have enough capacity to maintain all running virtual Services.

Based on the maximum allowed ratio of 11 virtual services per Service Engine two Service Engines would cover the requirement to maintain all running virtual services, but this is not true with 10 VRFs and the requirement of one vNic per VRF. Still we only have two Service Engines, because the number of used vNics will not be taken in consideration for the N+M formular.

What does this mean in case of a SE failure? After one of the two SEs did fail the Controller trys to reschedule the virtual services, but is unable to get all ten VRFs/ ten vNics connected to the same SE, since one vNIC is already used for the management connection of the SE. Based on this missing capacity of available vNICs, a new SE needs to be created and one of the virtual Services is out of service until this SE is initialized. This can be prevented by increasing the valued for the SEs which should be deployed as a buffer or by reducing the amount of accepted virtual services per SE, but this does also mean you will increase your used licenses and the corresponding costs.

The second drawing shows the effect of further scaling. In the specific example we inreased the minimum scale per virtual service from one to two. Based on this each virtual service have to run on at least two different SEs. This will also increase the number of required service engines from two to three, but the problem regarding the capacity of available vNICs is still valid.

Conclusion

This test indicates that the limitation of vSphere Namespaces will not be reduced by the number of available vNIC interfaces per SE, but in case of an SE failure the recovery time can be much longer. This can be solved by changing the values responsible for the scaling of the number of SEs in the N+M HA mode for example, but this might have an impact on the number of licenses you will require to keep you workload running and reduce the impact of an SE failure.

vSphere 8.0.3 Nested on Proxmox/KVM

Maximilian Marschall — Thu, 04 Jul 2024 09:15:32 +0200

While I was testing Broadcom’s new Release 8.0.3, I encountered some issues getting my nested lab to work in a Proxmox environment. Despite enabling Nested Virtualization on all Proxmox nodes and setting the VM CPU type to ‘host,’ the vCLS service wouldn’t start due to EVC issues, causing DRS to malfunction.

Troubleshooting Nested Virtualization

First, I checked whether nested virtualization was enabled on the Proxmox nodes. This can be verified with the following command:
```
root@pve0:~# cat /sys/module/kvm_intel/parameters/nested
Y
```
Next, I ensured the CPU type was set to ‘host’:
Finally, I checked if the ESXi node reported HV enabled:
```
[root@esx3:~] esxcfg-info|grep "HV Support"
 |----HV Support............................................3
```
- 0 - VT/AMD-V support is not available for this hardware.
- 1 - VT/AMD-V might be available but is not supported for this hardware.
- 2 - VT/AMD-V is available but is not enabled in the BIOS.
- 3 - VT/AMD-V is enabled in the BIOS and can be used.

Deeper Look into vCLS

Since version 8.0.3, vCLS 2.0 is active, meaning the vCLS VMs are now Pods instead of traditional VMs. The issue is that the Pods configuration cannot be altered in any documented way. Therefore, I looked into reverting to the “old” vCLS model.

Solution

To disable the new vCLS version and revert to the old one, I found an advanced vCenter option. Follow these steps:

Navigate to vCenter > Configure > Advanced Settings.
Set config.vcls.disablePodCrx to true (default is false).

With this change, vCLS is deployed as VMs again. You can now change the EVC settings of the VMs after upgrading their VM compatibility to vSphere 8.0.2.

Conclusion

I hope Broadcom addresses this problem or at least documents an official way to work around it. This issue is a significant roadblock for homelabbers like me who use non-ESXi based host systems on their hypervisors. I suspect the same problem might occur on unsupported CPUs, which are also widespread in the homelab community.

vSphere IaaS Control Plane - Consumption Interface

Maximilian Marschall — Tue, 02 Jul 2024 18:15:32 +0200

VMware by Broadcom has recently released the “Consumption Interface” for the vSphere IaaS Control Plane (formerly known as vSphere with Tanzu or TKGS). This is a significant enhancement to the existing features of TKG, allowing for better management and visibility of your resources.

With this new interface, you can now view and manage all your resources within vSphere namespaces directly from the vCenter GUI. This update simplifies the creation and updating of resources, enhancing the user experience.

Installation Guide

To install the Consumption Interface, follow these steps:

Prerequisites: Ensure your Supervisor Cluster is running vCenter 8.0.3 or later.
Download Required Files: Access the required files from the Supervisor Services GitHub Repository.
Import Configuration: Download the necessary .yml file and import it into the Workload Management Services tab in vCenter.

Here is a step-by-step guide with screenshots to assist you through the installation process:

Step 1: Download the .yml file from the repository.
Step 2: Navigate to the Workload Management Services tab.
Step 3: Import the .yml file.
Step 4: Configure the Supervisor Service.
Step 5: Adjust the package settings as needed, such as overriding the image registry to use your local registry.

Post-Installation Overview

After enabling the Consumption Interface, it will appear in the resources tab of your vSphere namespace.

From this interface, you can create and manage your IaaS resources, such as Kubernetes Clusters.

You can even preview the generated YAML that will be deployed on your behalf.

Conclusion

With these steps, you can seamlessly integrate the Consumption Interface into your existing vSphere environment, enhancing your control and visibility over your IaaS resources and providing an easy-to-use UI for your vSphere Admins to create Kubernetes Clusters and other resources.

This addon has proven to be highly beneficial for many of my customers who were seeking a simple interface for creating Kubernetes clusters.

Using Antrea Egress to Identify Your Workloads at an External Firewall

Maximilian Marschall — Tue, 26 Mar 2024 19:13:01 +0200

Identifying different applications running in the same Kubernetes cluster at an external firewall can be challenging. By default, all services running on a node are SNATed to the node’s IP address.

With Antrea Egress we can make use of dedicated egress IPs for different Pods in a Cluster. If you just want to get to the quickstart, click here.

What is Antrea?

Antrea is a Kubernetes CNI which is currently a CNCF Sandbox Project. It is heavily used in Production by VMware and its Customers as it is the default CNI for VMware Tanzu.

It leverages Open vSwitch as dataplane and thus is pretty performant.

How does Antrea Egress work

Instead of applying an SNAT to outgoing traffic to the Nodes IP Antrea applies a SNAT Policy to your custom Egress-IP.

Quickstart

When already using Antrea as CNI it’s quite straight forward to make use of Antrea Egress:

For Antrea versions older than 1.0, you need to upgrade. For versions older than 1.6, you need to enable a FeatureGate. For newer versions, proceed to step 2.

apiVersion: v1
kind: ConfigMap
metadata:
 name: antrea-config
 namespace: kube-system
data:
 antrea-agent.conf: |
 featureGates:
 Egress: true
 antrea-controller.conf: |
 featureGates:
 Egress: true

Create an Egress Policy:

apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
 name: example-egress
spec:
 appliedTo:
 namespaceSelector:
 matchLabels:
 env: custom-egress-ip
 podSelector:
 matchLabels:
 role: custom-egress-ip
 egressIP: 10.10.30.30

With this policy, all pods labeled role: custom-egress-ip in namespaces labeled env: custom-egress-ip will use 10.10.30.30 as their egress IP when connecting to external IPs.

Drone CI/CD

Maximilian Marschall — Wed, 23 Mar 2022 03:08:41 +0200

As I wanted to get back to writing some more blog posts I was annoyed by the fact that I need to build, push and deploy the updated posts everytime something changes.

As I already managed my whole blog with hugo and within a git repository hosted on my very-own instance of Gitea I decided it’s time to get into self-hosting CI/CD for my personal projects.

why drone?

I checked out multiple other solutions like Gitlab, Jenkins, Bamboo and Circle CI but all of them seemed a bit overkill for my needs.

Also drone has ready to use helm charts for both the server and the runner (I’ll explain both later on) which is great considering I already have nearly all my services running inside a kubernetes cluster.

what makes drone different

Drone is based on docker images, basically any action in your pipeline is a Docker-Container. That makes the system incredibly flexible because you can simply build plugins with new Containers or use existing containers for your build-steps, for example golang to build your app.

server and runner

Basically the drone server is the component you are interacting with either via drone-cli oder the web-gui. The server also receives the webhooks from your git repo and dispatches jobs to runners which do the actual work.

Runners come in a variety of sizes and shapes, for example they can run..

in kubernetes
via SSH
in Docker
on AWS
on DigitalOcean

..and some more you can find all of them in their docs.

Obviously I’ll be using the kubernetes runner.

deploying drone to kubernetes

I choose to deploy drone via helm3 because it’s easy and supported.

You can simply add the drone helm repos to your helm3 installation via the following commands:

helm repo add drone https://charts.drone.io
helm repo update

To deploy I’d suggest checking the chart docs over at Github. Depending on your VCS the steps may vary slightly.

.drone.yml

This is the file which tells drone how to handle your repository, it contains a pipeline with all steps needed to build your project.

This website is built using the following .drone.yml (I redacted confidential information)

---
kind: pipeline
type: kubernetes
name: default

steps:
 - name: generatetag
 image: hub.docker.com/alpine
 commands:
 - echo "${DRONE_REPO_BRANCH}-${DRONE_COMMIT_SHA},latest" >> .tags

 - name: submodules
 image: hub.docker.com/alpine/git
 commands:
 - git submodule update --init --recursive

 - name: dockerbuild
 image: plugins/docker
 settings:
 username:
 from_secret: docker-username
 password:
 from_secret: docker-password
 repo: hub.docker.com/youruser/yourimage
 registry: hub.docker.com

 - name: deploy
 image: myprivatedockerrepo/drone-plugins/k8s-deploy
 settings:
 token:
 from_secret: k8stoken
 server:
 from_secret: k8sserver
 namespace:
 from_secret: k8snamespace
 deployment: marschallsystems
 container: marschallsystems
 repo: hub.docker.com/youruser/yourimage
 tag: ${DRONE_REPO_BRANCH}-${DRONE_COMMIT_SHA}

As you see the file isn`t too complicated, every step has a name and an image. The image is the Docker-Image used for the step.

At first, we are using alpine to generate a .tags file which contains all target tags for the resulting Image.

The Second Step is building the Dockerfile from which this blog is created (I will cover this in another post and put a link over here). This step will also take the tags from .tags and tag the image and push it. All credentials are supplied from drone secrets, these can be either set via the web gui or via drone-cli.

At last, the image which is used in the kubernetes deployment is updated via a little custom drone plugin. Actually this “plugin” is really only a bash one-liner in a docker container.

deployment update plugin

To update the image we just need to issue a kubectl set image deployment/deploymentname containername=newimagename:tag

building the image

As previously stated the image is no big deal. It consists of a script.sh and a corresponding Dockerfile. These two files need to be in the following places.

.
├── src
│ └── script.sh
└── Dockerfile

script.sh

#!/bin/sh
/bin/kubectl --token $PLUGIN_TOKEN --server $PLUGIN_SERVER --namespace $PLUGIN_NAMESPACE --insecure-skip-tls-verify set image deployment/$PLUGIN_DEPLOYMENT $PLUGIN_CONTAINER=$PLUGIN_REPO:$PLUGIN_TAG

Dockerfile

FROM alpine
RUN apk add --no-cache curl ca-certificates
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && \
 chmod 755 kubectl && \
 mv kubectl /bin/
ADD src/script.sh /bin/
RUN chmod +x /bin/script.sh
ENTRYPOINT /bin/script.sh

Now we can build the image.

docker build . -t hub.docker.com/youruser/yourimage:tag

kubectl auth

Obviously we need some kind of authentication for the kubectl command to work, and we will leverage the power of drone secrets again.

To get a token to use for kubectl we use a service-account.

Todo so we apply following yml in the cluster (change namespaces according to your deployments needs)

apiVersion: v1
kind: ServiceAccount
metadata:
 name: drone-deploy
 namespace: marschallsystems
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
 name: drone-deploy
 namespace: marschallsystems
rules:
 - apiGroups: ["apps"]
 resources: ["deployments"]
 verbs: ["get","list","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: drone-deploy
 namespace: marschallsystems
subjects:
 - kind: ServiceAccount
 name: drone-deploy
 namespace: marschallsystems
roleRef:
 kind: Role
 name: drone-deploy
 apiGroup: rbac.authorization.k8s.io

This will create the service account and the corresponding role and bind them together. Now we need to get the exact name of the secret which kubernetes created for the service account. To do so we use kubectl get secrets -n yournamespace. After that we can receive the token with the following command (replace the secret name to match yours).

kubectl get secrets drone-deploy-token-tn8t2 -o=jsonpath="{.data.token}" | base64 -d -i -

Congratulations you now have a token which you can save in drone as a secret with the name k8stoken.

conclusion

After just a few hours we have a working CI/CD solution with very little footprint.

At first, it might seem a bit overwhelming to describe every step in the yml but after all it’s quite easy if you get the hang of it.

I’ve been also surprised how easy it is to make new plugins, and I just love it!

I hope I’ve been able to show you how amazing drone is. Simple, fast and easy to learn!

Really nice hardware for a Firewall-Appliance

Maximilian Marschall — Sun, 20 Mar 2022 23:34:01 +0200

As my whole lab and network was getting to the 10Gbit era I was looking for a nice piece of hardware to run OPNSense.

Because I already have a Mikrotik 8-port SFP+ switch (CRS309-1G-8S+IN), I wanted SFP+ ports because DAC Cables are just ridiculous cheap. I really can recommend the guys over at fs.com, they have good quality and are very cheap. Also, none of the equipment I ordered there let me down till now.

At first, I was looking into Thin-Clients and also some SFF PCs like the Lenovo or Dell 1l variants but just wasn’t satisfied with the performance. The big drawback of mini PCs is the limited expandability with PCIe devices.

Supermicro SYS-E300-9D-4CN8TP

Finally, I settled on the Supermicro SYS-E300-9D-4CN8TP, CPU is already on-board, and I had some ECC DDR4 memory lying around. I just needed to get a decent datacenter SSD for reliability and settled on the Samsung PM883 240 GB.

This little beast has 4 x 10Gbit, 2 of them being RJ45 and the other 2 SFP+. And with its Xeon D-2123IT 4c/8t it’s beefy enough for anything you throw at it.

Another really nice feature is the dedicated IPMI port which removes the need to ever plug a monitor into this bad boy.

Conclusion

I am really happy how the whole process went and OPNSense plays very well with the hardware. Every update went fine, and I always felt good knowing I’d be able to remote into the machine without plugging a display n it.

Maybe the machine is a bit overpowered for my “home-use”, but at least I’m sure I’ll never need to be worried about a bottleneck for my WireGuard VPN tunnels.

IPMI was also great for first time setup as you can mount an ISO remote via the IPMI which eliminates the need of flashing an installer to a flash drive.

PS: 32 Gigs of memory are way too much for a lab firewall ;)

Using Systemwide Darkmode settings on your Website

Maximilian Marschall — Mon, 27 Apr 2020 17:01:01 +0200

Systemwide dark/lightmode settings are becoming an standard in most Operating Systems like Android, iOS, MacOS, Manjaro and many many more. So why not use it on your Website or in your Webapp?

CSS-Media Queries

To check whether the user prefers the dark mode, a CSS3 feature called Media Queries is used. Media Queries are quite powerful, not only you can check for the preferred color-scheme. They are also used for responsive design based on the screen-size or specific styles for the printed versions.

CSS-Variables

For easy changes to the color theme you can utilize CSS Variables. You can define them like that.

html {
--variable-1: #dfdfdf;
--variable-2: url(https://example.com/example.png)
}

To use them just use var(<varname>) where you want the value to be inserted.

React to the setting

To activate the different themes we will put following code in our CSS file

html {
--color: #42464c;
--background: #f8f8f8;
}
@media (prefers-color-scheme: dark) {
html {
--color: #dfdfdf;
--background: #333435;
--color-subtitle: #ddd;
}
}

If the user activated darkmode the second definition will overrule the first one and the variables change, and so does your page.

Optimizing your rclone mount

Maximilian Marschall — Tue, 21 Apr 2020 19:13:01 +0200

Because your Internet speed is not always as fast as your needs for more storage space you should use some kind of an buffer.

What is mergerfs?

Mergerfs is a tool to logicaly combine multiple drives or folders into one.

On the offical Github theres the following example which pretty much describes everything you need to know about how mergerfs works.

A + B = C
/disk1 /disk2 /merged
| | |
+-- /dir1 +-- /dir1 +-- /dir1
| | | | | |
| +-- file1 | +-- file2 | +-- file1
| | +-- file3 | +-- file2
+-- /dir2 | | +-- file3
| | +-- /dir3 |
| +-- file4 | +-- /dir2
| +-- file5 | |
+-- file6 | +-- file4
|
+-- /dir3
| |
| +-- file5
|
+-- file6

How we will use it

We’ll merge our rclone mount and some local folder, after that every read will first be done on the local folder unless the file is only availabe on the rclone mount. Similar rules apply to writes, files will be writen to the local folder unless the local folder runs out of space.

To automaticaly empty our local folder we’ll be using a little script that will upload the files to our rclone mount and delete the local copies after that.

Installing mergerfs

First of all you need to grab the latest release for your distro over at Github.

After installing you should be able to run mergerfs -v in your shell to check the installed version and get output similar to mine.

mergerfs version: 2.28.2
FUSE library version: 2.9.7-mergerfs_2.28.0
fusermount version: 2.9.2
using FUSE kernel interface version 7.29

Creating the mount

To make sure the mount we will start as soon as the dependicies are ready we’ll be using a systemd service.

Creating a service for systemd is as easy as creating a file under /etc/systemd/system.

Our file should look something like this.

[Unit]
Description=mergerfs mount
Requires=rclone-merge.service
After=< your rclone mount service name goes here>.service
RequiresMountsFor=< your local folder goes here >
RequiresMountsFor=< your rclone mountpath goes here >
[Service]
Type=forking
ExecStart=/usr/bin/mergerfs < your local folder goes here >:< your rclone mountpath goes here > < your target mountpath goes here > \
-o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target

After that you need to reload your systemd with following command systemctl daemon-reload. Now you should be able to start and stop your mergerfs with systemctl start rclone-merge.

The upload script

To automate the process of uploading local data to your remote we’ll use this fancy little script. The script itself checks if another instance is running and only runs if it’s the only one.

#!/bin/bash
## RClone Config file
RCLONE_CONFIG=/etc/rclone/rclone.conf
export RCLONE_CONFIG
#exit if running
if [[ "`/usr/sbin/pidof -x $(basename $0) -o %PPID`" ]]; then exit; fi
## Move older local files to the cloud
/usr/bin/rclone move < your local folder goes here > < your remote name goes here >: \
--log-file /var/log/rclone-upload.log \
--exclude-from /opt/scripts/excludes \
--delete-empty-src-dirs \
--fast-list \
--bwlimit "6M" \
--min-age 2h

You can and should ajust the arguments acording to your needs, they are documented over here.

The most importend flags in my opinion are:

--exclude-from to exclude some files/folders defined in a text file like this:
```
*.rar
download/**
```
--delete-empty-src-dirs to keep the local folder clean.
--fast-list to speed up the process.
--bwlimit to limit the maximum upload
--min-age to upload only files that are older than some time (this helps to make sure only completed copys or downloads are moved)

Executing the Script with cron

Last but not least we should make sure the script will be run in some interval. I choose to run the script every 30 minutes, to do so I use the command crontab -e.

0,30 * * * * /opt/scripts/upload_gdrive.sh

Mounting Google Drive to Linux made easy

Maximilian Marschall — Sat, 27 Jul 2019 10:13:01 +0200

Because VPS with storage are quite expensive I was looking for a cheap way to store my backups and media in the cloud without the fear of Google or Amazon having my whole data.

Google Drive Business

With google drive it’s pretty cheap to have “unlimited” online storage, their business plan includes unlimited storage (1TB per user if fewer than 5 users). But the limit of 1TB per user is not enforcing. Here’s a link to their pricing page.

But how should I mount it? rclone!

This is where an open source project comes in handy, rclone. It lets you FUSE mount google drive and some other cloud providers (Amazon Drive, Amazon S3, Backblaze B2, Dropbox, …) to use them as block sotrage.

But the best part: it enables you to encrypt every file you store on this mount and even the file names.

Install rclone

To install rclone on Linux just run following command:

curl https://rclone.org/install.sh | sudo bash

If you don’t trust the Script you can also manually download and install rclone, you can find help over here you can also find versions for other Platforms in their download section.

Now you can test your install with rclone version and you will get something like that:

rclone v1.48.0-099-gc2635e39-beta
- os/arch: linux/amd64
- go version: go1.12.7

Configure rclone

To configure rclone you can use a config file or the config command. We will use the config command as follows: rclone config

It will now ask you to create a new remote, press n to do so. First of all you need to choose a name for your remote, I’ll be using GD for my googledrive.

Next you’ll be choosing the backend, in this case we’ll be using drive which stands for Google Drive.

For the next step we’ll need an API Key from Google, you can obtain one at your API Console. You need to create a project here, and enable the Google Drive API under “ENABLE APIS AND SERVICES”. After that you’ll need to click on “Credentials” (not “Create credentials”), then Create credentials, then “OAuth client ID”. You will be prompted to set the OAuth screen product name, if you haven’t set one yet. Now choose application type “other” and “Create”. Now you will be shown a client ID and client secret use this two values in your rclone config.

You won’t need the advanced config, so skip this for now.

If you are configuring rclone on a remote/headless machine you’ll need to use manual config so say now to auto config and follow the steps described by rclone.

We won’t configure this as a Team Drive so no to the next question.

After that you’ll be shown the configfile and it will be OK, so just say yes.

Encrypting your data

To encrypt the data which is uploaded to your remote we need to make some changes to your rclone config.

To do so we need to run rclone config again and press n to add a new remote, I’ll call it gcrypt. Next we’ll need to choose 9 (crypt) after that you need to enter the remote you created in the last step, in my case it’s GD.

The next step is filename encryption, I’d suggest to use 2 (standard) after that it’s all about directoryname encryption choose 1 to encrypt directory names.

Here’s the important step either choose your own password to encrypt files or let rclone generate a random one. This password will be stored encrypted in your rclone.conf but store this password in a safe place, if you loose it you’ll lose access to your data. Same rules apply to the next step: Password for salt.

Congratulations you made it!

Mount on startup

To auto mount your drive on boot you can use a service file for systemd. Just create a .service file under /etc/systemd/system.

If you encrypted your data use the name of your encrypted remote.

/etc/systemd/system/rclone-gmedia.service:

[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
Environment=RCLONE_CONFIG=/etc/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount <your remote name goes here>: <your mountpath goes here> \
--allow-other \
--log-level INFO \
--log-file /var/log/rclone.log \
--umask 002
Restart=on-failure
User=<your user goes here>
Group=<your group goes here>
[Install]
WantedBy=multi-user.target

After that you are able to start and stop your mount with systemctl as follows:

action	command
start	`systemctl start rclone-gmedia`
stop	`systemctl stop rclone-gmedia`
check status	`systemctl status rclone-gmedia`
add to autostart	`systemctl enable rclone-gmedia`

If you want to know how to make your setup even better check out my new post.