Skip to main content

Exposing Workloads Through Tailscale

Overview

This document describes how to securely expose Kubernetes services to your Tailscale network (tailnet) using the Tailscale Kubernetes Operator. This approach provides private access to cluster services without exposing them to the public internet or modifying cluster template configurations.

Goals

  • Expose internal services (dashboards, admin panels) securely via Tailscale
  • Automatic DNS via Tailscale MagicDNS
  • Automatic TLS certificates from Let's Encrypt
  • No changes to cluster templates or domain configuration
  • Simple, reusable pattern for any service

Architecture

Components

Tailscale Kubernetes Operator

  • Manages Tailscale proxy pods for exposed services
  • Integrates with Kubernetes Ingress API
  • Provisions TLS certificates automatically
  • Registers services in Tailscale MagicDNS

Tailscale Ingress Pattern

  • Uses standard Kubernetes Ingress resources
  • Set ingressClassName: tailscale to trigger operator
  • Creates dedicated proxy pod per Ingress
  • Proxy appears as device in Tailscale admin console

Traffic Flow

How It Works

  1. Ingress Creation: Create Kubernetes Ingress with ingressClassName: tailscale
  2. Operator Detection: Tailscale operator watches for new Ingress resources
  3. Proxy Deployment: Operator creates dedicated Tailscale proxy pod
  4. Tailnet Registration: Proxy registers as device in your Tailscale network
  5. DNS Registration: Service hostname added to MagicDNS (<namespace>-<ingress-name>.<tailnet>.ts.net)
  6. TLS Provisioning: Let's Encrypt certificate requested on first HTTPS connection
  7. Traffic Routing: Client � Tailscale network � Proxy pod � Service

Implementation

Prerequisites

  • Tailscale Kubernetes Operator installed in cluster
  • Tailscale account with MagicDNS enabled
  • HTTPS enabled on tailnet (required for Let's Encrypt)
  • Service to expose running in cluster

Step 1: Create Tailscale Ingress

Create ingress-tailscale.yaml in your application directory:

---
# Example: Rook Ceph Dashboard
# Access via: https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rook-ceph-dashboard-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: rook-ceph-mgr-dashboard
port:
number: 7000
tls:
- hosts:
- rook-ceph-dashboard

Key fields:

  • ingressClassName: tailscale - Triggers Tailscale operator (required)
  • defaultBackend.service.name - Target service name
  • defaultBackend.service.port.number - Target service port
  • tls.hosts - Hostname (operator appends namespace and tailnet domain)

Step 2: Add to Kustomization

Update app/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./helmrelease.yaml
- ./ingress-tailscale.yaml # Add this line

Step 3: Deploy

# Validate manifests
task configure

# Commit changes
git add -A
git commit -m "feat: expose service via tailscale ingress"
git push

# Apply to cluster
task reconcile

Step 4: Verify

# Check Ingress status
kubectl get ingress -n <namespace> <ingress-name>-tailscale

# Check for Tailscale proxy pod (may take 1-2 minutes)
kubectl get pods -n <namespace> | grep tailscale

# View proxy logs
kubectl logs -n <namespace> -l app.kubernetes.io/name=tailscale

# Verify in Tailscale admin console
# Navigate to Machines � Find "<namespace>-<ingress-name>-tailscale"

Step 5: Access Service

Access via Tailscale MagicDNS:

https://<namespace>-<ingress-name>.<TAILNET>.ts.net

Example (Rook Ceph Dashboard):

https://storage-rook-ceph-dashboard.<TAILNET>.ts.net

Note: Replace <TAILNET> with your actual Tailscale network name (e.g., example-network.ts.net)

Step 6: First Access and Certificate Provisioning

IMPORTANT: The first HTTPS connection triggers automatic TLS certificate provisioning from Let's Encrypt. This process takes 30-60 seconds.

What happens on first access:

  1. Browser attempts HTTPS connection
  2. Tailscale proxy detects missing certificate
  3. Registers ACME account with Let's Encrypt
  4. Requests DNS-01 challenge
  5. Updates Tailscale DNS records
  6. Let's Encrypt validates domain ownership
  7. Certificate issued and cached

Expected behavior:

  • First request may timeout or take 30-60 seconds to load
  • Subsequent requests are instant (certificate cached)

Proxy logs during certificate provisioning:

# Watch certificate provisioning in real-time
kubectl logs -n tailscale ts-<ingress-name>-<suffix> -f

# You'll see logs like this:
cert("<service>.<TAILNET>.ts.net"): registered ACME account.
cert("<service>.<TAILNET>.ts.net"): starting SetDNS call...
cert("<service>.<TAILNET>.ts.net"): did SetDNS
cert("<service>.<TAILNET>.ts.net"): requesting cert...
cert("<service>.<TAILNET>.ts.net"): got cert

Troubleshooting first access:

If the browser shows an error on first access:

  1. Wait 60 seconds - Certificate provisioning takes time
  2. Refresh the page - Certificate may have been issued during timeout
  3. Check proxy logs - Look for "got cert" message
  4. Verify HTTPS enabled - Must be enabled in Tailscale admin console (Settings → HTTPS)

Common first-access errors:

ErrorCauseSolution
ERR_CONNECTION_TIMED_OUTCertificate provisioning in progressWait 60 seconds, refresh page
ERR_NAME_NOT_RESOLVEDMagicDNS not workingReconnect Tailscale client, verify MagicDNS enabled
SSL_ERROR_BAD_CERTHTTPS not enabled on tailnetEnable HTTPS in Tailscale admin console
NET::ERR_CERT_COMMON_NAME_INVALIDWrong hostname usedUse full hostname <service>.<TAILNET>.ts.net

After successful first access:

  • Certificate valid for 90 days
  • Auto-renewed by Tailscale proxy
  • All subsequent connections are instant
  • Certificate stored in proxy pod memory

Real-World Example: Rook Ceph Dashboard

File Structure

kubernetes/apps/storage/rook-ceph-cluster/
�� ks.yaml
�� app/
�� helmrelease.yaml
�� ingress-tailscale.yaml # Tailscale ingress
�� kustomization.yaml

Ingress Configuration

kubernetes/apps/storage/rook-ceph-cluster/app/ingress-tailscale.yaml:

---
# Tailscale Ingress for Rook Ceph Dashboard
# Access via: https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
# Requires: Tailscale Operator installed in cluster
# TLS: Automatically provisioned via Let's Encrypt on first access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rook-ceph-dashboard-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: rook-ceph-mgr-dashboard
port:
number: 7000
tls:
- hosts:
- rook-ceph-dashboard

Kustomization

kubernetes/apps/storage/rook-ceph-cluster/app/kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./helmrelease.yaml
- ./ingress-tailscale.yaml

Verification

# Check Ingress
kubectl get ingress -n storage rook-ceph-dashboard-tailscale
NAME CLASS HOSTS ADDRESS PORTS AGE
rook-ceph-dashboard-tailscale tailscale * 80, 443 2m

# Check Tailscale proxy pod
kubectl get pods -n storage | grep tailscale
ts-rook-ceph-dashboard-tailscale-xyz 1/1 Running 0 2m

# Check service endpoints
kubectl get svc -n storage rook-ceph-mgr-dashboard
NAME TYPE CLUSTER-IP PORT(S) AGE
rook-ceph-mgr-dashboard ClusterIP 10.43.184.102 7000/TCP 5h

Access

From any device on your Tailscale network:

# Test DNS resolution
nslookup storage-rook-ceph-dashboard.<TAILNET>.ts.net

# Access dashboard
curl -I https://storage-rook-ceph-dashboard.<TAILNET>.ts.net

Browser: https://storage-rook-ceph-dashboard.<TAILNET>.ts.net

Hostname Pattern

Tailscale operator generates hostnames automatically:

Pattern: <namespace>-<ingress-name>.<tailnet>.ts.net

Examples:

NamespaceIngress NameFull Hostname
storagerook-ceph-dashboard-tailscalestorage-rook-ceph-dashboard.<TAILNET>.ts.net
monitoringgrafana-tailscalemonitoring-grafana.<TAILNET>.ts.net
defaultmyapp-tailscaledefault-myapp.<TAILNET>.ts.net

Tips:

  • Keep Ingress names short (hostname length limits)
  • Use meaningful names (hostname = <namespace>-<ingress-name>)
  • Avoid redundant words (e.g., grafana-tailscale not grafana-dashboard-tailscale)

Common Use Cases

Monitoring Dashboards

Grafana:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana-tailscale
namespace: monitoring
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: grafana
port:
number: 3000
tls:
- hosts:
- grafana

Access: https://monitoring-grafana.<TAILNET>.ts.net

Prometheus:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-tailscale
namespace: monitoring
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: prometheus-server
port:
number: 9090
tls:
- hosts:
- prometheus

Access: https://monitoring-prometheus.<TAILNET>.ts.net

GitOps Tools

ArgoCD:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-tailscale
namespace: argocd
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: argocd-server
port:
number: 443
tls:
- hosts:
- argocd

Access: https://argocd-argocd.<TAILNET>.ts.net

Flux Webhook:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flux-webhook-tailscale
namespace: flux-system
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: webhook-receiver
port:
number: 80
tls:
- hosts:
- flux-webhook

Access: https://flux-system-flux-webhook.<TAILNET>.ts.net

Development Tools

Code Server:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: code-server-tailscale
namespace: dev
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: code-server
port:
number: 8080
tls:
- hosts:
- code-server

Access: https://dev-code-server.<TAILNET>.ts.net

Advanced Configuration

Custom Annotations

Tailscale Hostname Override:

metadata:
name: my-app-tailscale
annotations:
tailscale.com/hostname: custom-name

Result: https://custom-name.<TAILNET>.ts.net

Funnel (Public Access):

metadata:
annotations:
tailscale.com/funnel: "true"

Exposes service publicly via Tailscale Funnel (requires Funnel enabled on tailnet)

Tags for ACLs:

metadata:
annotations:
tailscale.com/tags: "tag:k8s,tag:monitoring"

Applies Tailscale ACL tags to proxy device

Path-Based Routing

Multiple paths to same service:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-tailscale
spec:
ingressClassName: tailscale
rules:
- http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 8080
- path: /web
pathType: Prefix
backend:
service:
name: web-service
port:
number: 3000
tls:
- hosts:
- app

HTTPS Backend Services

Service using HTTPS:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: secure-app-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: secure-service
port:
number: 443 # HTTPS port
# OR
name: https # Port named 'https'
tls:
- hosts:
- secure-app

Tailscale operator detects HTTPS backend via:

  • Port number 443
  • Port name https

Comparison with Other Approaches

Tailscale Ingress vs Multi-Domain Setup

AspectTailscale IngressMulti-Domain (docs/notes/adding-a-2nd-domain.md)
Template ChangesNone requiredSchema + templates + DNS filters
DNS ManagementAutomatic (MagicDNS)Manual (external-dns + Cloudflare)
TLS CertificatesAutomatic (Let's Encrypt)Requires cert-manager configuration
Access ControlTailscale ACLsPublic or firewall rules
Use CaseInternal services, admin panelsPublic websites, customer-facing apps
Setup ComplexityLow (single Ingress resource)High (multiple template files)
SecurityPrivate by defaultPublic by default

When to Use Tailscale Ingress

 Use Tailscale Ingress for:

  • Internal dashboards (Grafana, Prometheus, Rook Ceph)
  • Admin interfaces (ArgoCD, Flux)
  • Development tools (code-server, Jupyter)
  • Database management (pgAdmin, phpMyAdmin)
  • Services requiring VPN-level security

L Don't use Tailscale Ingress for:

  • Public websites or APIs
  • Customer-facing applications
  • Services requiring anonymous access
  • High-throughput public services

Tailscale Ingress vs Gateway API

Tailscale Ingress (this document):

  • Uses standard Kubernetes Ingress API
  • Simple: single resource per service
  • Tailscale-managed TLS

Gateway API (https://tailscale.com/kb/1620/kubernetes-operator-byod-gateway-api):

  • Uses Kubernetes Gateway API (Envoy Gateway in this cluster)
  • Complex: Gateway + HTTPRoute + EnvoyProxy
  • Custom TLS via cert-manager
  • Integrates with existing envoy-external/envoy-internal gateways

Recommendation: Use Tailscale Ingress (this document) for simplicity. Use Gateway API approach if you need:

  • Integration with existing Gateway infrastructure
  • Custom DNS zones (via external-dns)
  • Advanced routing rules

Troubleshooting

Ingress Created But No Proxy Pod

Symptom: Ingress exists but no ts-* pod created

Diagnosis:

# Check Ingress events
kubectl describe ingress -n <namespace> <ingress-name>

# Check Tailscale operator logs
kubectl logs -n tailscale -l app=operator

# Verify IngressClass exists
kubectl get ingressclass tailscale

Common causes:

  • Tailscale operator not running
  • IngressClass tailscale not found
  • Invalid Ingress spec

Fix:

# Restart operator
kubectl rollout restart -n tailscale deployment/operator

# Verify operator permissions
kubectl get clusterrolebinding | grep tailscale

Certificate Not Provisioned

Symptom: HTTPS returns certificate error or timeout on first access

This is NORMAL behavior - See Step 6: First Access and Certificate Provisioning for details.

Diagnosis:

# Check proxy pod logs for certificate provisioning status
kubectl logs -n tailscale ts-<ingress-name>-<suffix> -f

# Look for these log messages (in order):
# cert("<service>.<TAILNET>.ts.net"): registered ACME account.
# cert("<service>.<TAILNET>.ts.net"): starting SetDNS call...
# cert("<service>.<TAILNET>.ts.net"): did SetDNS
# cert("<service>.<TAILNET>.ts.net"): requesting cert...
# cert("<service>.<TAILNET>.ts.net"): got cert

Expected timeline:

  1. 0-10 seconds: ACME account registration
  2. 10-40 seconds: DNS challenge setup and validation
  3. 40-60 seconds: Certificate issued
  4. 60+ seconds: Certificate cached, ready for connections

Common causes:

  • First HTTPS request (cert provisioning takes 30-60 seconds) - THIS IS NORMAL
  • HTTPS not enabled on tailnet
  • Let's Encrypt rate limits (5 certs per domain per week)
  • DNS challenge failing

Fix:

# 1. Wait 60 seconds after first HTTPS request, then refresh browser
# 2. Watch logs to confirm "got cert" message appears
kubectl logs -n tailscale ts-<ingress-name>-<suffix> | grep cert

# 3. If still failing, verify HTTPS is enabled
# Go to Tailscale admin console: Settings → HTTPS (must be ON)

# 4. Check for Let's Encrypt rate limit errors in logs
kubectl logs -n tailscale ts-<ingress-name>-<suffix> | grep -i "rate limit"

After successful provisioning:

  • All subsequent HTTPS requests are instant
  • Certificate auto-renews before expiration
  • No manual intervention needed

DNS Not Resolving

Symptom: nslookup or browser can't find hostname

Diagnosis:

# Check MagicDNS enabled
tailscale status | grep MagicDNS

# Verify device online
tailscale status | grep <namespace>-<ingress-name>

# Check from client device
ping <namespace>-<ingress-name>.<TAILNET>.ts.net

Common causes:

  • MagicDNS disabled on tailnet
  • Client device not connected to Tailscale
  • Proxy pod not registered

Fix:

  • Enable MagicDNS in Tailscale admin console: DNS � MagicDNS
  • Reconnect client device to Tailscale
  • Delete and recreate Ingress

Service Returns 502 Bad Gateway

Symptom: HTTPS connects but returns 502 error

Diagnosis:

# Verify backend service exists
kubectl get svc -n <namespace> <service-name>

# Check service endpoints
kubectl get endpoints -n <namespace> <service-name>

# Test service directly from cluster
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
curl http://<service-name>.<namespace>.svc.cluster.local:<port>

Common causes:

  • Service not running
  • Service has no endpoints (no pods)
  • Wrong port in Ingress

Fix:

  • Verify pods running: kubectl get pods -n <namespace>
  • Check service selector matches pod labels
  • Correct port number in Ingress spec

Proxy Pod Crash Loop

Symptom: ts-* pod in CrashLoopBackOff state

Diagnosis:

# Check pod events
kubectl describe pod -n <namespace> ts-<ingress-name>-<suffix>

# Check pod logs
kubectl logs -n <namespace> ts-<ingress-name>-<suffix>

# Verify Tailscale auth
kubectl get secret -n tailscale tailscale-auth

Common causes:

  • Tailscale auth key expired
  • Network policy blocking egress
  • Resource limits too low

Fix:

  • Regenerate Tailscale auth key
  • Check NetworkPolicy allows egress to Tailscale control plane
  • Increase resource requests/limits

Security Considerations

Access Control

Tailscale ACLs: Control which users/devices can access services

Example ACL (in Tailscale admin console):

{
"acls": [
{
"action": "accept",
"src": ["group:admins"],
"dst": ["tag:k8s:*"]
},
{
"action": "accept",
"src": ["group:developers"],
"dst": ["tag:k8s-dev:*"]
}
]
}

Tagging Ingresses:

metadata:
annotations:
tailscale.com/tags: "tag:k8s-admin"

Network Policies

Restrict proxy pod egress:

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tailscale-proxy-egress
namespace: storage
spec:
podSelector:
matchLabels:
app: tailscale
policyTypes:
- Egress
egress:
- to:
- podSelector: {} # Allow to pods in same namespace
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: rook-ceph-mgr-dashboard

Secret Management

Tailscale operator stores auth credentials in Secrets:

# View operator secrets (do not expose)
kubectl get secrets -n tailscale

# Rotate auth key
# 1. Generate new key in Tailscale admin console
# 2. Update Secret: kubectl edit secret -n tailscale tailscale-auth
# 3. Restart operator: kubectl rollout restart -n tailscale deployment/operator

Performance Considerations

Resource Usage

Each Tailscale Ingress creates a proxy pod:

  • CPU: ~50m (idle) to 200m (active)
  • Memory: ~50Mi (idle) to 100Mi (active)

Optimization:

  • Share Ingress for multiple paths (use rules: instead of multiple Ingresses)
  • Set resource limits to prevent resource exhaustion

Connection Limits

Tailscale proxy pods support:

  • Concurrent connections: ~1000 per pod
  • Throughput: ~1 Gbps per pod

For high-traffic services, use traditional LoadBalancer or Ingress approaches.

Pod Lifecycle

Proxy pod behavior:

  • Starts when Ingress created
  • Stops when Ingress deleted
  • Survives cluster restarts (Flux recreates)
  • Maintains Tailscale session across pod restarts

Cleanup

Remove Service from Tailscale

# Delete Ingress
kubectl delete ingress -n <namespace> <ingress-name>

# Verify proxy pod removed
kubectl get pods -n <namespace> | grep tailscale

# Remove from Git
rm kubernetes/apps/<namespace>/<app>/app/ingress-tailscale.yaml
# Edit kustomization.yaml to remove reference

# Commit and push
git add -A
git commit -m "chore: remove tailscale ingress for <app>"
git push
task reconcile

Note: Deleting Ingress automatically:

  • Deletes proxy pod
  • Removes device from Tailscale admin console
  • Removes MagicDNS entry

References

Appendix: Quick Reference

Create New Tailscale Ingress

# 1. Create ingress-tailscale.yaml
cat > kubernetes/apps/<namespace>/<app>/app/ingress-tailscale.yaml <<EOF
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: <app>-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: <service-name>
port:
number: <port>
tls:
- hosts:
- <app>
EOF

# 2. Add to kustomization.yaml
echo " - ./ingress-tailscale.yaml" >> kubernetes/apps/<namespace>/<app>/app/kustomization.yaml

# 3. Deploy
task configure
git add -A && git commit -m "feat: expose <app> via tailscale"
git push
task reconcile

# 4. Access
# URL: https://<namespace>-<app>.<TAILNET>.ts.net

Debug Commands

# Check Ingress status
kubectl get ingress -A | grep tailscale

# Check proxy pods
kubectl get pods -A | grep ts-

# Check operator logs
kubectl logs -n tailscale -l app=operator --tail=50 -f

# Test DNS
nslookup <namespace>-<ingress-name>.<TAILNET>.ts.net

# Test connectivity
curl -I https://<namespace>-<ingress-name>.<TAILNET>.ts.net

# Verify Tailscale status (from client)
tailscale status | grep <namespace>-<ingress-name>