Exposing Workloads Through Tailscale
Overview
This document describes how to securely expose Kubernetes services to your Tailscale network (tailnet) using the Tailscale Kubernetes Operator. This approach provides private access to cluster services without exposing them to the public internet or modifying cluster template configurations.
Goals
- Expose internal services (dashboards, admin panels) securely via Tailscale
- Automatic DNS via Tailscale MagicDNS
- Automatic TLS certificates from Let's Encrypt
- No changes to cluster templates or domain configuration
- Simple, reusable pattern for any service
Architecture
Components
Tailscale Kubernetes Operator
- Manages Tailscale proxy pods for exposed services
- Integrates with Kubernetes Ingress API
- Provisions TLS certificates automatically
- Registers services in Tailscale MagicDNS
Tailscale Ingress Pattern
- Uses standard Kubernetes Ingress resources
- Set
ingressClassName: tailscaleto trigger operator - Creates dedicated proxy pod per Ingress
- Proxy appears as device in Tailscale admin console
Traffic Flow
How It Works
- Ingress Creation: Create Kubernetes Ingress with
ingressClassName: tailscale - Operator Detection: Tailscale operator watches for new Ingress resources
- Proxy Deployment: Operator creates dedicated Tailscale proxy pod
- Tailnet Registration: Proxy registers as device in your Tailscale network
- DNS Registration: Service hostname added to MagicDNS (
<namespace>-<ingress-name>.<tailnet>.ts.net) - TLS Provisioning: Let's Encrypt certificate requested on first HTTPS connection
- Traffic Routing: Client � Tailscale network � Proxy pod � Service
Implementation
Prerequisites
- Tailscale Kubernetes Operator installed in cluster
- Tailscale account with MagicDNS enabled
- HTTPS enabled on tailnet (required for Let's Encrypt)
- Service to expose running in cluster
Step 1: Create Tailscale Ingress
Create ingress-tailscale.yaml in your application directory:
---
# Example: Rook Ceph Dashboard
# Access via: https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rook-ceph-dashboard-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: rook-ceph-mgr-dashboard
port:
number: 7000
tls:
- hosts:
- rook-ceph-dashboard
Key fields:
ingressClassName: tailscale- Triggers Tailscale operator (required)defaultBackend.service.name- Target service namedefaultBackend.service.port.number- Target service porttls.hosts- Hostname (operator appends namespace and tailnet domain)
Step 2: Add to Kustomization
Update app/kustomization.yaml:
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./helmrelease.yaml
- ./ingress-tailscale.yaml # Add this line
Step 3: Deploy
# Validate manifests
task configure
# Commit changes
git add -A
git commit -m "feat: expose service via tailscale ingress"
git push
# Apply to cluster
task reconcile
Step 4: Verify
# Check Ingress status
kubectl get ingress -n <namespace> <ingress-name>-tailscale
# Check for Tailscale proxy pod (may take 1-2 minutes)
kubectl get pods -n <namespace> | grep tailscale
# View proxy logs
kubectl logs -n <namespace> -l app.kubernetes.io/name=tailscale
# Verify in Tailscale admin console
# Navigate to Machines � Find "<namespace>-<ingress-name>-tailscale"
Step 5: Access Service
Access via Tailscale MagicDNS:
https://<namespace>-<ingress-name>.<TAILNET>.ts.net
Example (Rook Ceph Dashboard):
https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
Note: Replace <TAILNET> with your actual Tailscale network name (e.g., example-network.ts.net)
Step 6: First Access and Certificate Provisioning
IMPORTANT: The first HTTPS connection triggers automatic TLS certificate provisioning from Let's Encrypt. This process takes 30-60 seconds.
What happens on first access:
- Browser attempts HTTPS connection
- Tailscale proxy detects missing certificate
- Registers ACME account with Let's Encrypt
- Requests DNS-01 challenge
- Updates Tailscale DNS records
- Let's Encrypt validates domain ownership
- Certificate issued and cached
Expected behavior:
- First request may timeout or take 30-60 seconds to load
- Subsequent requests are instant (certificate cached)
Proxy logs during certificate provisioning:
# Watch certificate provisioning in real-time
kubectl logs -n tailscale ts-<ingress-name>-<suffix> -f
# You'll see logs like this:
cert("<service>.<TAILNET>.ts.net"): registered ACME account.
cert("<service>.<TAILNET>.ts.net"): starting SetDNS call...
cert("<service>.<TAILNET>.ts.net"): did SetDNS
cert("<service>.<TAILNET>.ts.net"): requesting cert...
cert("<service>.<TAILNET>.ts.net"): got cert
Troubleshooting first access:
If the browser shows an error on first access:
- Wait 60 seconds - Certificate provisioning takes time
- Refresh the page - Certificate may have been issued during timeout
- Check proxy logs - Look for "got cert" message
- Verify HTTPS enabled - Must be enabled in Tailscale admin console (Settings → HTTPS)
Common first-access errors:
| Error | Cause | Solution |
|---|---|---|
ERR_CONNECTION_TIMED_OUT | Certificate provisioning in progress | Wait 60 seconds, refresh page |
ERR_NAME_NOT_RESOLVED | MagicDNS not working | Reconnect Tailscale client, verify MagicDNS enabled |
SSL_ERROR_BAD_CERT | HTTPS not enabled on tailnet | Enable HTTPS in Tailscale admin console |
NET::ERR_CERT_COMMON_NAME_INVALID | Wrong hostname used | Use full hostname <service>.<TAILNET>.ts.net |
After successful first access:
- Certificate valid for 90 days
- Auto-renewed by Tailscale proxy
- All subsequent connections are instant
- Certificate stored in proxy pod memory
Real-World Example: Rook Ceph Dashboard
File Structure
kubernetes/apps/storage/rook-ceph-cluster/
�� ks.yaml
�� app/
�� helmrelease.yaml
�� ingress-tailscale.yaml # Tailscale ingress
�� kustomization.yaml
Ingress Configuration
kubernetes/apps/storage/rook-ceph-cluster/app/ingress-tailscale.yaml:
---
# Tailscale Ingress for Rook Ceph Dashboard
# Access via: https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
# Requires: Tailscale Operator installed in cluster
# TLS: Automatically provisioned via Let's Encrypt on first access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rook-ceph-dashboard-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: rook-ceph-mgr-dashboard
port:
number: 7000
tls:
- hosts:
- rook-ceph-dashboard
Kustomization
kubernetes/apps/storage/rook-ceph-cluster/app/kustomization.yaml:
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./helmrelease.yaml
- ./ingress-tailscale.yaml
Verification
# Check Ingress
kubectl get ingress -n storage rook-ceph-dashboard-tailscale
NAME CLASS HOSTS ADDRESS PORTS AGE
rook-ceph-dashboard-tailscale tailscale * 80, 443 2m
# Check Tailscale proxy pod
kubectl get pods -n storage | grep tailscale
ts-rook-ceph-dashboard-tailscale-xyz 1/1 Running 0 2m
# Check service endpoints
kubectl get svc -n storage rook-ceph-mgr-dashboard
NAME TYPE CLUSTER-IP PORT(S) AGE
rook-ceph-mgr-dashboard ClusterIP 10.43.184.102 7000/TCP 5h
Access
From any device on your Tailscale network:
# Test DNS resolution
nslookup storage-rook-ceph-dashboard.<TAILNET>.ts.net
# Access dashboard
curl -I https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
Browser: https://storage-rook-ceph-dashboard.<TAILNET>.ts.net
Hostname Pattern
Tailscale operator generates hostnames automatically:
Pattern: <namespace>-<ingress-name>.<tailnet>.ts.net
Examples:
| Namespace | Ingress Name | Full Hostname |
|---|---|---|
storage | rook-ceph-dashboard-tailscale | storage-rook-ceph-dashboard.<TAILNET>.ts.net |
monitoring | grafana-tailscale | monitoring-grafana.<TAILNET>.ts.net |
default | myapp-tailscale | default-myapp.<TAILNET>.ts.net |
Tips:
- Keep Ingress names short (hostname length limits)
- Use meaningful names (hostname =
<namespace>-<ingress-name>) - Avoid redundant words (e.g.,
grafana-tailscalenotgrafana-dashboard-tailscale)
Common Use Cases
Monitoring Dashboards
Grafana:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana-tailscale
namespace: monitoring
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: grafana
port:
number: 3000
tls:
- hosts:
- grafana
Access: https://monitoring-grafana.<TAILNET>.ts.net
Prometheus:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-tailscale
namespace: monitoring
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: prometheus-server
port:
number: 9090
tls:
- hosts:
- prometheus
Access: https://monitoring-prometheus.<TAILNET>.ts.net
GitOps Tools
ArgoCD:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-tailscale
namespace: argocd
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: argocd-server
port:
number: 443
tls:
- hosts:
- argocd
Access: https://argocd-argocd.<TAILNET>.ts.net
Flux Webhook:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flux-webhook-tailscale
namespace: flux-system
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: webhook-receiver
port:
number: 80
tls:
- hosts:
- flux-webhook
Access: https://flux-system-flux-webhook.<TAILNET>.ts.net
Development Tools
Code Server:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: code-server-tailscale
namespace: dev
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: code-server
port:
number: 8080
tls:
- hosts:
- code-server
Access: https://dev-code-server.<TAILNET>.ts.net
Advanced Configuration
Custom Annotations
Tailscale Hostname Override:
metadata:
name: my-app-tailscale
annotations:
tailscale.com/hostname: custom-name
Result: https://custom-name.<TAILNET>.ts.net
Funnel (Public Access):
metadata:
annotations:
tailscale.com/funnel: "true"
Exposes service publicly via Tailscale Funnel (requires Funnel enabled on tailnet)
Tags for ACLs:
metadata:
annotations:
tailscale.com/tags: "tag:k8s,tag:monitoring"
Applies Tailscale ACL tags to proxy device
Path-Based Routing
Multiple paths to same service:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-tailscale
spec:
ingressClassName: tailscale
rules:
- http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 8080
- path: /web
pathType: Prefix
backend:
service:
name: web-service
port:
number: 3000
tls:
- hosts:
- app
HTTPS Backend Services
Service using HTTPS:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: secure-app-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: secure-service
port:
number: 443 # HTTPS port
# OR
name: https # Port named 'https'
tls:
- hosts:
- secure-app
Tailscale operator detects HTTPS backend via:
- Port number
443 - Port name
https
Comparison with Other Approaches
Tailscale Ingress vs Multi-Domain Setup
| Aspect | Tailscale Ingress | Multi-Domain (docs/notes/adding-a-2nd-domain.md) |
|---|---|---|
| Template Changes | None required | Schema + templates + DNS filters |
| DNS Management | Automatic (MagicDNS) | Manual (external-dns + Cloudflare) |
| TLS Certificates | Automatic (Let's Encrypt) | Requires cert-manager configuration |
| Access Control | Tailscale ACLs | Public or firewall rules |
| Use Case | Internal services, admin panels | Public websites, customer-facing apps |
| Setup Complexity | Low (single Ingress resource) | High (multiple template files) |
| Security | Private by default | Public by default |
When to Use Tailscale Ingress
Use Tailscale Ingress for:
- Internal dashboards (Grafana, Prometheus, Rook Ceph)
- Admin interfaces (ArgoCD, Flux)
- Development tools (code-server, Jupyter)
- Database management (pgAdmin, phpMyAdmin)
- Services requiring VPN-level security
L Don't use Tailscale Ingress for:
- Public websites or APIs
- Customer-facing applications
- Services requiring anonymous access
- High-throughput public services
Tailscale Ingress vs Gateway API
Tailscale Ingress (this document):
- Uses standard Kubernetes Ingress API
- Simple: single resource per service
- Tailscale-managed TLS
Gateway API (https://tailscale.com/kb/1620/kubernetes-operator-byod-gateway-api):
- Uses Kubernetes Gateway API (Envoy Gateway in this cluster)
- Complex: Gateway + HTTPRoute + EnvoyProxy
- Custom TLS via cert-manager
- Integrates with existing envoy-external/envoy-internal gateways
Recommendation: Use Tailscale Ingress (this document) for simplicity. Use Gateway API approach if you need:
- Integration with existing Gateway infrastructure
- Custom DNS zones (via external-dns)
- Advanced routing rules
Troubleshooting
Ingress Created But No Proxy Pod
Symptom: Ingress exists but no ts-* pod created
Diagnosis:
# Check Ingress events
kubectl describe ingress -n <namespace> <ingress-name>
# Check Tailscale operator logs
kubectl logs -n tailscale -l app=operator
# Verify IngressClass exists
kubectl get ingressclass tailscale
Common causes:
- Tailscale operator not running
- IngressClass
tailscalenot found - Invalid Ingress spec
Fix:
# Restart operator
kubectl rollout restart -n tailscale deployment/operator
# Verify operator permissions
kubectl get clusterrolebinding | grep tailscale
Certificate Not Provisioned
Symptom: HTTPS returns certificate error or timeout on first access
This is NORMAL behavior - See Step 6: First Access and Certificate Provisioning for details.
Diagnosis:
# Check proxy pod logs for certificate provisioning status
kubectl logs -n tailscale ts-<ingress-name>-<suffix> -f
# Look for these log messages (in order):
# cert("<service>.<TAILNET>.ts.net"): registered ACME account.
# cert("<service>.<TAILNET>.ts.net"): starting SetDNS call...
# cert("<service>.<TAILNET>.ts.net"): did SetDNS
# cert("<service>.<TAILNET>.ts.net"): requesting cert...
# cert("<service>.<TAILNET>.ts.net"): got cert
Expected timeline:
- 0-10 seconds: ACME account registration
- 10-40 seconds: DNS challenge setup and validation
- 40-60 seconds: Certificate issued
- 60+ seconds: Certificate cached, ready for connections
Common causes:
- First HTTPS request (cert provisioning takes 30-60 seconds) - THIS IS NORMAL
- HTTPS not enabled on tailnet
- Let's Encrypt rate limits (5 certs per domain per week)
- DNS challenge failing
Fix:
# 1. Wait 60 seconds after first HTTPS request, then refresh browser
# 2. Watch logs to confirm "got cert" message appears
kubectl logs -n tailscale ts-<ingress-name>-<suffix> | grep cert
# 3. If still failing, verify HTTPS is enabled
# Go to Tailscale admin console: Settings → HTTPS (must be ON)
# 4. Check for Let's Encrypt rate limit errors in logs
kubectl logs -n tailscale ts-<ingress-name>-<suffix> | grep -i "rate limit"
After successful provisioning:
- All subsequent HTTPS requests are instant
- Certificate auto-renews before expiration
- No manual intervention needed
DNS Not Resolving
Symptom: nslookup or browser can't find hostname
Diagnosis:
# Check MagicDNS enabled
tailscale status | grep MagicDNS
# Verify device online
tailscale status | grep <namespace>-<ingress-name>
# Check from client device
ping <namespace>-<ingress-name>.<TAILNET>.ts.net
Common causes:
- MagicDNS disabled on tailnet
- Client device not connected to Tailscale
- Proxy pod not registered
Fix:
- Enable MagicDNS in Tailscale admin console: DNS � MagicDNS
- Reconnect client device to Tailscale
- Delete and recreate Ingress
Service Returns 502 Bad Gateway
Symptom: HTTPS connects but returns 502 error
Diagnosis:
# Verify backend service exists
kubectl get svc -n <namespace> <service-name>
# Check service endpoints
kubectl get endpoints -n <namespace> <service-name>
# Test service directly from cluster
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
curl http://<service-name>.<namespace>.svc.cluster.local:<port>
Common causes:
- Service not running
- Service has no endpoints (no pods)
- Wrong port in Ingress
Fix:
- Verify pods running:
kubectl get pods -n <namespace> - Check service selector matches pod labels
- Correct port number in Ingress spec
Proxy Pod Crash Loop
Symptom: ts-* pod in CrashLoopBackOff state
Diagnosis:
# Check pod events
kubectl describe pod -n <namespace> ts-<ingress-name>-<suffix>
# Check pod logs
kubectl logs -n <namespace> ts-<ingress-name>-<suffix>
# Verify Tailscale auth
kubectl get secret -n tailscale tailscale-auth
Common causes:
- Tailscale auth key expired
- Network policy blocking egress
- Resource limits too low
Fix:
- Regenerate Tailscale auth key
- Check NetworkPolicy allows egress to Tailscale control plane
- Increase resource requests/limits
Security Considerations
Access Control
Tailscale ACLs: Control which users/devices can access services
Example ACL (in Tailscale admin console):
{
"acls": [
{
"action": "accept",
"src": ["group:admins"],
"dst": ["tag:k8s:*"]
},
{
"action": "accept",
"src": ["group:developers"],
"dst": ["tag:k8s-dev:*"]
}
]
}
Tagging Ingresses:
metadata:
annotations:
tailscale.com/tags: "tag:k8s-admin"
Network Policies
Restrict proxy pod egress:
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tailscale-proxy-egress
namespace: storage
spec:
podSelector:
matchLabels:
app: tailscale
policyTypes:
- Egress
egress:
- to:
- podSelector: {} # Allow to pods in same namespace
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: rook-ceph-mgr-dashboard
Secret Management
Tailscale operator stores auth credentials in Secrets:
# View operator secrets (do not expose)
kubectl get secrets -n tailscale
# Rotate auth key
# 1. Generate new key in Tailscale admin console
# 2. Update Secret: kubectl edit secret -n tailscale tailscale-auth
# 3. Restart operator: kubectl rollout restart -n tailscale deployment/operator
Performance Considerations
Resource Usage
Each Tailscale Ingress creates a proxy pod:
- CPU: ~50m (idle) to 200m (active)
- Memory: ~50Mi (idle) to 100Mi (active)
Optimization:
- Share Ingress for multiple paths (use
rules:instead of multiple Ingresses) - Set resource limits to prevent resource exhaustion
Connection Limits
Tailscale proxy pods support:
- Concurrent connections: ~1000 per pod
- Throughput: ~1 Gbps per pod
For high-traffic services, use traditional LoadBalancer or Ingress approaches.
Pod Lifecycle
Proxy pod behavior:
- Starts when Ingress created
- Stops when Ingress deleted
- Survives cluster restarts (Flux recreates)
- Maintains Tailscale session across pod restarts
Cleanup
Remove Service from Tailscale
# Delete Ingress
kubectl delete ingress -n <namespace> <ingress-name>
# Verify proxy pod removed
kubectl get pods -n <namespace> | grep tailscale
# Remove from Git
rm kubernetes/apps/<namespace>/<app>/app/ingress-tailscale.yaml
# Edit kustomization.yaml to remove reference
# Commit and push
git add -A
git commit -m "chore: remove tailscale ingress for <app>"
git push
task reconcile
Note: Deleting Ingress automatically:
- Deletes proxy pod
- Removes device from Tailscale admin console
- Removes MagicDNS entry
References
- Tailscale on Kubernetes - Overview of Tailscale Kubernetes integration
- Kubernetes Operator Cluster Ingress - Official Ingress documentation
- Tailscale Kubernetes Operator - Operator installation and configuration
- MagicDNS - Tailscale DNS configuration
- Tailscale ACLs - Access control configuration
- Let's Encrypt with Tailscale - HTTPS certificate setup
Appendix: Quick Reference
Create New Tailscale Ingress
# 1. Create ingress-tailscale.yaml
cat > kubernetes/apps/<namespace>/<app>/app/ingress-tailscale.yaml <<EOF
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: <app>-tailscale
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: <service-name>
port:
number: <port>
tls:
- hosts:
- <app>
EOF
# 2. Add to kustomization.yaml
echo " - ./ingress-tailscale.yaml" >> kubernetes/apps/<namespace>/<app>/app/kustomization.yaml
# 3. Deploy
task configure
git add -A && git commit -m "feat: expose <app> via tailscale"
git push
task reconcile
# 4. Access
# URL: https://<namespace>-<app>.<TAILNET>.ts.net
Debug Commands
# Check Ingress status
kubectl get ingress -A | grep tailscale
# Check proxy pods
kubectl get pods -A | grep ts-
# Check operator logs
kubectl logs -n tailscale -l app=operator --tail=50 -f
# Test DNS
nslookup <namespace>-<ingress-name>.<TAILNET>.ts.net
# Test connectivity
curl -I https://<namespace>-<ingress-name>.<TAILNET>.ts.net
# Verify Tailscale status (from client)
tailscale status | grep <namespace>-<ingress-name>