wiki.getshifting.com

— Sjoerd Hooft's InFormation Technology —

User Tools

Site Tools


scalewayprivatenetworkwithk8s

Private Kubernetes on Scaleway

Summary: This wiki page shows how to create a private kubernetes cluster in Scaleway, one of the upcoming European Cloud Providers. It is a followup of Getting Started on Scaleway.
Date: 14 June 2026
Update: 27 June 2026: Added next steps with terraform

Let's first review the network diagram of what we are going to create:

Network diagram for Scaleway Private Network with K8s


When we review the network diagram we see immediately that one resource is not inside the private network. Unfortunately, Scaleway does not have an private network offering for the container registry.

Note: Scaleway icons for drawio can be downloaded here

We will now implement this solution manually from the Scaleway console.

Create Project

  • Go to Organization Dashboard → Projects → Create Project
    • Name: “Private K8S”

Create VPC

VPC allows you to build your own Virtual Private Cloud on top of Scaleway's shared public cloud.

Now, create the VPC:

  • Go to Products → Network → VPC → Create VPC
    • Region: “AMSTERDAM”
    • Name: “vpc-ams-private-k8s”

Create Private Network

Note: Create Private Networks for your resources, allowing them to communicate on a secure network without needing a public IP address. There are no subnets; Max network size is /20 (4094 addresses); Max 8 Private networks per VPC; Max 512 resources per Private network

Now, create our private networks:

  • In the created VPC → Private networks → Create Private Network
    • Name: “pnet-ams-pgw”
    • Advanced settings
      • CIDR range: “10.10.1.0/24”
  • In the created VPC → Private networks → Create Private Network
    • Name: “pnet-ams-vms”
    • Advanced settings
      • CIDR range: “10.10.2.0/24”
      • Receive all default routes: “enabled”
  • In the created VPC → Private networks → Create Private Network
    • Name: “pnet-ams-k8s”
    • Advanced settings
      • CIDR range: “10.10.16.0/20”
Note: Receive all default routes were not enabled, I had to enable this setting again after creation.

Create VM

Reserve an IP Address

  • Go to Network → IPAM → Reserve private IP
    • Select the region, VPC and Private Network
    • IP address to reserve → Specific IP address: “10.10.2.2”

Create an instance

  • Go to Compute → CPU & GPU Instances → Create CPU Instance
    • Set the Region to “Amsterdam” and the vCPU type to “Shared”
    • Select the instance type “BASIC3-X2C-4G”, and choose AMS1 as the specific region to create the instance
      • Name: “vm-jumpbox”
      • OS Image: Fedora 44
      • Keep default volume 10 GB
      • Configure metwork and security:
        • Note: Do not yet select the “Attach to Private Network” option as we can't select the reserved IP address in the instance creation wizard
        • Deselect the public IP addresses for IPv4 and IPv6
      • SSH Key: Add your SSH key. Give it a descriptive name and add the public key. This can be used to logon to the VM later.
  • Go to Network → VPC → Private Networks → Select the private network → Attach resource
    • Type of resource: instance
    • Resource: “vm-jumpbox”
    • Private IPv4 Address → Reserved IPs: “10.10.2.2/24”

Create Public Gateway

  • Go to Network → Public Gateways → Create Public Gateway
  • Availability Zone: Amsterdam 1
  • Size: VPC-GW-S (100 Mbps)
  • IP: Allocate a new IP
  • Name: “pgw-ams”
  • Go to Network → Public Gateways → Select Public Gateway → Private Networks → Attach to a Private Network
    • Select the private network “pnet-ams-pgw”
    • Private IPv4 Address: Auto-Allocate
    • Advertise default route: Enabled
Repeat these steps for the “pnet-ams-k8s” private network. This is a requirement for the full isolation mode for k8s.

Enable SSH Bastion

SSH bastion works as a single entry point to other resources behind your Public Gateway. This allows easier and more secure management of your infrastructure and SSH keys

Enable the SSH bastion for the public gateway:

  • Go to Network → Public Gateways → Select Public Gateway → SSH Bastion → Enable SSH Bastion
    • Port: 61000
  • Still under the Public Gateway → SSH Bastion → Allowed IPs
    • Add your personal public IP address to prevent unallowed access
      • 84.104.50.96/32
    • After adding an IP address you get an notification that the 0.0.0.0/0 entry is still in the list. Confirm to delete it.

When the gateway is created it should now be possible to access the linux jumpbox. From a commandline issue ssh -J bastion@51.15.70.204:61000 root@10.10.2.2, and if you have added your ssh private key to your local ssh agent you'll be logged in immediately.

Create Kapsule

Scaleway offers two Kubernetes implementations, Kapsule and Kosmos. As Kapsule is the only one that supports private networks, we'll use that one.

  • Go to Containers → Kubernetes → Set Region to AMS → Create cluster
    • Cluster type: Kubernetes Kapsule
    • Region: Amsterdam
    • Control plane offer: Free
    • Kubernetes version: 1.35.3 (latest version at the moment)
    • Name: “k8s-ams”
    • Private network: pnet-ams-k8s
  • Pools:
    • Zone: Amsterdam 1
    • Node type: BASIC3-X2C-4G (2 vCPU / 4 GB, 350 Mbps, x86 arch)
      • Note: These are very small nodes, but we are just checking the options for private networking in scaleway
    • System volume: 5K - 20 GB
    • Autoscale: Disabled
      • Number of nodes: 1
    • Autoheal: On
    • Full isolation: On
      • Note: Public IPs will be disabled and your worker nodes will have only private IPs. To route outgoing traffic with a single output, a Public Gateway is required.
        • Public Gateway is required to enable this feature
          • Make sure your Public Gateway uses IPAM mode (the pgw-ams is assigned a IPAM address automatically)
          • Dynamic NAT must be activated (by default on)
          • Advertise DefaultRoute must be activated (by default on)
    • Name:“pool-ams1”

Connect to Kapsule

Before continuing, make sure the “pool-ams1” is ready.

  • Disable access to the control plane by going to the just created cluster → Network → Allowed IPs for control plane → Remove the default 0.0.0.0/0 entry, and
    • Add your personal public IP address to prevent unallowed access
      • 84.104.50.96/32
    • Add the public IP address from the API gateway
      • 51.15.70.204/32
  • Get kubeconfig from the cluster overview page, through the “Action” button
    • Generate an API key. Make sure to give it an good description such as “Sjoerd - Private K8S test June 2026” and set the API key expiration.
    • Generate the kubeconfig and save in “C:\Users<username>.kube\kubeconfig-scw-k8s-ams.yaml”
  • Now using Lens, you can right-click on Local Kubeconfigs and click on Add Kubeconfigs → From filesystem and navigate to the kubeconfig you've just added.

Alternatively, you could use the linux jumpbox we've created:

# Download the latest stable release of kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
# Install kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# Create a kubeconfig directory and file
mkdir .kube
cd .kube/
vi config
# Make sure to add the content of the kubeconfig you downloaded
export KUBECONFIG=/root/.kube/config
# Verify the installation
kubectl version
kubectl cluster-info
kubectl get namespaces

Create Container Registry

  • Go to Containers → Container Registry → Select AMS → Create Namespace
    • Name: “cr-ams”
    • Region: Amsterdam
    • Privacy Policy: Private

After creation, you can see the registry endpoint from the Container Registry namespace overview: rg.nl-ams.scw.cloud/cr-ams. Now we can add an image to the registry so that we can do a pod deployment on the cluster.

Push Images

Note: When working with docker commands on Windows make sure to have docker desktop started.

To push images, we need an API key. We can use the same as we used for the kubeconfig generation. Note that the container registry is not part of the private network, so we need to use the public endpoint:

# Use API key as the SCW_SECRET_KEY
echo <SCW_SECRET_KEY> | docker login rg.nl-ams.scw.cloud/cr-ams -u nologin --password-stdin
docker pull docker.io/library/alpine:3.24.0
docker tag docker.io/library/alpine:3.24.0 rg.nl-ams.scw.cloud/cr-ams/alpine:3.24.0
docker push rg.nl-ams.scw.cloud/cr-ams/alpine:3.24.0

Test a K8S Deployment with Container Registry

To test our cluster we can simply create a pod manifest which uses the just uploaded image from the container registry. After that we can logon to the pod to verify the network config.

I usually use an alpine image, for which I've created a custom manifest used for basic testing:

kind: Pod
apiVersion: v1
metadata:
  name:  test-alpine-busybox-curl
spec:
  restartPolicy: Never
  containers:
    - name: alpine
      image: rg.nl-ams.scw.cloud/cr-ams/alpine:3.24.0
      command:
        - /bin/sh
      args:
        - "-c"
        - "-e"
        - |-
          apk update
          apk add --no-cache curl busybox-extras bash
          echo ""
          echo "Example commands to use:"
          echo "Verify outgoing IP address: curl -s https://ipinfo.io/ip"
          echo "Check DNS resolution:       nslookup example.com"
          sleep infinity
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 250m
          memory: 256Mi

Now we'll login to the cluster and make sure we can apply the manifest. I'll be using the powershell prompt available from within Lens:

# Create namespace
kubectl create ns shift
# Create an Image Pull Secret - use the same API again!
$SCW_SECRET_KEY = "SCW_SECRET_KEY"
kubectl create secret docker-registry registry-secret --docker-server=rg.nl-ams.scw.cloud --docker-username=cr-ams --docker-password=$SCW_SECRET_KEY -n shift
# Start the pod
kubectl apply -f .\alpine.yaml -n shift
# Show logs
kubectl logs test-alpine-busybox-curl -n shift
# Login to the pod
kubectl exec -it test-alpine-busybox-curl -n shift -- /bin/sh
# From within the pod
# Check for outgoing traffic, this will show the IP address from the public gateway
curl -s https://ipinfo.io/ip
# output
51.15.70.204
# Check the local IP address
ip addr
# output
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 4e:84:ee:c2:d0:30 brd ff:ff:ff:ff:ff:ff
    inet 100.64.1.17/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::4c84:eeff:fec2:d030/64 scope link
       valid_lft forever preferred_lft forever

From the ip addr command you'd see that the pod has an IP address from another range. That is set in the cillium-config configmap:

kubectl get configmap cilium-config -n kube-system -o jsonpath='{.data.cluster-pool-ipv4-cidr}'
# output
100.64.0.0/15

So the whole range is “100.64.0.1 - 100.65.255.254”. You can also check the network settings for the nodes:

kubectl get ciliumnodes -A
# output
NAME                                             CILIUMINTERNALIP   INTERNALIP   AGE
scw-k8s-ams-pool-ams1-x86-a22123616e534799b7eb   100.64.1.20        10.10.16.4   17m

When checking the services, we can see that they are in another network range:

kubectl get services -A
# output
NAMESPACE     NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes       ClusterIP   10.32.0.1      <none>        443/TCP                  27h
kube-system   coredns          ClusterIP   10.32.0.10     <none>        53/UDP,53/TCP,9153/TCP   27h
kube-system   metrics-server   ClusterIP   10.32.10.127   <none>        443/TCP                  27h
# This is the default option, so it doesn't show in any config:
# Check the cilium-config configmap
kubectl get configmap cilium-config -n kube-system -o jsonpath='{.data.cluster-pool-ipv4-service-cidr}'
# Check the cluster dump
kubectl cluster-info dump | sls service-range
# output
time=2026-06-13T14:34:57.382969308Z level=info msg="  --ipv4-service-range='auto'"
time=2026-06-13T14:34:57.382984658Z level=info msg="  --ipv6-service-range='auto'"

So you see that because the setting is default, you can't actually see it. This can only be set during the creation of the cluster using the --service-cluster-ip-range, but from the console that option was not available.

Overall Conclusion

Even though Scaleway tried their best, it's quite clear that even the “full isolation” mode does not completely make the cluster private. When creating the cluster lots of options to configure the network are missing. So, I'd would say, it's a good starting point but it's definitely not there yet.

Next Steps

Even though it was fun doing things over the console, let's now use terraform to deploy the same resources: Private Kubernetes on Scaleway with Terraform

This wiki has been made possible by:

scalewayprivatenetworkwithk8s.txt · Last modified: by 127.0.0.1