This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Getting started

Getting started with Elastx Kubernetes CaaS

1: Cluster configuration
2: Order a new cluster
3: Accessing your cluster
4: Cluster upgrades
5: Your first deployment
6: Recommendations

1 - Cluster configuration

Cluster configuration and optional features

There are a lot of options possible for your cluster. Most options have a sane default however could be overridden on request.

A default cluster comes with 3 controlplane and 3 worker nodes. To connect all nodes we create a network, default (10.128.0.0/22). We also deploy monitoring to ensure functionality of all cluster components. However most things are just a default and could be overridden.

Common options

Nodes

The standard configuration consists of the following:

Three control plane nodes, one in each of our availability zones. Flavor: v2-c2-m8-d80
Three worker nodes, one in each of our availability zones, in a single nodegroup. Flavor: v2-c2-m8-d80

Minimal configuration

Three control plane nodes, one in each of our availability zones. Flavor: v2-c2-m8-d80
One worker node, Flavor: v2-c2-m8-d80

This is the minimal configuration offered. Scaling to larger flavors and adding nodes are supported. Autoscaling is not supported with a single worker node.

Note: SLA is different for minimal configuration type of cluster. SLA’s can be found here.

Nodegroups and multiple flavors

To try keep node management as easy as possible we make use of nodegroups. A nodegroup contains of one or multiple nodes with one flavor and a list of availability zones to deploy nodes in. Clusters are default deliverd with a nodegroup called workers containing 3 nodes one in each AZ. A nodegroup is limited to one flavor meaning all nodes in the nodegroup will have the same amount of CPU, RAM and disk.

You could have multiple nodegroups, if you for example want to target workload on separate nodes or in case you wish to consume multiple flavors.

A few examples of nodegroups:

Name	Flavour	AZ list	Min node count	Max node count (autoscaling)
worker	v2-c2-m8-d80	STO1, STO2, STO3	3	0
database	d2-c8-m120-d1.6k	STO1, STO2, STO3	3	0
frontend	v2-c4-m16-d160	STO1, STO2, STO3	3	12
jobs	v2-c4-m16-d160	STO1	1	3

In the examples we could see worker our default nodegroup and an example of having separate nodes for databases and frontend where the database is running on dedicated nodes and the frontend is running on smaller nodes but can autoscale between 3 and 12 nodes based on current cluster request. We also have a jobs nodegroup where we have one node in sto1 but can scale up to 3 nodes where all are placed inside STO1. You can read more about autoscaling here.

Nodegroups can be changed at any time. Please also note that we have auto-healing meaning in case any of your nodes for any reason stops working we will replace them. More about autohealing could be found here

Network

By default we create a cluster network (10.128.0.0/22). However we could use another subnet per customer request. The most common scenario is when customer request another subnet is when exposing multiple Kubernetes clusters over a VPN.

Please make sure to inform us if you wish to use a custom subnet during the ordering process since we cannot replace the network after creation, meaning we would then need to recreate your entire cluster.

We currently only support cidr in the 10.0.0.0/8 subnet range and at least a /24. Both nodes and loadbalancers are using IPs for this range meaning you need to have a sizable network from the beginning.

Cluster domain

We default all clusters to “cluster.local”. This is simular to most other providers. If you wish to have another cluster domain please let us know during the ordering procedure since it cannot be replaced after cluster creation.

Worker nodes Floating IPs

By default, our clusters come with nodes that do not have any Floating IPs attached to them. If, for any reason, you require Floating IPs on your workload nodes, please inform us, and we can configure your cluster accordingly. It’s worth noting that the most common use case for Floating IPs is to ensure predictable source IPs. However, please note that enabling or disabling Floating IPs will necessitate the recreation of all your nodes, one by one but could be enabled or disabled at any time.

Since during upgrades we create a new node prior to removing an old node you would need to have an additional IP adress on standby. If you wish we to preallocate a list or range of IP adresses just mention this and we will configure your cluster accordingly.

Please know that only worker nodes are consume IP adresses meaning controlplane nodes does not make use of Floating IPs.

Less common options

OIDC

If you wish to integrate with your existing OIDC compatible IDP, example Microsoft AD And Google Workspace that is supported directy in the kubernetes api service.

By default we ship clusters with this option disabled however if you wish to make use of OIDC just let us know when order the cluster or afterwards. OIDC can be enabled, disabled or changed at any time.

Cluster add-ons

We currently offer managed cert-manager, NGINX Ingress and elx-nodegroup-controller.

Cert-manager

Cert-manager (link to cert-manager.io) helps you to manage TLS certificates. A common use case is to use lets-encrypt to “automatically” generate certificates for web apps. However the functionality goes much deeper. We also have usage instructions and have a guide if you wish to deploy cert-manager yourself.

Ingress

An ingress controller in a Kubernetes cluster manages how external traffic reaches your services. It routes requests based on rules, handles load balancing, and can integrate with cert-manager to manage TLS certificates. This simplifies traffic handling and improves scalability and security compared to exposing each service individually. We have a usage guide with examples that can be found here.

We have chosen to use ingress-nginx and to support ingress, we limit what custom configurations can be made per cluster. We offer two “modes”. One that we call direct mode, which is the default behavior. This mode is used when end-clients connect directly to your ingress. We also have a proxy mode for when a proxy (e.g., WAF) is used in front of your ingress. When running in proxy mode, we also have the ability to limit traffic from specific IP addresses, which we recommend doing for security reasons. If you are unsure which mode to use or how to handle IP whitelisting, just let us know and we will help you choose the best options for your use case.

If you are interested in removing any limitations, we’ve assembled guides with everything you need to install the same IngressController as we provide. This will give you full control. The various resources give configuration examples and instructions for lifecycle management. These can be found here.

elx-nodegroup-controller

The nodegroup controller is useful when customers want to use custom taints or labels on their nodes. It supports matching nodes based on nodegroup or by name. The controller can be found on Github if you wish to inspect the code or deploy it yourself.

2 - Order a new cluster

How to order a new cluster

How to order or remove a cluster

Ordering and scaling of clusters is currently a manual process involving contact with either our sales department or our support. This is a known limitation, but may change in the future.

3 - Accessing your cluster

How to access your cluster

In order to access your cluster there are a couple of things you need to do. First you need to make sure you have the correct tools installed, the default client for interacting with Kubernetes clusters is called kubectl. Instructions for installing it on your system can be found by following the link.

You may of course use any Kubernetes client you wish to access your cluster however setting up other clients is beyond the scope of this documentation.

Credentials (kubeconfig)

Once you have a client you can use to access the cluster you will need to fetch the credentials for you cluster. You can find the credentials for your cluster by logging in to Elastx OpenStack IaaS. When logged in you can find the kubeconfig file for your cluster by clicking on the “Object Storage” menu option in the left-hand side menu. And then click on “Containers”, you should now see a container with the same name as your cluster (clusters are named “customer-cluster_name”). Clicking on the container should reveal a file called admin.conf in the right-hand pane. Click on the `Download" button to the right of the file name to download it to your computer.

NOTE These credentials will be rotated when your cluster is upgraded so you should periodically fetch new credentials to make sure you have a fresh set.

NOTE The kubeconfig you just downloaded has full administrator privileges.

Configuring kubectl to use your credentials

In order for kubectl to be able to use the credentials you just downloaded you need to either place the credentials in the default location or otherwise configure kubectl to utilize them. The official documentation covers this process in detail.

Verify access

To verify you’ve got access to the cluster you can run something like this:

$ kubectl get nodes
NAME                           STATUS   ROLES           AGE   VERSION
hux-lab1-control-plane-c9bmm   Ready    control-plane   14h   v1.27.3
hux-lab1-control-plane-j5p42   Ready    control-plane   14h   v1.27.3
hux-lab1-control-plane-wlwr8   Ready    control-plane   14h   v1.27.3
hux-lab1-worker-447sn          Ready    <none>          13h   v1.27.3
hux-lab1-worker-9ltbp          Ready    <none>          14h   v1.27.3
hux-lab1-worker-vszmc          Ready    <none>          14h   v1.27.3

If your output looks similar then you should be good to go! If it looks very different or contains error messages, don’t hesitate to contact our support if you can’t figure out how to solve it on your own.

Restrict access

Access to the API server is controlled in the loadbalancer in front of the API. Currently, managing the IP-range allowlist requires a support ticket here. All Elastx IP ranges are always included.

Instructions for older versions

Everything under this section is only for clusters running older versions of our private Kubernetes service.

Security groups

Note: This part only applies to clusters not already running Private Kubernetes 2.0 or later.

If your cluster was created prior to Kubernetes 1.26 or when we specifically informed you that this part applies.

If you are not sure if this part applies, you can validate it by checking if there is a security group called cluster-name-master-customer in your openstack project.

To do so, log in to Elastx Openstack IaaS. When logged in click on the “Network” menu option in the left-hand side menu. Then click on “Security Groups”, finally click on the “Manage Rules” button to the right of the security group named cluster-name-master-customer. To add a rule click on the “Add Rule” button.

For example, to allow access from the ip address 1.2.3.4 configure the rule as follows:

Rule: Custom TCP Rule
Direction: Ingress
Open Port: Port
Port: 6443
Remote: CIDR
CIDR: 1.2.3.4/32

Once you’ve set up rules that allow you to access your cluster you are ready to verify access.

4 - Cluster upgrades

How cluster upgrades are managed

Introduction

Kubernetes versions are released approximately three times a year, introducing enhancements, security updates, and bug fixes. The planning and initiation of a cluster upgrade is a manual task that requires coordination with our customers.

To schedule the upgrade of your cluster(s), we require a designated point of contact for coordination.
For customers with multiple clusters, please provide your preferred sequence and timeline for upgrades. If you haven’t shared this information yet, kindly submit a support ticket with these details.

Upgrade Planning

Upgrades are scheduled in consultation with the customer and can be done on at the initiative of either Elastx or the customer. If the customer does not initiate the planning of an upgrade, we will reach out to the designated contact in a support ticket at least twice a year with suggested upgrade dates.

NOTE: Upgrades are not performed during our changestop periods:

In general the full month of July and through the first week of August

December 23rd to January 2nd

Before scheduling and confirming a time slot, please review the relevant changelog and the Kubernetes Deprecated API Migration guide:

Elastx PcaaS Changelog: https://docs.elastx.cloud/docs/kubernetes/changelog/
Kubernetes Deprecated API Migration Guide: https://kubernetes.io/docs/reference/using-api/deprecation-guide/

Upgrade Process

NOTE Please refrain from making any changes while the upgrade is in progress.

The duration of the upgrade typically ranges from 1 to 3 hours, depending on the size of the cluster.
The upgrade starts with the control plane nodes followed by the worker nodes, one nodegroup at a time.

Steps Involved

A new node with the newer version is added to the cluster to replace the old node.
Once the new node is ready, the old node is drained.
Once all transferable loads have been migrated, the old node is removed from the cluster.
This process is repeated until all nodes in the cluster have been upgraded.

NOTE When using public IPs on worker nodes to ensure predictable egress IP, a previously unused IP will be assigned to the new worker node. This IP should have been provided to you in a list of all allocated IPs during your request for adding public IPs on the worker nodes.

Support and Communication During Upgrades

The engineer responsible for executing the upgrade will notify you through the support ticket when the upgrade begins and once it is completed. The support ticket serves as the primary channel for communication during the upgrade process. If you have any concerns or questions about the upgrade, please use the support ticket to reach out.

Additional Information

Upon request, upgrades can be scheduled outside office hours if needed. Upgrades outside of office hours depend on personnel availability and comes at an additional fee, see current price for professional services.
Our Kubernetes service includes up to four version upgrades per year; additional upgrades can be performed at an extra cost.
To address critical security vulnerabilities, additional upgrades can be performed and will not count against the four upgrades included per year.
In a previous Tech-fika, we discussed how to build redundancy and implement autoscaling with our Kubernetes service. You can access the presentation here to help you prepare for a smoother upgrade experience.

5 - Your first deployment

An example deployment to get started with your Kubernetes cluster

This page will help you getting a deployment up and running and exposed as a load balancer.

Note: This guide is optional and only here to help new Kubernetes users with an example deployment.

Before following this guide you need to have ordered a cluster and followed the Accessing your cluster guide

You can verify access by running kubectl get nodes and if the output is similar to the example below you are set to go.

❯ kubectl get nodes
NAME                           STATUS   ROLES           AGE     VERSION
hux-lab1-control-plane-c9bmm   Ready    control-plane   2d18h   v1.27.3
hux-lab1-control-plane-j5p42   Ready    control-plane   2d18h   v1.27.3
hux-lab1-control-plane-wlwr8   Ready    control-plane   2d18h   v1.27.3
hux-lab1-worker-447sn          Ready    <none>          2d18h   v1.27.3
hux-lab1-worker-9ltbp          Ready    <none>          2d18h   v1.27.3
hux-lab1-worker-htfbp          Ready    <none>          15h     v1.27.3
hux-lab1-worker-k56hn          Ready    <none>          16h     v1.27.3

Creating an example deployment

To get started we need a deployment to deploy. Below we have a deployment called echoserver we can use for this example.

Start off by creating a file called deployment.yaml with the content of the deployment below:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: echoserver
  name: echoserver
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: echoserver
  template:
    metadata:
      labels:
        app.kubernetes.io/name: echoserver
    spec:
      containers:
      - image: gcr.io/google-containers/echoserver:1.10
        name: echoserver

After you have created your file we can apply the deployment by running the following command:
```
❯ kubectl apply -f deployment.yaml
deployment.apps/echoserver created
```

After running the apply command we can verify that 3 pods have been created. This can take a few seconds.

❯ kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
echoserver-545465d8dc-4bqqn   1/1     Running   0          51s
echoserver-545465d8dc-g5xxr   1/1     Running   0          51s
echoserver-545465d8dc-ghrj6   1/1     Running   0          51s

Exposing our deployment

After your pods are created we need to make sure to expose our deployment. In this example we are creating a service of type loadbalancer. If you run this application in production you would likely install an ingress controller

First of we create a file called service.yaml with the content of the service below

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: echoserver
  name: echoserver
  annotations:
    loadbalancer.openstack.org/x-forwarded-for: "true"
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
    name: http
  selector:
    app.kubernetes.io/name: echoserver
  type: LoadBalancer

After creating the service.yaml file we apply it using kubectl

❯ kubectl apply -f service.yaml
service/echoserver created

We should now be able to use our service by running kubectl get service
```
❯ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
echoserver   LoadBalancer   10.98.121.166   <pending>     80:31701/TCP   54s
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        2d20h
```
For the echo service we can see that EXTERNAL-IP says <pending> this means that a load balancer is being created but is not yet ready. As soon as the load balancer is up and running we will instead use an IP address here we can use to access our application.

Loadbalancers usually take around a minute to be created however can sometimes take a little longer.

Once the load balancer is up and running the kubectl get service should return something like this:

❯ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
echoserver   LoadBalancer   10.98.121.166   185.24.134.39   80:31701/TCP   2m24s
kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP        2d20h

Access the example deployment

Now if we open our web browser and visits the IP address we should get a response looking something like this:

Hostname: echoserver-545465d8dc-ghrj6

Pod Information:
  -no pod information available-

Server values:
  server_version=nginx: 1.13.3 - lua: 10008

Request Information:
  client_address=192.168.252.64
  method=GET
  real path=/
  query=
  request_version=1.1
  request_scheme=http
  request_uri=http://185.24.134.39:8080/

Request Headers:
  accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
  accept-encoding=gzip, deflate
  accept-language=en-US,en;q=0.9,sv;q=0.8
  host=185.24.134.39
  upgrade-insecure-requests=1
  user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36
  x-forwarded-for=90.230.66.18

Request Body:
  -no body in request-

The Hostname shows which pod we reached and if we refresh the page we should be able to see this value change.

Cleanup

To clean up everything we created you an run the following set of commands

We can start off by removing the deployment. To remove a deployment we can use kubectl delete and point it towards the file containing our deployment:
```
❯ kubectl delete -f deployment.yaml
deployment.apps "echoserver" deleted
```
After our deployment are removed we can go ahead and remove our service and load balancer. Please note that this takes a few seconds since we are waiting for the load balancer to be removed.
```
❯ kubectl delete -f service.yaml
service "echoserver" deleted
```

6 - Recommendations

An list of things we recommend to get the best experience from your Kubernetes cluster

This page describes a list of things that could help you get the best experience out of your cluster.

Note: You do not need to follow this documentation in order to use your cluster

Ingress and cert-manager

To make it easier to expose applications an ingress controller is commonly deployed.

An ingress controller makes sure when you go to a specific webpage you are routed towards the correct application.

There are a lot of different ingress controllers available. We on Elastx are using ingress-nginx and have a guide ready on how to get started. However you can deploy any ingress controller you wish inside your clusters.

To get a single IP-address you can point your DNS towards we recommend to deploy an ingress-controller with a service of type LoadBalancer. More information regarding Load Balancers can be found in this link.

In order to automatically generate and update TLS certificates cert-manager is commonly deployed side by side with an ingress controller.

We have created a guide on how to get started with ingress-nginx and Cert-manager that can be found here in this link to guide.

Requests and limits

Below we describe requests and limits briefly. For a more detailed description or help setting requests and limits we recommend to check out Kubernetes documentation: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests

Requests and limits are critical to enable Kubernetes to make informed decisions on when and where to schedule and limit your workload.

Requests are important for the scheduler. Requests can be seen as “Amount of resources the pod would utilize during normal operation”. This means that the scheduler will allocate the required amount of resources and make sure they are always available to your pod.

Requests also enables the auto-scaler to make decisions on when to scale a cluster up and down.

Limits

Limits define the maximum allowed resource usage for a pod. This is important to avoid slowdowns in other pods running on the same node.

CPU limit. Your application will be throttled or simply run slower when trying to exceed the limit. Run slower equals to fewer cpu cycles per given time. That is, introducing latency. Memory limit is another beast. If any pod trying to use memory above the the limit, the pod will be Out of memory killed.

Autoscaling

Autoscaling can operate on both node-level and pod-level. To get the absolute best experience we recommend a combination of both.

Scaling nodes

We have built-in support for scaling nodes. To get started with autoscaling we recommend to check the guide in this link.

Scaling pods

Kubernetes official documentation has a guide on how to accomplish this in this piece of documentation.

In short, node autoscaling is only taken into consideration if you have any pods that cannot be scheduled or if you have set requests on your pods. In order to automatically scale an application pod scaling can make sure you get more pods before reaching your pod limit and if more nodes are needed in order to run the new pods nodes will automatically be added and then later removed when no longer needed.

Network policies

Network policies can in short be seen as Kubernetes built in firewalls.

Network policy can be used to limit both incoming and outgoing traffic. This is useful to specify a set of pods that are allowed to communicate with the database.

Kubernetes documentation have an excellent guide on how to get started with network policies in this link.

Pod Security Standards / Pod Security Admission

Pod Security Admission can be used to limit what your pods can do. For example you can make sure pods are not allowed to run as root.

In order to get to know this more in detail and getting started we recommend to follow the Kubernetes documentation in this link.

Load Balancers

Load Balancers allow your application to be accessed from the internet. Load Balancers can automatically split traffic to all your nodes to even out load. Load Balancers can also detect if a node is having problems and remove it to avoid displaying errors to end users.

We have a guide on how to get started with Boad Balancers in this link.