K3s on Libvirt Quickstart

Setting up extra terraform provider

Due to an issue in the terraform provider system, you need to download and place the libvirt provider at a very specific location in your home directory before deploying K3s. You can find the libvirt provider version in modules/k3s/libvirt/versions.tf. To keep things simple this version will be referred to as $LIBVIRT_PROVIDER_VERSION in this documentation. Go to https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v$LIBVIRT_PROVIDER_VERSION and find the correct link for your OS/CPU_ARCH (an example is linux_amd64, referred to as $OS_CPU_ARCH in the rest of this documentation). 
mkdir -p ~/.local/share/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/$LIBVIRT_PROVIDER_VERSION/$OS_CPU_ARCH/
mv terraform-provider-libvirt ~/.local/share/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/$LIBVIRT_PROVIDER_VERSION/$OS_CPU_ARCH/terraform-provider-libvirt

Prerequisites

  • Access to a functional Libvirt daemon

  • Knowledge of Terraform basics

  • jq binary

  • argocd CLI

Create your Terraform root module

Camptocamp’s DevOps Stack is instantiated using a Terraform composition module.

Here is a minimal working example:

# terraform/main.tf

module "cluster" {
  source = "git::https://github.com/camptocamp/devops-stack.git//modules/k3s/libvirt?ref=master"

  cluster_name = "my-cluster"
  node_count   = 2
}

Terraform Outputs

Define outputs:

# terraform/outputs.tf

output "kubeconfig" {
  sensitive = true
  value     = module.cluster.kubeconfig
}

output "argocd_url" {
  value = format("https://argocd.apps.%s.%s", var.cluster_name, module.cluster.base_domain)
}

output "keycloak_url" {
  value = format("https://keycloak.apps.%s.%s", var.cluster_name, module.cluster.base_domain)
}

output "grafana_url" {
  value = format("https://grafana.apps.%s.%s", var.cluster_name, module.cluster.base_domain)
}

output "prometheus_url" {
  value = format("https://prometheus.apps.%s.%s", var.cluster_name, module.cluster.base_domain)
}

output "alertmanager_url" {
  value = format("https://alertmanager.apps.%s.%s", var.cluster_name, module.cluster.base_domain)
}

output "argocd_server_admin_password" {
  sensitive = true
  value     = module.cluster.argocd_server_admin_password
}

output "keycloak_admin_password" {
  sensitive = true
  value     = module.cluster.keycloak_admin_password
}

output "grafana_admin_password" {
  sensitive = true
  value     = module.cluster.grafana_admin_password
}

output "keycloak_users" {
  value     = module.cluster.keycloak_users
  sensitive = true
}

Deploy the cluster

$ terraform init
$ terraform apply

You should see the services URL as Terraform outputs.

Get kubeconfig and Keycloak users credentials

Retrieve the Kubeconfig file:

$ terraform output -json kubeconfig | jq -r . > kubeconfig.yaml
$ export KUBECONFIG=kubeconfig.yaml

By default, two users are defined in Keycloak:

user Keycloak role Keycloak realm terraform output comment

admin

Administrator

all

keycloak_admin_password

This user has admin rights only in Keycloak. Use keycloak_url and select "Administration Console" to login.

jdoe

applications

devops-stack

keycloak_users

This user has related applications rights within Kubernetes realm. Use keycloak_url/auth/realms/devops-stack/account/ to login.

To retrieve password:

$ terraform output keycloak_admin_password
$ terraform output keycloak_users
# a user map is displayed that includes jdoe password

Wait for Keycloak to be ready

$ kubectl -n keycloak get sts
NAME       READY   AGE
keycloak   1/1     8m58s

Wait until the READY column says 1/1.

Inspect the DevOps Stack Applications

You can view the ingress routes for the various DevOps Stack Applications with:

$ kubectl get ingress --all-namespaces

Access the URLs in https or use the URL output from terraform (see table below).

Application URL user password comment

Argo CD

argocd_url

admin

argocd_server_admin_password

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. The argo CD web application allows you to visualise the application deployment, configurations, status to name a few.

Grafana

grafana_url

admin

grafana_admin_password

Grafana is a visualisation web application for metrics or log data. The devops-stack provides pre-defined dashboards ready to use. The devops-stack provides pre-defined dashboards ready to use.

Grafana

grafana_url

jdoe

jdoe_password

To visualise logs in Grafana (use "Explore" menu), users need "Editor" Grafana rights. By default in devops-stack, Grafana user rights is set to "Editor" such as John Doe (our user example).

Prometheus

prometheus_url

n/a

n/a

Prometheus web app is mainly used to test queries, a one time metrics visualisation. This application is not used for dashboarding but Grafana instead.

Alertmanager

alertmanager_url

n/a

n/a

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration (e.g. email, PagerDuty, etc). It also takes care of silencing and inhibition of alerts.

Access the Keycloak dashboard

The keycloak dashboard uses the devops-stack realm. You can log in to it using the /auth/realms/devops-stack/account/ path with the Keycloak ingress.

Destroy the cluster

$ terraform destroy

Reference

See the K3s Libvirt reference page.