Skip to main content

PXE Boot/Provisioning

The CM4 / CM5 can natively boot via PXE, but requires additional files on the TFTP server for bootstrapping. In order to provide a standardized UEFI boot process (including PXE), there's an EDK2 UEFI build available for the ComputeBlade. Using this image for the SD Card (CM4 / CM5 Lite modules) or eMMC enables commodity provisioning software supporting ARM64 to bootstrap clusters.

Canonical MaaS​

Canonical MaaS is a bare-metal provisioning tool focused on Ubuntu. The How-To Guide works out of the box therefore no further documentation is specified here.

PXE Booting Nodes with Tinkerbell​

In this example, we're booting Talos Linux. This works with any other distribution supporting a netboot environment as well.

The component responsible for the PXE booting within the Tinkerbell stack is Smee. It can be setup either bare-metal or within a Kubernetes cluster.

Installing Tinkerbell on a single-node Kubernetes installation​

Tinkerbell, or more specific for this use-case Smee, can be installed using helm:

values.yaml:

tink:
  server:
    deploy: false
  controller:
    deploy: false
rufio:
  deploy: false
hegel:
  deploy: false
smee:
  hostNetwork: true
  publicIP: 100.64.0.2  # Set to the IP address of your node
  trustedProxies:
  - 0.0.0.0/0
  nodeSelector:
    kubernetes.io/hostname: talos-di6-jbg  # Set to the k8s node name. Has to match the publicIP value above
stack:
  enabled: false

helm install tink-stack oci://ghcr.io/tinkerbell/charts/stack --version "0.4.5" --create-namespace --namespace tink-system --wait -f values.yaml

Defining Boot Process​

After everything is up and running, you can declare Hardware resources with networking configuration alongside boot information. Here an exampel for a ComputeModule with the MAC address d8:3a:dd:bb:36:be:

---
apiVersion: "tinkerbell.org/v1alpha1"
kind: Hardware
metadata:
  name: bld-d83addbb36be
  namespace: tink-system
spec:
  interfaces:
    - dhcp:
        ip:
          address: 100.64.0.100
          gateway: 100.64.0.1
          netmask: 255.255.255.0
        mac: d8:3a:dd:bb:36:be
        name_servers:
          - 100.64.0.1
        uefi: true
      netboot:
        allowPXE: true
        allowWorkflow: false
        ipxe:
          url: https://pxe.factory.talos.dev/pxe/779804eb249e184f2986a0cacb95aa6ed8f133878449da8df9a68829f90902b9/v1.7.5/metal-arm64

Upon powering up the blade, the boot process can be followed in the log output of smee:

smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086063.1273975,"caller":"reservation/handler.go:86","msg":"received DHCP packet","mac":"d8:3a:dd:bb:36:be","xid":"0xa89f634e","interface":"eth0","type":"DISCOVER"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086063.1281593,"caller":"reservation/handler.go:141","msg":"sent DHCP response","mac":"d8:3a:dd:bb:36:be","xid":"0xa89f634e","interface":"eth0","type":"OFFER","bootFileName":"snp.efi","nextServer":"100.64.0.2","ipAddress":"100.64.0.100","destination":"255.255.255.255:68"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086071.1267426,"caller":"reservation/handler.go:101","msg":"received DHCP packet","mac":"d8:3a:dd:bb:36:be","xid":"0xa89f634e","interface":"eth0","type":"REQUEST"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086071.1273892,"caller":"reservation/handler.go:141","msg":"sent DHCP response","mac":"d8:3a:dd:bb:36:be","xid":"0xa89f634e","interface":"eth0","type":"ACK","bootFileName":"snp.efi","nextServer":"100.64.0.2","ipAddress":"100.64.0.100","destination":"255.255.255.255:68"}
smee-dfcb5c7b6-h52h9 smee {"level":"error","ts":1722086071.1357145,"logger":"github.com/tinkerbell/ipxedust","caller":"itftp/itftp.go:106","msg":"file serve failed","service":"github.com/tinkerbell/smee","event":"get","filename":"snp.efi","uri":"snp.efi","client":{"IP":"100.64.0.100","Port":1389,"Zone":""},"macFromURI":"","b":0,"contentSize":267776,"error":"sending block 0: code=8, error: User aborted the transfer\u0000","stacktrace":"github.com/tinkerbell/ipxedust/itftp.Handler.HandleRead\n\t/home/runner/go/pkg/mod/github.com/tinkerbell/ipxedust@v0.0.0-20231215220341-a535c5deb47a/itftp/itftp.go:106\ngithub.com/pin/tftp/v3.(*Server).handlePacket.func2\n\t/home/runner/go/pkg/mod/github.com/pin/tftp/v3@v3.1.0/server.go:455"}
smee-dfcb5c7b6-h52h9 smee {"level":"error","ts":1722086072.9095297,"logger":"github.com/tinkerbell/ipxedust","caller":"itftp/itftp.go:106","msg":"file serve failed","service":"github.com/tinkerbell/smee","event":"get","filename":"snp.efi","uri":"snp.efi","client":{"IP":"100.64.0.101","Port":1097,"Zone":""},"macFromURI":"","b":142848,"contentSize":267776,"error":"Channel timeout: 100.64.0.101:1097","stacktrace":"github.com/tinkerbell/ipxedust/itftp.Handler.HandleRead\n\t/home/runner/go/pkg/mod/github.com/tinkerbell/ipxedust@v0.0.0-20231215220341-a535c5deb47a/itftp/itftp.go:106\ngithub.com/pin/tftp/v3.(*Server).handlePacket.func2\n\t/home/runner/go/pkg/mod/github.com/pin/tftp/v3@v3.1.0/server.go:455"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086073.7314184,"logger":"github.com/tinkerbell/ipxedust","caller":"itftp/itftp.go:111","msg":"file served","service":"github.com/tinkerbell/smee","event":"get","filename":"snp.efi","uri":"snp.efi","client":{"IP":"100.64.0.100","Port":1390,"Zone":""},"macFromURI":"","bytesSent":267776,"contentSize":267776}
smee-dfcb5c7b6-h52h9 smee {"level":"error","ts":1722086073.7342472,"logger":"github.com/tinkerbell/ipxedust","caller":"itftp/itftp.go:91","msg":"file unknown","service":"github.com/tinkerbell/smee","event":"get","filename":"autoexec.ipxe","uri":"autoexec.ipxe","client":{"IP":"100.64.0.100","Port":1391,"Zone":""},"macFromURI":"","error":"file [autoexec.ipxe] unknown: file does not exist","stacktrace":"github.com/tinkerbell/ipxedust/itftp.Handler.HandleRead\n\t/home/runner/go/pkg/mod/github.com/tinkerbell/ipxedust@v0.0.0-20231215220341-a535c5deb47a/itftp/itftp.go:91\ngithub.com/pin/tftp/v3.(*Server).handlePacket.func2\n\t/home/runner/go/pkg/mod/github.com/pin/tftp/v3@v3.1.0/server.go:455"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086082.7369385,"caller":"reservation/handler.go:86","msg":"received DHCP packet","mac":"d8:3a:dd:bb:36:be","xid":"0x98b69526","interface":"eth0","type":"DISCOVER"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086082.7379577,"caller":"reservation/handler.go:141","msg":"sent DHCP response","mac":"d8:3a:dd:bb:36:be","xid":"0x98b69526","interface":"eth0","type":"OFFER","bootFileName":"https://pxe.factory.talos.dev/pxe/779804eb249e184f2986a0cacb95aa6ed8f133878449da8df9a68829f90902b9/v1.7.5/metal-arm64","nextServer":"100.64.0.2","ipAddress":"100.64.0.100","destination":"255.255.255.255:68"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086082.7425363,"caller":"reservation/handler.go:101","msg":"received DHCP packet","mac":"d8:3a:dd:bb:36:be","xid":"0x98b69526","interface":"eth0","type":"REQUEST"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086082.7433407,"caller":"reservation/handler.go:141","msg":"sent DHCP response","mac":"d8:3a:dd:bb:36:be","xid":"0x98b69526","interface":"eth0","type":"ACK","bootFileName":"https://pxe.factory.talos.dev/pxe/779804eb249e184f2986a0cacb95aa6ed8f133878449da8df9a68829f90902b9/v1.7.5/metal-arm64","nextServer":"100.64.0.2","ipAddress":"100.64.0.100","destination":"255.255.255.255:68"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.658168,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"Configuring (net0 d8:3a:dd:bb:36:be)...... ok","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.6583776,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"net0: 100.64.0.100/255.255.255.0 gw 100.64.0.1","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.6585057,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"net0: fe80::da3a:ddff:febb:36be/64","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.658876,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"Next server: 100.64.0.2","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.6590402,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"Filename: https://pxe.factory.talos.dev/pxe/779804eb249e184f2986a0cacb95aa6ed8f133878449da8df9a68829f90902b9/v1.7.5/metal-arm64","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.686273,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"https://pxe.factory.talos.dev/pxe/779804eb249e184f2986a0cacb95aa6ed8f133878449da8df9a68829f90902b9/v1.7.5/metal-arm64... ok","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086083.6873302,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"metal-arm64 : 259 bytes [script]","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086087.3916304,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"/kernel-arm64... ok","severity":"INFO"}}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086090.1534588,"caller":"syslog/receiver.go:160","msg":"msg","msg":{"app-name":"ipxe","facility":"kern","host":"100.64.0.100","msg":"/initramfs-arm64.xz... ok","severity":"INFO"}}


smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086112.4891257,"caller":"reservation/handler.go:86","msg":"received DHCP packet","mac":"d8:3a:dd:bb:36:be","xid":"0x3bb97cad","interface":"eth0","type":"DISCOVER"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086112.4899402,"caller":"reservation/handler.go:141","msg":"sent DHCP response","mac":"d8:3a:dd:bb:36:be","xid":"0x3bb97cad","interface":"eth0","type":"OFFER","nextServer":"100.64.0.2","ipAddress":"100.64.0.100","destination":"255.255.255.255:68"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086112.4966073,"caller":"reservation/handler.go:101","msg":"received DHCP packet","mac":"d8:3a:dd:bb:36:be","xid":"0x3bb97cad","interface":"eth0","type":"REQUEST"}
smee-dfcb5c7b6-h52h9 smee {"level":"info","ts":1722086112.5007463,"caller":"reservation/handler.go:141","msg":"sent DHCP response","mac":"d8:3a:dd:bb:36:be","xid":"0x3bb97cad","interface":"eth0","type":"ACK","nextServer":"100.64.0.2","ipAddress":"100.64.0.100","destination":"255.255.255.255:68"}

To confirm the provisioning has been successful:

130 xvzf@zephyr ..exus/kube/clusters/aperture/tinkerbell (git)-[main] % talosctl get addresses --insecure  -n 100.64.0.100
NODE   NAMESPACE   TYPE            ID                                                       VERSION   ADDRESS                                  LINK
       network     AddressStatus   enxd83addbb36be/100.64.0.100/24                          1         100.64.0.100/24                          enxd83addbb36be
       network     AddressStatus   enxd83addbb36be/fd33:b4d:c0fe:a:da3a:ddff:febb:36be/64   1         fd33:b4d:c0fe:a:da3a:ddff:febb:36be/64   enxd83addbb36be
       network     AddressStatus   enxd83addbb36be/fe80::da3a:ddff:febb:36be/64             2         fe80::da3a:ddff:febb:36be/64             enxd83addbb36be
       network     AddressStatus   lo/127.0.0.1/8                                           1         127.0.0.1/8                              lo
       network     AddressStatus   lo/::1/128                                               1         ::1/128                                  lo