Basic Troubleshooting Procedures
Introduction
Fuzzball Orchestrate is architected as a collection of microservices deployed in a Kubernetes (K8s) cluster. To understand what is happening with the Fuzzball deployment, you must be able to inspect the status of the pods supporting these containerized microservices, and gather relevant logs.
Checking pod status
Any Fuzzball troubleshooting starts with confirming the presence and inspecting the status of containers/microservices within their respective pods.
You can view and inspect the status of pods hosting containerized microservices using the standard
K8s control plane tool, kubectl. After executing the following kubectl command, a table
is displayed giving information about all of the pods within the K8s cluster.
# # kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-56f459fcb7-j7wmt 1/1 Running 1 (35m ago) 19d
cert-manager cert-manager-56f459fcb7-m8gq2 1/1 Running 1 (35m ago) 19d
cert-manager cert-manager-56f459fcb7-qj9zd 1/1 Running 1 (35m ago) 19d
cert-manager cert-manager-cainjector-78f497d9d8-5fc5r 1/1 Running 1 (35m ago) 19d
cert-manager cert-manager-webhook-5bf5cccd59-qg5qs 1/1 Running 1 (35m ago) 19d
cert-manager trust-manager-778f7b488-gdmrh 1/1 Running 1 (35m ago) 19d
database database-debug-0 1/1 Running 1 (35m ago) 19d
database database-postgresql-0 1/1 Running 1 (35m ago) 19d
fuzzball-system fuzzball-operator-controller-manager-55cc86985f-c7tzn 2/2 Running 2 (35m ago) 19d
fuzzball audit-archive-28836720-4wcf5 0/1 Completed 0 2d6h
fuzzball audit-archive-28838160-86zjk 0/1 Completed 0 30h
fuzzball audit-archive-28839600-tbhj2 0/1 Completed 0 6h25m
fuzzball fuzzball-account-7749678b7c-m272g 1/1 Running 1 (35m ago) 19d
fuzzball fuzzball-account-external-5f5cd6f56-86k4z 1/1 Running 1 (35m ago) 19d
fuzzball fuzzball-account-worker-7f69c98747-gbtvc 1/1 Running 4 (33m ago) 19d
fuzzball fuzzball-admin-0 1/1 Running 1 (35m ago) 19d
[...snip]
If you want to view only pods running within the fuzzball namespace, you can change the command
like so:
# kubectl get pods -n fuzzball
NAME READY STATUS RESTARTS AGE
audit-archive-28836720-4wcf5 0/1 Completed 0 2d8h
audit-archive-28838160-86zjk 0/1 Completed 0 32h
audit-archive-28839600-tbhj2 0/1 Completed 0 8h
fuzzball-account-77bc99544-n2dgb 1/1 Running 0 119s
fuzzball-account-external-7977f8c5dc-hmfhf 1/1 Running 0 119s
fuzzball-account-worker-6b646c8b76-jq2sf 1/1 Running 0 119s
fuzzball-admin-0 1/1 Running 1 (173m ago) 19d
fuzzball-audit-74bb656b6b-lltx4 1/1 Running 0 119s
fuzzball-auth-spicedb-57bb79cf9b-k2zh7 1/1 Running 0 119s
fuzzball-dns-569f675947-cf59z 1/1 Running 0 119s
fuzzball-kube-68fc8d8b8d-f7gkg 1/1 Running 0 119s
fuzzball-log-845fc88dd9-bvr4c 1/1 Running 0 119s
fuzzball-openapi-686488649-7fj2s 1/1 Running 0 119s
fuzzball-organization-7f7f5f6878-qs55p 1/1 Running 0 119s
fuzzball-organization-external-cc78884d5-z56j5 1/1 Running 0 118s
fuzzball-organization-worker-55599cd8d8-htknd 1/1 Running 0 118s
fuzzball-permission-external-55d965c957-n5fsv 1/1 Running 0 118s
fuzzball-schedule-0 1/1 Running 0 170m
[...snip]
Note the values in the READY column indicating 1/1, (or potentially 2/2, etc.) and the
STATUS column indicating Running. Any pods that do not report a Running or Completed status,
or those that report fewer instances than expected should be investigated further by inspecting
logs.
Retrieving and Inspecting logs
While provisioning Fuzzball within a K8s cluster (using kubectl apply -f fuzzball.yaml), you can
watch the log output with the following command:
# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
You can stream logs from an individual pod using the following syntax. Note that the namespace must
be specified since Fuzzball pods are not deployed in the "default" namespace. For example, to view
the logs streaming from the fuzzball-schedule-0 pod, you could use the following command. (Omit
the --follow --tail=0 options/arguments if you want to view the log at a single instant without
streaming it.)
# kubectl logs -n fuzzball fuzzball-admin-0 --follow --tail=0
The kubectl logs command outputs verbose logs in json format. These can be filtered and
parsed into human-readable output using jq like so:
# kubectl logs -n fuzzball fuzzball-admin-0 | \
jq -r 'select(.level != "debug") | [.ts, .msg, .Error] | @tsv'
The kubectl utility does not provide a straightforward command for capturing all Fuzzball logs in
a single stream. However, the logs for a collected Fuzzball “app” can be inspected using a label
selector like so. (This is the strategy used above to stream logs during the provisioning process.)
# kubectl logs --prefix --namespace=fuzzball --all-containers \
--max-log-requests=99 --tail=0 --follow \
--selector='app.kubernetes.io/name=fuzzball-schedule'
The list of available labels can be viewed using kubectl get pods -n fuzzball --show-labels.
Finally, it may be useful to gather a full log dump (for example, to send to CIQ support). This can be accomplished by requesting logs from all app labels. The app labels may change with different versions of Fuzzball and can be inspected as described above. Here is an example:
# kubectl logs --prefix --all-containers --max-log-requests=99 \
--namespace=fuzzball \
--selector='app.kubernetes.io/name in (fuzzball-account, fuzzball-admin, fuzzball-audit, fuzzball-auth, fuzzball-dns, fuzzball-kube, fuzzball-log, fuzzball-openapi, fuzzball-organization, fuzzball-permission, fuzzball-schedule, fuzzball-secret, fuzzball-storage, fuzzball-sync, fuzzball-ui, fuzzball-user, fuzzball-workflow)'
Restarting Fuzzball
You can invoke the command kubectl rollout restart to restart the Fuzzball deployment. For
example:
# kubectl rollout restart deployment -n fuzzball
Other namespaces that may be pertinent for restart include
database- provides the PostgreSQL database that backs Fuzzball Orchestrate stateingress- maps HTTP(S) URLs to relevant Fuzzball services, including the Fuzzball UI, Fuzzball API, and Keycloakkyverno- provides a policy engine used by Fuzzball Orchestrateworkflow- provides the temporal service that is used by Fuzzball Orchestrateidentity- provides a Keycloak instance for identity and access management
Uninstalling Fuzzball
If a deployment failed or you have a need to uninstall a deployed Fuzzball installation, you may need to tear down the installation and start fresh. The following command will delete an existing Fuzzball deployment, allowing you to start over if needed.
# kubectl delete FuzzballOrchestrate fuzzball-orchestrate
You can watch the deprovisioning process as it occurs with the following command:
# kubectl logs -n fuzzball-system -l app.kubernetes.io/name=fuzzball-operator -f --tail=-1
You may also want to remove the substrate configuration directory that is shared out to compute
nodes via NFS. For instance:
# # rm -rf /srv/fuzzball/shared/substrate
This directory will be recreated if you re-provision your Fuzzball instance.