How to Collect Diagnostic Data for Swimlane Support
swimlane 10 x support bundle a support bundle should be requested by a support engineer for initial triage this bundle contains logs from all relevant pods, as well as other useful information from your deployment using the command line (online & airgap) download and unarchive the latest release curl l https //github com/replicatedhq/troubleshoot/releases/latest/download/support bundle linux amd64 tar gz | tar xzvf for airgap (offline) environments, run the above command from a host that is connected to the internet to download the plugin, then do the following copy the downloaded file to your offline swimlane server scp support bundle linux amd64 tar gz user@\<swimlanedns> /path/to/file unarchive the downloaded file from the step above as follows tar xzvf support bundle linux amd64 tar gz move the plugin to your $path sudo mv /support bundle /usr/local/bin/kubectl support bundle note if you choose not to put the plugin into your $path, then replace all instances of kubectl support bundle in the remainder of this document with /support bundle or with the absolute path to the binary where it was unzipped gather the support bundle note replace \<namespace> with the applicable namespace (default for embedded cluster) kubectl support bundle interactive=false secret/\<namespace>/kotsadm swimlane platform supportbundle alternatively, if you have an online installation, you can also generate a support bundle directly from the swimlane admin console under application > troubleshoot host collector support bundle a host collector support bundle should be needed by a support engineer in the following situations a cluster is down the application is not installed the admin console is not working to debug a host specific performance and configuration problems even when the cluster is running this bundle contains information gathered directly from the host, such as cpu, memory, available block devices, and more note install / unarchive the latest support bundle release before proceeding to gather a host collector support bundle, execute the following command on each node in the cluster with elevated privileges / as root kubectl support bundle interactive=false https //raw\ githubusercontent com/replicatedhq/troubleshoot specs/main/host/default yaml the resulting host collector support bundle will be available in the current working directory as support bundle yyyy mm ddthh mm ss tar gz note for an air gapped swimlane deployment, on a server with internet access, download https //raw\ githubusercontent com/replicatedhq/troubleshoot specs/main/host/default yaml in your browser and copy default yaml to your swimlane server(s) then execute the following command on each node in the cluster with elevated privileges / as root kubectl support bundle interactive=false /default yaml application level data application level data should be requested by a support engineer in the following situations application builder or workflow is not working as expected parts of the swimlane ui are not displaying correctly to further debug issues related to a specific swimlane application to gather swimlane application level data, perform the following steps recreate the issue and obtain a browser trace ( har) export the application in ssp format ( applications & applets > āyour applicationā > export ) collect application specific documents from mongodb obtain the application id if you are unaware of your applicationās id, navigate to applications and applets and click your application the application id will be visible in the url like so\ https //swimlane url/app builder/al7ugqeshrcqaxfvr determine the mongodb primary by running the following command kubectl exec swimlane sw mongo 0 mongo u admin p $(kubectl get secret swimlane sw mongo admin o jsonpath="{ data password}" | base64 decode) authenticationdatabase admin tls tlsallowinvalidhostnames tlsallowinvalidcertificates admin eval='rs ismaster() primary;' you should see output similar to the following swimlane sw mongo 0 swimlane sw mongo default svc cluster local 27017 this example tells us that swimlane sw mongo 0 is the primary take note of this value execute the following set of commands (replace mongo primary and application id with the values above) kubectl exec it mongo primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates tlsallowinvalidhostnames swimlane eval='db applications find({ " id" "application id"}) pretty()' > app json kubectl exec it mongo primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates tlsallowinvalidhostnames swimlane eval='db settings find({}) pretty()' > settings json kubectl exec it mongo primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates tlsallowinvalidhostnames swimlane eval='db tasks find({ "applicationid" "application id" }) pretty()' > tasks json kubectl exec it mongo primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates tlsallowinvalidhostnames swimlane eval='db workflow\ find({ "applicationid" "application id" }) pretty()' > workflow\ json kubectl exec it mongo primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates tlsallowinvalidhostnames swimlane eval='db workflowruns find({ "applicationid" "application id" }) pretty()' > workflow runs json \# archive the resulting file star czvf app data tar gz app json settings json tasks json workflow\ json workflow runs json upload the browser trace har, application export ssp, and the app data tar gz archive http client logging for tasks http client logging should be requested by a support engineer in the following situations when a task results in a client or server error (http 4xx or 5xx) to further investigate outbound requests made by a specific task to enable http client logging for a task, perform the following steps fork the task, i e create a new task (same as the problematic task), expand the dropdown next to ācreateā, then click āforkā navigate to the newly forked task, go to the configuration tab and add the following code to the beginning of the task import requests import logging from http client import httpconnectionhttpconnection debuglevel = 1logging basicconfig(level=logging debug) take note of the original taskās inputs and add them manually to the forked task for example, here is the original input from the problematic task then, we add it manually to the forked task like so execute the forked task in the debugger and copy the output to a text file once complete, upload the resulting file mongodb diagnostics note this applies to swimlane 10 x mongodb logs and diagnostic information should be requested by a support engineer in the following situations if any swimlane sw mongo pod is not in ārunningā status (i e stuck in init 2/3, crashloopbackoff, etc ) to address general performance issues / slow queries to debug mongodb specific performance and configuration problems to gather mongodb diagnostics, perform the following steps determine the mongodb primary by running the following command kubectl exec swimlane sw mongo 0 mongo u admin p authenticationdatabase admin tls tlsallowinvalidhostnames tlsallowinvalidcertificates admin eval='rs ismaster() primary;' you should see output similar to the following swimlane sw mongo 0 swimlane sw mongo default svc cluster local 27017 this example tells us that swimlane sw mongo 0 is the primary take note of this value create the script using the code file found at bottom of article and set user executable permissions like so chmod u+x mongo collector sh execute the above script and provide it with the mongodb primary when prompted (or press enter for default, swimlane sw mongo 0) /mongo collector sh | tee mongo collector results log upload the resulting file, /tmp/hostname mongo logs xxxxxxxxxx tar gz velero bundle the command below will generate a debug bundle file (bundle \<date> tar gz) which contains detail logs of the backup in question velero get backup velero debug backup \<backupname> code files mongo collector sh \#!/bin/bash \# set mongodb replicaset primary member # read p 'enter the primary mongodb replicaset member \[press enter for default swimlane sw mongo 0] ' mongodb primary mongodb primary="${mongodb primary swimlane sw mongo 0}" \# backup mongodb logs \## get current date ## now=$(date +"%s") \## configure output location on the host ## loc="/tmp/mongo logs $ now" mkdir p $ loc file="/tmp/$(hostname) mongo logs $ now\ tar gz" \## collect mongod logs and diagnostic data content ## for pod in $(kubectl get pods l app=sw mongo o=name | cut d "/" f 2) do echo "collect pod describe for $pod" echo "==============================" kubectl describe pod $pod > $ loc/$pod describe txt echo "collecting mongod logs from $pod to $ loc/$pod log" echo "=====================================================================================================" kubectl logs since=24h $pod > $ loc/$pod log echo "collecting diagnostic data from $pod to $ loc/$pod diagnostic data tar gz" echo "============================================================================================================================" kubectl exec $pod tar zcvf /tmp/mongo diagnostic data tar gz /data/db/diagnostic data warning=no file changed kubectl cp $pod /tmp/mongo diagnostic data tar gz $ loc/$pod diagnostic data tar gz echo "===========================================================" echo "deleting temporary diagnostic data from $pod" echo "===========================================================" kubectl exec $pod rm /tmp/mongo diagnostic data tar gz done \## collect general mongodb information ## echo "collecting records stats" echo "========================" kubectl exec $mongodb primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="db records stats();" > $ loc/db records stats txt echo "collecting records getindexes" echo "=============================" kubectl exec $mongodb primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="db records getindexes();" > $ loc/db records getindexes txt echo "collecting rs status" echo "====================" kubectl exec $mongodb primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs status();" > $ loc/rs status txt echo "collecting rs config" echo "====================" kubectl exec $mongodb primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs config();" > $ loc/rs config txt echo "collecting rs printreplicationinfo" echo "==================================" kubectl exec $mongodb primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs printreplicationinfo();" > $ loc/rs printreplicationinfo txt echo "collecting printsecondaryreplicationinfo" echo "========================================" kubectl exec $mongodb primary mongo quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs printsecondaryreplicationinfo();" > $ loc/rs printsecondaryreplicationinfo txt \## tar the $ loc folder ## echo "compressing mongo logs and information"echo "======================================"tar zcvf $ file $ loc echo "removing $ loc" echo "==========================" rm rf $ loc echo "completed " ls lh $ file echo echo "please upload $ file to our secure portal and update the support ticket " echo mongo collector mongosh sh \#!/bin/bash \# set mongodb replicaset primary member # read p 'enter the primary mongodb replicaset member \[press enter for default swimlane sw mongo 0] ' mongodb primary mongodb primary="${mongodb primary swimlane sw mongo 0}" \# backup mongodb logs \## get current date ## now=$(date +"%s") \## configure output location on the host # loc="/tmp/mongo logs $ now" mkdir p $ loc file="/tmp/$(hostname) mongo logs $ now\ tar gz" \## collect mongod logs and diagnostic data content ## for pod in $(kubectl get pods l app=sw mongo o=name | cut d "/" f 2) do echo "collect pod describe for $pod" echo "==============================" kubectl describe pod $pod > $ loc/$pod describe txt echo "collecting mongod logs from $pod to $ loc/$pod log" echo "=====================================================================================================" kubectl logs since=24h $pod > $ loc/$pod log echo "collecting diagnostic data from $pod to $ loc/$pod diagnostic data tar gz" echo "============================================================================================================================" kubectl exec $pod tar zcvf /tmp/mongo diagnostic data tar gz /data/db/diagnostic data warning=no file changed kubectl cp $pod /tmp/mongo diagnostic data tar gz $ loc/$pod diagnostic data tar gz echo "===========================================================" echo "deleting temporary diagnostic data from $pod" echo "===========================================================" kubectl exec $pod rm /tmp/mongo diagnostic data tar gz done \## collect general mongodb information ## echo "collecting records stats" echo "========================" kubectl exec n swimlane $mongodb primary mongosh quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="db records stats();" > $ loc/db records stats txt echo "collecting records getindexes" echo "=============================" kubectl exec n swimlane $mongodb primary mongosh quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="db records getindexes();" > $ loc/db records getindexes txt echo "collecting rs status" echo "====================" kubectl exec n swimlane $mongodb primary mongosh quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs status();" > $ loc/rs status txt echo "collecting rs config" echo "====================" kubectl exec n swimlane $mongodb primary mongosh quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs config();" > $ loc/rs config txt echo "collecting rs printreplicationinfo" echo "==================================" kubectl exec n swimlane $mongodb primary mongosh quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs printreplicationinfo();" > $ loc/rs printreplicationinfo txt echo "collecting printsecondaryreplicationinfo" echo "========================================" kubectl exec n swimlane $mongodb primary mongosh quiet u admin p authenticationdatabase admin tls tlsallowinvalidcertificates swimlane eval="rs printsecondaryreplicationinfo();" > $ loc/rs printsecondaryreplicationinfo txt \## tar the $ loc folder ## echo "compressing mongo logs and information" echo "======================================" tar zcvf $ file $ loc echo "removing $ loc" echo "==========================" rm rf $ loc echo "completed " ls lh $ file echo echo "please upload $ file to our secure portal and update the support ticket " echo