Troubleshooting using swimlane-tools pod

The Swimlane application container images such as swimlane-web, swimlane-tasks, and swimlane-api by design are lightweight and do not include the basic utilities such as curl, telnet, ping, nslookup, etc. The reduced image size helps speed up the building and deploying of the containers.

To make troubleshooting easier, the Swimlane Platform Installer includes a swimlane-tools container that can be used for diagnosing the most common issues, especially for air gapped/offline installs. This would include, for example, tools like telnet, nslookup, curl, netcat, mtr, tcptraceroute (to test network connectivity), ldapsearch (for LDAP issues), mongo client, and other useful utilities.

Accessing the swimlane-tools container

There are few ways to execute into the container. Here are two options:

kubectl -n <swimlane-namespace> get pods kubectl exec -it <swimlane-tools-pod> -- /bin/bash

Available Tools

The container includes the following tools:

Debian Linux apt tools

curl

dnsutils

htop

iftop

iotop

iputils-ping

ldap-utils

nano

net-tools

netcat

ngrep

nmap

nmon

openssl

procps

snmp

sysstat

tcpdump

tcptrace

tcptraceroute

telnet

traceroute

tshark

wget

gnupg

python3

python3-pip

For a full list of installed packages, run ‘apt list --installed’.

Examples:

root@swimlane-tools-fdd568c9d-m9vgk:/# traceroute -n sw-api traceroute to sw-api (10.96.2.208), 30 hops max, 60 byte packets1 10.96.2.208 0.095 ms 0.050 ms 0.043 ms

root@swimlane-tools-fdd568c9d-m9vgk:/# nslookup swimlane-sw-mongo Server:10.96.0.10Address:10.96.0.10#53 Name:swimlane-sw-mongo.default.svc.cluster.localAddress: 10.32.0.17

root@swimlane-tools-fdd568c9d-m9vgk:/# python3 Python 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. import socket print(socket.gethostbyname_ex('swimlane.com')) ('swimlane.com', [], ['172.67.147.162', '104.21.47.122']) import requests requests.get('https://swimlane.com') <Response [200]> import certifi certifi.where() '/usr/local/lib/python3.7/dist-packages/certifi/cacert.pem' exit()

Test port connectivity using nc

TCP: nc -z -v [hostname/IP address] [port number] UDP: nc -z -v -u [hostname/IP address] [port number]

root@swimlane-tools-fdd568c9d-m9vgk:/# nc -z -v sw-api 5000 sw-api.default.svc.cluster.local [10.96.2.208] 5000 (?) open

root@swimlane-tools-fdd568c9d-m9vgk:/# nc -z -v swimlane-sw-mongo 27017 swimlane-sw-mongo.default.svc.cluster.local [10.32.0.12] 27017 (?) open

root@swimlane-tools-fdd568c9d-m9vgk:/# nc -z -v sw-web 443 sw-web.default.svc.cluster.local [10.96.1.116] 443 (https) open

Use OpenSSL s_client to check and verify SSL/TLS connectivity

Open an SSL connection and print the SSL certificate:

openssl s_client -connect <hostname or IP>:<port>

Print all certificates in the certificate chain presented by the SSL service. This is useful when troubleshooting missing intermediate CA certificate issues:

openssl s_client -connect <hostname or IP>:<port> -showcerts

To specify the TLS version in the connection for testing various protocols:

openssl s_client -connect <hostname or IP>:<port> -tls1_3

For more information, see OpenSSL s_client commands man page in the OpenSSL toolkit.

MongoDB Client and Tools

Mongo shell:

root@swimlane-tools-fdd568c9d-m9vgk:/# mongo -u Admin -p --host swimlane-sw-mongo --port 27017 --authenticationDatabase admin --tls --tlsAllowInvalidCertificates admin

MongoDB Backup and Restore script:

root@swimlane-tools-fdd568c9d-m9vgk:/# python3 /usr/local/bin/backup_restore_ha.py -h usage: backup_restore_ha.py [-h] [-v] [--cert CERT] {backup,restore} ... positional arguments: {backup,restore} backup Backup command. restore Restore command optional arguments: -h, --help show this help message and exit -v, --verbose Logging verbosity level. --cert CERT Location of Mongo pem certificate. root@swimlane-tools-fdd568c9d-m9vgk:/# python3 /usr/local/bin/backup_restore_ha.py backup root@swimlane-tools-fdd568c9d-m9vgk:/# python3 /usr/local/bin/backup_restore_ha.py restore

Test Internet Connectivity

$ env | grep -i proxy $ curl -I https://www.virustotal.com $ kubectl -n <swimlane-namespace> exec swimlane-tools-0 -- curl -I https://www.virustotal.com # Run the test connection script from the host $ python -m pip install requests $ vi test-connection.py # add the following content: # ----------BEGIN--------------- import requests # initializing URL url = "https://www.virustotal.com" proxy = {'http_proxy' : 'http://internet.example.com:83', 'https_proxy' : 'http://internet.example.com:83'} verify = False timeout = 60 try: # requesting URL request = requests.get(url, proxies=proxy, verify=verify, timeout=timeout) print("Internet is on") # catching exception except (requests.ConnectionError, requests.Timeout) as exception: print("Internet is off") # ----------END--------------- # Copy the same script to the tasks pod and test $ kubectl -n <swimlane-namespace> cp test-connection.py <swimlane-tasks-pod>:/tmp/test-connection.py # Run from the container $ kubectl -n <swimlane-namespace> exec -it <swimlane-tasks-pod> -- /bin/bash # Inside the container python /tmp/test-connection.py exit

tcpdump

# Run the following command on each of the node

sudo tcpdump -i any host <fqdn> sudo tcpdump -i any src host quay.io and "tcp[tcpflags] & (tcp-syn) != 0" and "tcp[tcpflags] & (tcp-ack) == 0"

# Wait for few mins to see if any traffic is being sent to <host> and identify the source

# Control-C to break

Test_Internet_Connection.json copy the below and save:

{"$type":"Core.Models.Integrations.Task, Core","action":{"$type":"Core.Models.Integrations.Actions.Python.Python3Action, Core","type":"python3","descriptor":{"$type":"Core.Models.Integrations.Actions.AvailableActionDescriptor, Core","description":"Execute Python 3 scripts","actionType":"python3","family":"Scripts","base64Image":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFIAAABSCAYAAADHLIObAAAQFElEQVR4Ae3aeXRUZZrH8UcRpaHxdNs62j12z7iIC0KWStW9lSUBQiphwRBaiON4ehAEyV4VkkolQRJcBB0VFQEhqSRV2XeSQIPa7guBhEBIwpKQxIiCC9DqOPa0y3nmd+sWhK6QpW5VIPTJH9/zWod/PB+eW++t94WY2Q2NNoowCjkKOQo52ijkKOQ/BeTEf/2aZhhq6NemtURrVhNlZioog67KfJK8VlWQ2lRHgnEHaVO2k5C4nbRptRM0qXUawVi3REzbtk5IqyoWU6vfEUxVLehjwVT5hZBSeUpIqZA6KRjLO9E+wVi2UzSWZQvJZSafhMr71CnFtwgri0ibVEQey2pIE19CGr2VvBPz6JrVT0n/DyhTUeNT19KR7XOID3iMMMhkQKbWzQTeZm1aTScAWUyVqmYBAdBeJQMRVdgrZyCiMlTKQrJUiVxS8XfoTUDGeSzbdvOIhfyVBPn4Y8DMUNBqGpP5hA1Sk1brK5i2vyGm17CUIkTkgGiviDGVrDEUfSkmFq1S660TRhakvoZuSF5HYx/LoLGr1igok8Y99hSp0quWa9Pq/k9MkwCHBREVyiUWsGDI36NKtEwak/GENARotXNlAHL1GqKUZ+hw3VzXIH/5u7/StOQ6CooqIf+YYlTidAFRxaQ1lEYI6bXAQxcDEYmJ+axJtB67beWGf7s98WW6bchtoN/GbSKf9Gdo+uOZFLrqCereFeriRP7uDAWtqUHbKShzBwWu2kkB6bvQTifada1v+vYuh0n8STRVbwHiIlVq+XK1qaJedB8iyrclGqzspy+o807KvdIrOZcGyzMplybFFZLplbX09d75xPumEddriBt9kEo55ITrv6Wpi9+lKWjqw++SkFKLjaLGqcTUWp0DIvsZK+PElEoS0raRZ2oJeZtKf4HPu88hGh02FqcRkcGKLKwx5PGU+LygyTF5NDnGcuFiLXR3tJVuWVZET+Xr6cfj04mPBhM3+wPTS0J0DZKot7HX/Ehe+u2kxqYBDDRk1AcdHucuTONViDCRpDFVkC9AtaaqBa4h5vdBlBL1eewRY3l6UlQeTVphuWC3LrfSvy/Lp1dyTcSf+CFAtumIDwQA0NvdkD+Qp2E7+TxWjkkCQGIdiYAC0mBFOnwnNgmplUCsQhWkTqomH+kvKLFGdC8iAqJcjkVMMNOF8onLJXVsLtVURRF/DLzuAKzBwwg5zgHSUEdaOySQsF44/Nmi8xBR1f/cmWK9Y2pqIaaxnFRJFSREbSdRv20dABUgogERc1mtz8nxNmSTt8HcGz7fG59L3noz7fzzUgBiCo/MJe4KvFSQtRLWQC1y3J1ViVUfaYwVd6oB6ZNSerUmvvJB0Vj6/XAgCglSOWYxIZd6yyHvWAsJgHxrxzLgBQMxFM27pJBIAqvur4WOrziiybapfIedeS86dO5xTkZARE48zlYHRIsjopQZkT0g5tnWd3YuBdxM4sOzRg6kYKqWS61yqPr+XsS+rzd9EUvcjIjizWZEUl4xubb13V7EkQkJMNKcFz7fr/xle/CNBQ2GyBpAauKzbYhqIL65axlxt4w4giFRigxqL/LCiGUXCxFlm33izISV3ty5nLgnBGizRjykDJhWRerVNXjvrLlZY6qMxA69yJYRJZ+tDJVcoKJFmsTCcwmJhW0uIEqZ715uof/OSiM+DsT2OZcNJJLfFTVp8ku3JqXMFiDPq5QAR5qksxWhYlRImkS0spAEfREJhoJdihGRd0y2eUbSJjqxbyGgZo1IyN8DMgyQDwLwARQpZX+c5VKQqTwSj/AFKokUktFKqWJU2LfEAqk2ZYgoLovvfTTXnPTCGuLu0Dv56Oz3pAApdzjMXmgdH56dyZ1BUy8i5LYFgql2J94RvxnSKY7R9QMIpYgyZI455cVMTFuoGogMQHthDEQUynxISif1Nz6q28ituonDAnnlFSz91r5BfKyq3ImjMNcRDS4hoq1873Kz2bgekJ2hqv4R7ZBtqDVE6gNuDriRGzyJG7yklEP+Rttu6zp0k//RGzXJ25u0Tp5su/VnnwJETewWTCQgXwDkMZ1qEERkQ0QzmVtmvMXtYeO4cy5x5xzlkOLaclvC0xVjxccrXlOOiICIFCAixYjnQT6fYYeUEdEgiOjgDOaO8NXcGQnIhcohvaPfJi8krNyp16ZvG0bEAjcgZl0IUYZclm2HDFENEREFyzVP/44bZ97OjaHKIafEvUYe+p03Cqm1X7mGWOICYp4LiK+yJuZVO+RqO2TokBExkXIt057lVn/lkL76OhKTa/XDf8eiHBENhIg2AzIry/gcIDsAORAickTERGKdfpw/DpmoGDLAtONK39Sa+ssZUYbc+qxpfRpxV5DKScTeWqeFKIbEFepdQPzJOUTlJ9vKELcOgIiiN/Pdj2QtfT7LAMjAIEWIB6ahwMcVQ6qN1Q8pRRwBkwjETSzEbPzhnqVZd+yofIS4Y/piRYgyZJ1iSE1C3TM4fLicEdnzka2FwQkb6Eyrjrh9+quAHBAR9UXcH4Q16JBiSDGturJfRKPbXraHjhjn+Dhv6Q8RbWQhdsPn/xLy51vWrVtH3BN2LR7rT4Hn3CQCUS7wK+WQpurdIw8RDYIoxNgQT4y7udvPU91K37T9UTrdWeECInNT4BnFkEDsuNwmEf2kjtpUPum/cm69Y9Ixaq6Nle6pf4/d+qTTj3MvIgo47QJk5Ul3ITp+H6Lv1QmWMz7xltMqqbg8e7mnVbG5p71jc+RizCjbXtZp72iprXJRW6ROoW6PZVnvTlmatfa2Za96CVGbKP2FdPq8EZPYHXwdN4e85yIi8z4XIIF4ymFTcRXxiCqh4OnJcSXTPGOKbhXiLdcHGbN+FZKy5Vq5V23NNG5Gm+SSN9p75dqZSVIb5Fa+LJf48sQ5SS9evSBlPa14/Cl6yRJPHR8tIG6fRnws2B+by/6+iMg5ROTvCmTFKdfuWNBKK2uTLN+rDNaEyTFF40PTsujp3CforTeW05E9f6IvWh6gbw5HXvNN2/3j8V3WW8sCFNHbQXRAaj4Kl9sfPv7rpvDx3x2YN+GHllnXcnvIDbjUmsLHZjzCbTN3YXf+uRcRDYyI+kV0GfKEYHThogqImrjib6c8tGOmlz6fXirMoNOHFxGfmHcDfzprKX8yt5x77mvm7vuOc/e8k9w1t7fOOWj2ST52tlknucNee6jcUSndST6iO8GHsR7SncEE/mR7T0TuQ0SNfsohNckVHUoR5azsFZXzJ48l+fT6+7HEn88fwz3hqdw9/wR3zWfuCmfulLqP+djZ5jF3SM1lbpeaw3xUajY673T7cJi9fn47IzciSrkykWV7BkcsvDAiTrbVBsuuSbE5lPN+IvGP8yZyV0SdBOgUInICEbkb0Q7Z4Kv89Uelr6wRh7SxFDgiIit7JhTMi35hPfFni67g4wtKHBHlnEJETiIi1xGlfL9UPpHG8pcVvyfqrZ/5RBdOfP+1GOKeuX8cXkTkOiLqFxH/LR5UDIlpjFJ6KOuRYKkKW/My/a1jAXF3+F/skCMcEfVFlNvjW6oY0stQ7jv0SZQB5Sx8z4rCZ6Ofe5b45Lw/YHP5HpN4+SLuRc1eBuWQcTUThaSS404iojycAZalP7l5LSDnBg2GiJxARM4hIlcQtVI/836Nh2JIbWIRiSuLtzqFaL+omhxdkP6SNZ3409nhShGduPFTgIj6IiJHRLRHfA/rlYohfZMKSJtcMF1IKnT6ynRyVEH6+rxVxMdnR4wYRKQAEWkiESmG9IkuJ5+oijHq+NK3HV5xkHXAQ1kZMh2QsyIuZ0Ss73Ozz1ikHPKOhytRBU2OLvbVGi0/OTzOAx2FATK/F9IJROQ2ROQSIvqGP/Kbwh8EEn8Q4MJEJrwuF/8XUkXXrQLeoJMIRBlyhTV9fU4a8SeAHBgROYmInEBESiZR8z2/GXwfVywgrpovpRxyzNU/n+uXN54mTXzBBjHRMqRD2Xsetaa/YIMMixgYEV0qxIb+JlFo5wYhmN8IIS5+gLh0kZR7/jXaL379VxKii0kdVxCr0Vs+EwwDnmwD0pL5fE4qcQ8gXUNEM4fwOCPXJ7GN67UZvFe4jhs1ZIMsiSQuWyjlHsjxvzljg/SJKSLP6JKbfGIL9KI+pxqIuwFYj3YDcbfGVnY9/qnx4ufNEmRohEuIB3V/54awZEAuBqTcwWA0YzHwzgVAWwC0F7gYiChgMRBtARH52QIgVi0SlvBe8T94j9aHd2vHApIASRcF0iOqlLwfLSVRbyZAohxkJkCSBqnjsmnqihz68PWlxB9jIpUiSjWH/C/XLpjATTri1mDiFtQ8gwCJdVpv+9GBIKxnCyRAEgCx+uKzSIBEfrYAiVXEn2sIkARIAiRdfMgVJf1C3rXcQtHPrCXunEXcMSdC8XciHmlA/p33hD0KwHDg2cIE2gsKB57cfqnA3poCwnkfagyYz/sFLz7oCVAZ87KAnBJloVmmjdS1N5K4K4yAGIGDWdc2ltbeiyqk5DuxCJG9kQ0pJvQidu6JJO7WEQDR7AjnT7ddvvFzPN3OuywgRUBOjbbQbNMmO2KIBHi2iEuLiBr88hD9Y75AHDmQQMwFopXCTJups34RcVcIAez8IpQjoiEhBnyJmoD4eV9EXykLoj41akcO5JTofJqduom6JMTOmTKeI+SAiApu/Hp/9mH1N/MB7U3c5H8FAG/A5vJS3/dEXyveE6nfGi4x5NQYK81NlxAXYhJ7ER1a4EZEdBYRHfA7yEemjuFWFQGSAAmEQMD47XZ42R4YEl0yyDuXVFJkxov0aUME4UKeANRfC92MiAJlyGbfV7gNiG0+EqR9E/FHfk85QObzHpEGTkK8yJC3Lqmi/8zcSKda5hJ/DMR2gHXo+tYuFRoAwOG6Ni22TeE+CRA1AqHVC5/FLbz3H376FSEaWg6Q8m9t90Kqo0po0pJyin76Ofq2ZRawggmTNki6idyq6x6Oa1NM4dd8SOXBTfbNYw8wWz1v533Cl4DsPYSoF1+RH1sn+9Cf+N0g4vcCiHfMdg8kjfuWVPhZuL40g374XEd8CojS+kXIIOmIPwldwS0hw3FtinxPYiKTudF3DiANXO/f43CmiDQ6RIraqyZu8CF+Tec65HNJefRWxRo60qAnPr2c+KsVxF9GDbFogK+4io/NL8FvZRnQdcShHYfVY92nsvLhu65A5FK7Ql2H/Kh2HTEbUQbKRGucKBM9CdAl43DYkO06IhocUW63WMrv+0/gt6cRvxOkvEN3u2ci3yx7npjTUabCnsAj/jDxgRmER/whALY6h4icQ+zB4xjDH2qvACTxW9MJmMpru2cEQjbriA8FTuCWoAeAWAHA0zKky4jfAvA1blI9yg3q6xHxR1r6J4cMwGQGEiCxhvyWDwbqcJ64EohbgLgTNQDyCBB7gHjC/tPvBBB7gHgE7cWrzi6UzfW+KbxPPQeIf0DEzZ7EjT40oiAdG+3yhxyFHIUcbRRyFHIUcrT/B7DbG7t6BVpAAAAAAElFTkSuQmCC","readonly":false,"pythonVersion":"python3","availableOutputVariables":{"$type":"System.Collections.Generic.Dictionary2[[System.String, System.Private.CoreLib],[Core.Models.Integrations.Descriptors.NameDescriptor, Core]], System.Private.CoreLib","sw_task_status":{"$type":"Core.Models.Integrations.Descriptors.NameDescriptor, Core","name":"Task status","description":"The result of task execution","outputType":"text","disableEdit":true,"isSystem":true},"sw_task_error_type":{"$type":"Core.Models.Integrations.Descriptors.NameDescriptor, Core","name":"Task error type","description":"In case of an error, this is the error type","outputType":"text","disableEdit":true,"isSystem":true},"sw_task_error_message":{"$type":"Core.Models.Integrations.Descriptors.NameDescriptor, Core","name":"Task error message","description":"In case of an error, this is the error message","outputType":"text","disableEdit":true,"isSystem":true},"sw_task_stack_trace":{"$type":"Core.Models.Integrations.Descriptors.NameDescriptor, Core","name":"Task error stack trace","description":"In case of an error, this is the error stack trace","outputType":"text","disableEdit":true,"isSystem":true}},"availableOutputTypes":["insertUpdateRecord","email","referentialTask","saveToFile","setFieldValue"],"inputParameters":{"$type":"System.Collections.Generic.Dictionary2[[System.String, System.Private.CoreLib],[Core.Models.Integrations.Descriptors.Descriptor, Core]], System.Private.CoreLib"},"meta":{"$type":"System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]], System.Private.CoreLib"},"createdDate":"0001-01-01T00:00:00Z","modifiedDate":"0001-01-01T00:00:00Z","id":"aNs5dV5VVv6nmmzCf","name":"Python 3","disabled":false},"readonly":false,"script":"# importing requests module\nimport requests\n \n# initializing URL\nurl = "https://www.swimlane.com\"\nproxy = {'http_proxy' : '', \n 'https_proxy' : '', \n 'no_proxy' : ''}\nverify = False\ntimeout = 60\n\ntry:\n # requesting URL\n request = requests.get(url,\n proxies=proxy,\n verify=verify,\n timeout=timeout)\n print("Internet is on")\n # catching exception\nexcept (requests.ConnectionError,\n requests.Timeout) as exception:\n print("Internet is off")"},"inputMapping":[],"isSystemTask":false,"outputs":[],"triggers":[],"createdByUser":{"$type":"Core.Models.Utilities.UserGroupSelection, Core","id":"aM78aN_Sl2M41apNJ","name":"Swimlane Admin"},"modifiedByUser":{"$type":"Core.Models.Utilities.UserGroupSelection, Core","id":"aM78aN_Sl2M41apNJ","name":"Swimlane Admin"},"createdDate":"2022-10-27T06:19:25.898Z","modifiedDate":"2022-12-06T07:53:51.137Z","valid":true,"uid":"test-internet-connection-51877","version":2,"id":"aOB2H6aHClvITb8lv","name":"Test Internet Connection","disabled":false}