Troubleshooting

Troubleshooting

1. How can I resolve 'Token not authorized' errors in ChkkAgent pods?

Answer: These errors typically indicate that your token is either invalid or missing. If you are using a secret-based approach, verify that the secret contains a valid token. If you are installing via Helm, ensure that the Helm chart is upgraded using a valid token.

1. Export a valid Access Token

   Login to Chkk Dashboard and export a valid Access Token.

   
   
2. Upgrade the Helm chart with your valid token

   helm upgrade chkk-operator chkk/chkk-operator --namespace chkk-system --set secret.chkkAccessToken=<ACCESS-TOKEN>
   

   Sample Output

   Release "chkk-operator" has been upgraded. Happy Helming!
   NAME: chkk-operator
   LAST DEPLOYED: Thu Aug 17 19:31:58 2023
   NAMESPACE: chkk-system
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   

3. Confirming the agent-manager pod status

   Use the command below to ensure the pod is running:

   kubectl get pods -n chkk-system
   

   Sample Output

   NAME                                            READY     STATUS    RESTARTS     AGE
   chkk-operator-chkk-agent-3kjfqe00fqpe-atpoiks   1/1       Running        0       4d13h
   

If these steps do not solve your issue, please reach out on your private Slack or MS Team Channel or email [e-mail].

2. How do I use an existing Service Account with the Chkk Kubernetes Connector?

Answer: Please refer to the Chkk Kubernetes Connector documentation for instructions on how to use an existing Service Account.

3. How can I use an existing secret with the Chkk Kubernetes Connector?

Answer: Please refer to the Chkk Kubernetes Connector documentation for instructions on how to use an existing Secret.

4. Why can't I create JIRA tickets for Operational Risks?

Answer: Make sure that the Jira project and issue type do not have any required custom fields. Chkk by default only supports providing the default fields to Jira. If you require the use of mandatory custom fields, contact us on Chkk support Slack/MS Team Channel or email us at [e-mail].

5. How do I ignore specific Risks using my Infrastructure as Code (IaC)?

Answer: You can ignore Risks by adding the chkk.io/ignore annotation to your Kubernetes resources in your IaC.

1. Ignoring all Risks

   Use a wildcard (*) in the annotation:

   yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     annotations:
       chkk.io/ignore: "*"
       deployment.kubernetes.io/revision: "1"
       meta.helm.sh/release-name: traefik-1
       meta.helm.sh/release-namespace: traefik-ns
   

2. Ignoring specific Risks

   Specify the ARSig IDs you wish to ignore:

   yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     annotations:
       chkk.io/ignore: CHKK-K8S-1002,CHKK-K8S-602
       deployment.kubernetes.io/revision: "1"
       meta.helm.sh/release-name: traefik-1
       meta.helm.sh/release-namespace: traefik-ns
   

6. How do I set Cluster Name and Environment using the ChkkAgent CRD?

Answer: Please refer to the Chkk Kubernetes Connector documentation for guidance on configuring the Cluster Name and Environment using the ChkkAgent CRD.

Alternatively, you can update these settings via the Chkk Dashboard by navigating to Risk Ledger > Clusters and clicking Edit on the relevant cluster card, or by modifying the values in the cluster’s Properties tab.

Note: If the Cluster Name or Environment is defined through Infrastructure as Code (IaC), it cannot be modified from the Dashboard.

7. How do I specify Cluster Name and Environment with the Terraform module?

Answer: Please refer to the Chkk Kubernetes Connector documentation for guidance on configuring the Cluster Name and Environment using the Chkk Kubernetes Connector Terraform Module.

Alternatively, you can update these settings via the Chkk Dashboard by navigating to Risk Ledger > Clusters and clicking Edit on the relevant cluster card, or by modifying the values in the cluster’s Properties tab.

Note: If the Cluster Name or Environment is defined through Infrastructure as Code (IaC), it cannot be modified from the Dashboard.

8. How do I manually clear the finalizer from the ChkkAgent custom resource?

Answer: When a Kubernetes custom resource is deleted, any configured finalizers must be cleared before the object is fully removed. If a finalizer is misconfigured or cannot complete its cleanup, the resource remains stuck in the terminating state. To force-remove the finalizer and allow the deletion to complete, run the following command:

kubectl patch chkkagent/chkk-agent -n chkk-system -p '{"metadata":{"finalizers":null}}' --type=merge

This command manually clears the finalizer from the metadata, allowing the resource to be removed successfully.

9. How do I fix 'failed to determine if *v1.ConfigMap is namespaced: Forbidden' errors in ChkkAgent pods?

Answer: This error commonly indicates that a proxy server or firewall is blocking requests to the Kube API Server. Verify that your Kube API Server address is allowlisted or permitted within your network’s proxy/firewall configurations.

Example log snippet:

2024-06-26T18:19:47Z ERROR setup unable to start manager {"error": "failed to determine if *v1.ConfigMap is namespaced: failed to get restmapping: failed to get server groups: Get \"https://172.20.0.1:443/api\": Forbidden"}

10. How do I fix 'Get <CHKK_API_ENDPOINT>/v1/connector/k8s/config: Forbidden' errors in ChkkAgent pods?

Answer: This error is likely caused by your proxy server or firewall blocking traffic to and from the “chkk.io” domain. The ChkkAgent needs to communicate with the Chkk API to sync the cluster state. To fix this issue, you need to allowlist the “chkk.io” domain and its subdomains in your proxy server or firewall.

11. How do I uninstall the Chkk Operator to remove a Cluster?

Answer:

1. Deactivate the Cluster in the Chkk Dashboard

   In the Dashboard, deactivate the cluster you want to remove.

2. Remove Custom Resources

   1. List all ChkkAgent resources:

   kubectl get chkkagent --all-namespaces
   

   2. Delete all ChkkAgent resources:

   kubectl delete chkkagent --all --all-namespaces
   

3. Remove Chkk Operator (Helm-based)

   1. Check installed charts:

   helm ls -n chkk-system
   

   2. Uninstall the chart:

   helm uninstall chkk-operator -n chkk-system
   

   3. Delete the namespace:

   kubectl delete ns chkk-system
   

4. Remove Chkk Operator (K8s YAML-based)

   1. List resources in chkk-system:

   kubectl get all -n chkk-system
   

   2. Delete Operator resources:

   kubectl delete -f https://helm.chkk.io/chkk-operator/manifest.yaml
   

   3. Delete the namespace:

   kubectl delete ns chkk-system
   

5. Remove the ChkkAgent CRDs

   Finally, remove the CRD:

   kubectl delete crds chkkagents.k8s.chkk.io
   

12. Why is my cluster stuck in onboarding or showing no or single-digit operational risks in Risk Ledger?

Answer: This can happen due to a few common misconfigurations: either the Chkk Agent RBAC is incomplete or incorrect, explicit filter rules (especially wildcard-based) are excluding key namespaces, or Chkk API endpoints are not reachable due to network restrictions.

1. Ensure Chkk Agent RBAC is correctly configured

   ChkkAgent requires specific Kubernetes permissions to access resources for analysis. Please ensure you are using the RBAC definitions provided with the official Chkk Operator Helm Chart or Kubernetes Manifests. Missing or custom-modified roles/clusterroles may cause incomplete onboarding.

2. Audit filter rules used to exclude namespaces

   If you have applied filter rules to exclude namespaces, review them carefully—especially if you’re using a wildcard (e.g., *). Wildcard exclusions can unintentionally block all namespaces from being scanned, resulting in no or limited coverage.

3. Verify network connectivity to Chkk API endpoints

   The Chkk Agent must be able to communicate with Chkk’s API services. Ensure your firewall or proxy settings allowlist all the domains listed in the Chkk Operator prerequisites documentation.

4. Wait for the next scheduled scan cycle

   Once any misconfigurations are resolved, the Chkk Agent will pick up the changes during the next scheduled scan. The cluster should then get onboarded.

If the issue persists after 24 hours, please reach out to your Chkk support contact for further investigation.

13. Why is my Chkk Operator failing with 'x509: certificate signed by unknown authority' when connecting to *.chkk.io?

Answer: This error often occurs when a proxy in the network is intercepting HTTPS traffic. Specifically, if you use Squid Proxy with SSL Bump enabled, the proxy acts as a proxy-in-the-middle and presents its own certificate instead of the actual server certificate. Since this certificate is not signed by a known Certificate Authority (CA), Chkk system refuses the connection due to failed certificate validation.

1. Understand the root cause

   Squid Proxy with SSL Bump intercepts encrypted traffic and re-signs it with an internal/self-signed CA. Chkk system does not trust this certificate by default, which causes the error:

   tls: failed to verify certificate: x509: certificate signed by unknown authority
   

2. Skip SSL Bump for Chkk domains in Proxy

   To allow Chkk Operator to establish a secure connection without interference, please configure Squid Proxy to skip the SSL Bump. This will allow Chkk Operator to use its own certificates.

If this does not resolve the issue, please contact your internal network/security team to confirm whether SSL Bump is still affecting .chkk.io traffic, and reach out to [e-mail] for further assistance.

14. How do I override default container images during cluster onboarding?

Answer: Both the Chkk Operator and the ChkkAgent Custom Resource Definition (CRD) support overriding default container images.

Default Images:

• Chkk Operator: public.ecr.aws/chkk/operator:<VERSION>
• ChkkAgent:
  • Agent Manager: public.ecr.aws/chkk/cluster-agent-manager:<VERSION>
  • Agent: public.ecr.aws/chkk/cluster-agent:<VERSION>

1. 1. Create the namespace

   kubectl create ns chkk-system
   

2. 2. Add the Chkk Helm repository

   helm repo add chkk https://helm.chkk.io
   helm repo update chkk
   

3. 3. Install the Chkk Operator with a custom image

   helm install chkk-operator chkk/chkk-operator \
     --namespace chkk-system \
     --set secret.create=true \
     --set secret.chkkAccessToken=<ACCESS-TOKEN> \
     --set image.repository=<REPOSITORY_URL> \
     --set image.tag=<TAG>
   

4. 4. Create a ChkkAgent resource with custom images

   kubectl apply -f - <<EOF
   apiVersion: k8s.chkk.io/v1beta1
   kind: ChkkAgent
   metadata:
     name: chkk-agent
     namespace: chkk-system
   spec:
     agentOverride:
       image:
         name: <AGENT_IMAGE_URL>
       managerImage:
         name: <AGENT_MANAGER_IMAGE_URL>
   EOF
   

15. How do I activate a cluster that was previously deactivated?

Answer:

1. Navigate to Configure > Settings > Clusters > Deactivated Clusters in your Chkk Dashboard.
2. Locate the cluster you wish to restore and select Activate Cluster.
3. After activation, the cluster will reappear in Risk Ledger and in the Artifact Register, allowing normal management.

16. How do I troubleshoot common issues with the Chkk MCP Server?

Answer:

Here are solutions to the most common Chkk MCP Server issues:

• Authorization error (e.g., Received Login Error. Code: 403 Body: {"error":"forbidden","message":"Not authorized"}):
  • Make sure your AWS credentials are valid and the associated IAM Role or User has been added to Chkk.
  • Double-check that your AWS identity has been added to Chkk.
• No risks returned:
  • Ensure your cluster ID is correct and you have access.
• Still stuck?
  • Check logs in the Cursor output panel or run the server manually for debug output.