I am working on integrating the Camunda Platform 8 into our SaaS platform, based on Tanzu Kubernetes Grid Integrated Edition (former PKS) and Swisscom Helix (shared services). The most important issue was the missing access security for Zeebe, the workflow execution engine. This article shows how simple and powerful the authentication and authorization configuration for Istio is. We will create roles and permissions with Camunda 8 Identity and use them with the Camunda Desktop Modeler.

Architectural Overview

A short introduction can be viewed here:

More details on the Istio configuration will follow, but let’s start with Camunda Identity:

Use Camunda Platform 8 security concepts

Create permissions per gRCP call:

Create Application in Camunda Identity

An application is a OAuth client. Let’s start with /identity/applications and add a new application:

After the application is created you can get the client ID and secret here:

And now some magic: assign permissions from the previously defined API. You can select the API from the dropdown list.

Test the OAuth client with postman

Postman has a nice OAuth 2.0 integration that automates the token generation:

add the audience “workflow-manager”:

Now you should be able to successfully login to keycloak and get a valid JWT token.

Parse your JWT token

https://jwt.io/

You should find something like this:

"iss": "https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform",
...
"permissions": {
    "account": [
        "manage-account",
        "manage-account-links",
        "view-profile"
    ],
    "workflow-manager-api": [
        "DeployProcess",
        "CreateProcessInstance",
        "Topology"
    ]
},

Use OAuth 2.0 with Camunda Desktop Modeler

Now we are ready to use the newly created OAuth client with the desktop modeler:

Before you can successfully use the Desktop Modeler if you use self-signed CAs for your certificates, you have to start the client with this powershell script in the directory where the exe is located – or add the environment variables otherwise to your OS:

$env:ZEEBE_NODE_LOG_LEVEL = "DEBUG"
$env:NODE_TLS_REJECT_UNAUTHORIZED = "0"
$env:NODE_DEBUG = "http,http2,pusher-js-aut,zeebe-node,pusher-js,pusher,node"
& '.\Camunda Modeler.exe'

This starts the clients and shows the output in the shell what is pretty handy for debugging problems. The client shows some errors without clear messages.

I completely deactivated TLS security after I tried the official environment variables for the node zeebe client:

$env:ZEEBE_CA_CERTIFICATE_PATH = "C:\Development\camunda-modeler-5.5.0-win-x64\trusted.crt"
$env:ZEEBE_SECURE_CONNECTION = "true"
$env:ZEEBE_CLIENT_SSL_ROOT_CERTS_PATH = "C:\Development\camunda-modeler-5.5.0-win-x64\ca.crt"
$env:ZEEBE_CLIENT_SSL_PRIVATE_KEY_PATH = "C:\Development\camunda-modeler-5.5.0-win-x64\tls.key"
$env:ZEEBE_CLIENT_SSL_CERT_CHAIN_PATH = "C:\Development\camunda-modeler-5.5.0-win-x64\trusted.crt"

This didn’t work for me on Windows 11.

But anyway, with TLS deactivated I could successfully use the OAuth login:

Implement Camunda Platform 8 security concepts with Istio

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app: helix-ingress-gateway-internal-camunda
    app.kubernetes.io/component: gateway
    app.kubernetes.io/instance: helix-ingress-gateway-internal-camunda
    app.kubernetes.io/part-of: istio
    istio: helix-ingress-gateway-internal-camunda
  name: internal-camunda
  namespace: helix-ingress-gateway-internal-camunda
spec:
  selector:
    istio: helix-ingress-gateway-internal-camunda
  servers:
  - hosts:
    - '*.camunda.tanzu.ch'
    port:
      name: http-internal-camunda
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: false
  - hosts:
    - '*.camunda.tanzu.ch'
    port:
      name: grpc-internal-camunda
      number: 443
      protocol: HTTPS
    tls:
      credentialName: internal-camunda-gateway-credential
      mode: SIMPLE

This component is deployed by the helix operation team as a dedicated ingress gateway per customer.

Istio Virtual Services

These are the virtual service manifests for the whole setup:

• camunda virtual service
• zeebe virtual service
• webmodeler virtual service
• webmodeler api virtual service
• webmodeler ws virtual service

The host names are defining the split into multiple virtual service files. Check out the Istio Service Entry as well.

For the RBAC configuration, we focus on the zeebe components and start with the zeebe virtual service – separated from the camunda virtual service because of the two fqdn approach from the camunda helm chart “combined ingress”.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: 
  name: camunda8-zeebe
spec:
  exportTo:
    - '*'
  gateways:
    - helix-ingress-gateway-internal-camunda/internal-camunda
  hosts:
    - 'zeebe.camunda.tanzu.ch'
  http:
    - match:
        - port: 443
          name: grpc-internal-camunda          
      route:
        - destination:
            host: camunda-zeebe-gateway
            port:
              number: 26500

Important are now the labels of the camunda-zeebe-gateway:

apiVersion: v1
kind: Service
metadata:
  name: "camunda-zeebe-gateway"
  labels:
    app: camunda-platform
    app.kubernetes.io/name: zeebe-gateway
    app.kubernetes.io/instance: camunda
    app.kubernetes.io/part-of: camunda-platform
    app.kubernetes.io/version: "8.1.5"
    app.kubernetes.io/component: zeebe-gateway
  annotations:
spec:
  type: ClusterIP
  selector:
      app: camunda-platform
      app.kubernetes.io/name: zeebe-gateway
      app.kubernetes.io/instance: camunda
      app.kubernetes.io/part-of: camunda-platform
      app.kubernetes.io/component: zeebe-gateway
  ports:
    - port: 9600
      protocol: TCP
      name: http-camunda-internal
    - port: 26500
      protocol: TCP
      name: grpc-camunda-internal

The label app.kubernetes.io/component: zeebe-gateway is subsequently used to attach Istio authorization and authentication policies.

Istio Service Entries

Some of the configurations offer internal service urls. Sometimes this was not always the case and therefore we had to setup Istio a ServiceEntry for this traffic:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-svc-https
spec:
  hosts:
  - camunda.camunda.tanzu.ch
  - zeebe.camunda.tanzu.ch
  - webmodeler.camunda.tanzu.ch
  - webmodeler-api.camunda.tanzu.ch
  - webmodeler-ws.camunda.tanzu.ch
  location: MESH_INTERNAL
  ports:
  - number: 443
    name: https
    protocol: TLS
  resolution: DNS

Istio RequestAuthentication

In order to protect traffic, Istio can be configured to enable JWT token verification. This can be done very easily based on the keycloack oauth 2.0 provider endpoints:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: "jwt-camunda-zeebe"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: zeebe-gateway
  jwtRules:
  - issuer: https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform
    #jwksUri: https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform/protocol/openid-connect/certs
    jwks: |
      {"keys":....}
    audiences:
      - camunda-platform
      - workflow-manager
      - workflow-manager-api

Very important hint: we had to use the jwks: property followed with the whole content of the jwksUri. If you use self-signed CA authorities that are not known to Istio, you cannot use the url directly. Kudos to this blog.

As you can see, the RequestAutentication is bound by its selector directly to the zeebe gateway component. The issuer will be used in further configuration to define users from this IAM.

We have restricted the JWT to the given audiences.

Istio AuthorizationPolicy

Based on the RequestAuthentication definition we can now define rules and apply them to certain contexts. In this case, again, we use the label to identify the zeebe-gateway components.

AuthorizationPolicy Collection

Default DENY

This rule denies any unauthenticated user access:

kind: AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
metadata:
  name: ext-authz-oauth2-keycloak
  namespace: camunda8-dev
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: zeebe-gateway
  action: DENY
  rules:
    - from:
        - source:
            notRequestPrincipals: ["*"]
      to:
        - operation:
            hosts: ["zeebe.camunda.tanzu.ch","zeebe.camunda.tanzu.ch:443"]

Default ALLOW for some paths

This was necessary, have to investigate why. It should be bound only to zeebe-gateway but it had side effects on camunda identity as well.

kind: AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
metadata:
  name: ext-authz-oauth2-keycloak2
  namespace: camunda8-dev
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: zeebe-gateway
  action: ALLOW
  rules:
    - from:
        - source:
            notRequestPrincipals: ["*"]
      to:
        - operation:
            hosts: ["zeebe.camunda.tanzu.ch","zeebe.camunda.tanzu.ch:443"]
            notPaths: ["/auth/*","/identity/*"]

RBAC ALLOW – the job we want to do!

Very handy solution on this protocol level. Remember the permissions we have created in camunda identity:

Create individual permissions per zeebe API call:

We see it in the JWT like so:

"iss": "https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform",
...
"permissions": {
    "account": [
        "manage-account",
        "manage-account-links",
        "view-profile"
    ],
    "workflow-manager-api": [
        "DeployProcess",
        "CreateProcessInstance",
        "Topology"
    ]
},

Attention to the syntax of the rules blocks:

I am using from: followed by to: followed by when:. Take care not to miss the dash “-” that indicates the beginning of a new rule.

First, we check in the from: clause that we have users authenticated by our keycloak installation by using the iss: claim from the JWT:

- from:
    - source:                
        requestPrincipals: ["https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform/*"]

Authenticated users are identified by iss/sub, in this case sub is replaced by “*” meaning all users from this issuer. You could add an explicit user-id (usually a UUID) as well.

Next, we use the to: to create useful business functions:

to:
  - operation: 
      hosts: ["zeebe.camunda.tanzu.ch","zeebe.camunda.tanzu.ch:443"]
      paths: ["/gateway_protocol.Gateway/Topology"]
      methods: ["POST"]

Have you seen? No dash at the beginning of to:! It is part of a block describing a rule. If you set a dash, you create a new rule…

OK, this rule matches to a host pattern (both entries are needed!), paths and methods. This one filters the /gateway_protocol.Gateway/Topology call on POST (grpc uses POST).

Now we can create when: conditions like

when:
- key: request.auth.claims[permissions][workflow-manager-api]
  values: [ "Topology" ]

meaning if a user has the right permissions this rule will apply. As you can see, this expression is also capable to evaluate nested maps also for combinations of values.

And here the full manifest:

kind: AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
metadata:
  name: ext-authz-oauth2-keycloak3
  namespace: camunda8-dev
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: zeebe-gateway
  action: ALLOW
  rules:
    #/gateway_protocol.Gateway/Topology
    - from:
        - source:                
            requestPrincipals: ["https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform/*"]
      to:
        - operation: 
            hosts: ["zeebe.camunda.tanzu.ch","zeebe.camunda.tanzu.ch:443"]
            paths: ["/gateway_protocol.Gateway/Topology"]
            methods: ["POST"]
      when:
      - key: request.auth.claims[permissions][workflow-manager-api]
        values: [ "Topology" ] 
    #/gateway_protocol.Gateway/DeployProcess
    - from:
        - source:
            requestPrincipals: ["https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform/*"]
      to:
        - operation: 
            hosts: ["zeebe.camunda.tanzu.ch","zeebe.camunda.tanzu.ch:443"]
            paths: ["/gateway_protocol.Gateway/DeployProcess"]
            methods: ["POST"]
      when:
      - key: request.auth.claims[permissions][workflow-manager-api]
        values: [ "DeployProcess" ] 
    #/gateway_protocol.Gateway/CreateProcessInstance
    - from:
        - source:
            requestPrincipals: ["https://camunda.camunda.tanzu.ch/auth/realms/camunda-platform/*"]
      to:
        - operation: 
            hosts: ["zeebe.camunda.tanzu.ch","zeebe.camunda.tanzu.ch:443"]
            paths: ["/gateway_protocol.Gateway/CreateProcessInstance"]
            methods: ["POST"]
      when:
      - key: request.auth.claims[permissions][workflow-manager-api]
        values: [ "CreateProcessInstance" ]

Wrap up

This is a good sample for Istio in action: Just define your RequestAuthentication and integrate your token endpoint. With the AuthorizationPolicy you can define your business rules. In this case Istio enables to add fine granular access permissions without any code change or other impact on the component (zeebe). This creates dependencies on the strings used for the API client and the permissions configured within Istio but it is very handy especially for the given case.

Keep in mind self-signed CAs!

I plan to release the camunda platform 8 kustomize project for Istio. Stay tuned!