Skip to main content

DetectFlow Requirements and Deployment Guide

N
Written by Nataliia Pukaliak

High-Level Architecture

Requirements

Technology stack:

Network and infrastructure:

  • Outbound internet access allowed:

    • to *.socprime.com for API access to the SOC Prime Platform for synchronizing detections

    • to *.amazonaws.com, *.apache.org for pulling Docker Images

    • to *.hub.docker.com for pulling Docker Images

    • to *.github.com for pulling open-source detections

  • Internal access to deployed resources

  • Network connectivity between deployed resources

  • Kubernetes cluster with at least 3 nodes(workers) (min 8 vCPUs, 32 GiB RAM each). For a more detailed calculation, see Appendix A

Delivery options:

  • An installation package (a set of YAML files) to be downloaded from an online resource

  • An installation package (a set of YAML files and Docker images) to be copied from a physical drive on-site

Deployment Process

Deployment should be performed by a person with expertise in Kubernetes and admin access to the Kubernetes Cluster provided for this project.

Deployment consists of four parts:

  1. Apache Flink (according to its official deployment recommendations)

  2. PostgreSQL

  3. Apache Kafka

  4. DetectFlow itself

Apache Flink Deployment

  1. Install Cert Manager according to https://cert-manager.io/docs/installation/

  2. Run the following command for creating NameSpace:

    > kubectl create ns flink

    Out: namespace/flink created

  3. Deploy PostgreSQL for DetectFlow

    Run the following command:

    > kubectl apply -f postgresql.yml -n flink

  4. Deploy Apache Kafka (One node) for DetectFlow

    Run the following command:

    > kubectl apply -f kafka-kraft.yml -n flink

  5. Get acquainted with the key concepts of Apache Flink Kubernetes Operator here: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/

  6. Deploy Apache Flink Kubernetes Operator using Helm.


    A Helm chart manages the installation of the operator. To install the operator (and also the Helm chart) to a specific namespace, run the following command:

    > helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.13.0/

    Out: "flink-operator-repo" has been added to your repositories

    > helm repo update

    Out: Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "flink-operator-repo" chart repository

    ...Successfully got an update from the "bitnami" chart repository
    Update Complete. ⎈Happy Helming!⎈

    > helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator --namespace flink-operator --create-namespace

    Out: NAME: flink-kubernetes-operator
    LAST DEPLOYED: Thu Feb  5 12:07:08 2026
    NAMESPACE: flink-operator
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None

    Source: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/helm/

  7. Deploy ConfigMap with static configuration for Flink Sigma Detector.

    Open the flink-configmap.yaml from the installation package and specify values for the following parameters:

    • namespace: flink

    For all the other parameters, keep the default values.

    Run the following command:

    > kubectl apply -f flink-configmap.yaml -n flink

    Out: configmap/flink-config created

  8. Deploy Secrets for Flink Sigma Detector.

    Open the flink-secret.yaml from the installation package and specify the Base64 values for the following parameters (if needed, either leave by default):

    • KAFKA_BOOTSTRAP_SERVERS:

    • KAFKA_AUTH_METHOD:

    Run the following command:

    > kubectl apply -f flink-secret.yaml -n flink

    Out: secret/flink-secret created

  9. Deploy PersistentVolumeClaims for Flink State Storage.

    Open the flink-pvcs.yaml from the installation package and specify values for the following parameters in each name: block:

    • namespace:

    • storage:

    • storageClassName:

    Run the following command:

    > kubectl apply -f flink-pvcs.yaml -n flink

    Out: 

    persistentvolumeclaim/flink-checkpoints-pvc created
    persistentvolumeclaim/flink-ha-pvc created
    persistentvolumeclaim/flink-savepoints-pvc created

    Optional for PVC: add a selector/label for nodes by running the following command:

    > kubectl label nodes NAME spec-node-ns-pod-disk=pvc-security

  10. Deploy RBAC Configuration for Flink ServiceAccount. It grants permissions for Flink to create and manage TaskManager pods.

    Open the flink-rbac.yaml from the installation package and specify a value for the following parameter:

    • Namespace: flink

    Run the following command:

    > kubectl apply -f flink-rbac.yaml -n flink

    Out:

    serviceaccount/flink created
    role.rbac.authorization.k8s.io/flink created
    rolebinding.rbac.authorization.k8s.io/flink-role-binding created
    role.rbac.authorization.k8s.io/event-reader created
    rolebinding.rbac.authorization.k8s.io/grant-event-reader created

Docker Images Pulling

Some of the images are stored at the Private SOC Prime Docker Image Registry Repo at Docker HUB.

SOC Prime will provide login credentials for Private Docker HUB.

  1. Set environments:

    export DOCKERHUB_USER="your_username"

    export DOCKERHUB_TOKEN="your_token"

  2. Check environment settings:

    echo $DOCKERHUB_USER

    echo $DOCKERHUB_TOKEN

  3. Set credentials for namespace flink

    kubectl create secret docker-registry dockerhub-creds \

    --docker-server=https://index.docker.io/v1/ \

    --docker-username="${DOCKERHUB_USER}" \

    --docker-password="${DOCKERHUB_TOKEN}" \

    -n flink \

    --dry-run=client -o yaml | kubectl apply -f -

DetectFlow Deployment

DetectFlow consists of two parts:

  • Admin Panel backend

  • Admin Panel UI

Admin Panel Backend Deployment

  1. Deploy ConfigMap with static configuration for Admin Panel Backend.

    Open admin-panel-be-configmap.yaml from the installation package and specify a value for the following parameter:

    • KAFKA_BOOTSTRAP_SERVERS:

      Set IP/Hostname and port of the Kafka Server

    Run the following command:

    > kubectl apply -f admin-panel-be-configmap.yaml -n flink

    Out: configmap/admin-panel-be-config created

  2. Deploy a Secret with sensitive data configuration for Admin Panel Backend.

    Open admin-panel-be-secret.yaml from the installation package and specify a value for the following parameter:

    # DATABASE_URL has the following format: postgresql+asyncpg://user:password@postgres_url:postgres_port/db_name

    It should be converted to Base64

    • DATABASE_URL:

    Run the following command:

    > kubectl apply -f admin-panel-be-secret.yaml -n flink

    Out: secret/admin-panel-be-secret created

  3. Deploy PersistentVolumeClaims for Admin Panel Backend using admin-panel-be-pvc.yaml from the installation package.

    Run the following command:

    > kubectl apply -f admin-panel-be-pvc.yaml -n flink

    Out: persistentvolumeclaim/admin-panel-metrics-pvc created

  4. Create a Deployment resource for Admin Panel Backend using admin-panel-be-deployment.yaml from the installation package.

    Run the following command:

    > kubectl apply -f admin-panel-be-deployment.yaml -n flink

    Out: deployment.apps/admin-panel-be created

  5. Deploy Network Service resource for Admin Panel Backend using admin-panel-be-service.yaml from the installation package.

    Run the following command:

    > kubectl apply -f admin-panel-be-service.yaml -n flink

    Out: service/admin-panel-be created

  6. Check that Admin-Panel-Backend-API is running and accessible by following the next link:

Admin Panel UI Deployment

  1. Deploy ConfigMap with a static configuration for Admin Panel UI.

    Open admin-panel-ui-configmap.yaml from the installation package and specify a value for the following parameter:

    Note: NAME.NAMESPACE.CLUSTER_DOMAIN_NAME (default: svc.cluster.local)

    • VITE_PREVIEW_ALLOWED_HOSTS:

    Set internal hostname of admin-panel-be, for example: admin-panel-be.flink.svc.cluster.local

    Run the following command:

    > kubectl apply -f admin-panel-ui-configmap.yaml -n flink

    Out: configmap/admin-panel-ui-config created

  2. Create a Deployment resource for Admin Panel UI using admin-panel-ui-deployment.yaml from the installation package.

    Run the following command:

    > kubectl apply -f admin-panel-ui-deployment.yaml -n flink

    Out: deployment.apps/admin-panel-ui created

  3. Deploy Network Service resource for Admin Panel UI using admin-panel-ui-service.yaml from the installation package.

    Run the following command:

    > kubectl apply -f admin-panel-ui-service.yaml -n flink

    Out: service/admin-panel-ui created

  4. Check that Admin-Panel-UI is running and accessible by following the next link:

Default:

Email: admin@soc.local

Password: admin

New Resources: DetectFlow Agent

v2.1.0/v2.0.0 introduces two new manifests for the LLM-based DetectFlow agent — a ConfigMap for runtime settings and a Secret for the API key.

detectflow-agent-config-configmap.yaml

ConfigMap detectflow-agent-config — non-sensitive runtime settings for the LLM agent.

apiVersion: v1
kind: ConfigMap
metadata:
 name: detectflow-agent-config
 namespace: flink
data:
 LLM_BASE_URL: http://localhost:8000/v1/   # LLM endpoint the agent calls
 LLM_CALLS_PER_MINUTE: "60"                 # rate limit — max requests/min
 LLM_MAX_RETRIES: "3"                       	# retry attempts on a failed call
 LLM_MAX_TOKENS: "2048"                     	# max tokens per completion
 LLM_MODEL: model                           	# model name/identifier to request
 LLM_PROMPT_CHAR_BUDGET: "24000"            	# max prompt size in characters
 MAX_CONCURRENT_LLM_REQUESTS: "1"           	# concurrency cap for in-flight calls

Note: LLM_BASE_URL points at localhost:8000 — the agent expects the LLM endpoint reachable as a sidecar/local proxy in its own pod, not the cluster-wide admin-panel-be service. Confirm this matches the actual agent deployment topology before rollout.

detectflow-agent-secret.yaml

Secret detectflow-agent-secrets — holds the LLM API key. Ships empty in the manifest — must be populated with a base64-encoded value before applying, otherwise the agent will fail to authenticate to the LLM endpoint.

apiVersion: v1
kind: Secret
metadata:
  name: detectflow-agent-secrets
  namespace: flink
data:
  LLM_API_KEY: ""   # base64-encoded API key — empty by default, fill before apply
type: Opaque

To populate and apply without hand-encoding base64:

kubectl -n flink create secret generic detectflow-agent-secrets \
  --from-literal=LLM_API_KEY='<your-api-key>' \
  --dry-run=client -o yaml | kubectl apply -f -

Upgrade Procedure

All five images live in the same Docker Hub repo (socprimecom/detectflow), tagged by component.

New Images in v2.1.0

Component

Image

Admin Panel Backend

socprimecom/detectflow:admin-panel-be.v2.1.0

Admin Panel UI

socprimecom/detectflow:admin-panel-ui.v2.1.0

Flink Sigma Detector

socprimecom/detectflow:flink-sigma-detector.v2.1.0

DF Agent job

socprimecom/detectflow:df-agent.v2.1.0

DF Correlation job

socprimecom/detectflow:df-correlation.v2.1.0

Upgrade Procedure (v2.0.0 → v2.1.0): Targeted Field Patch + Restart

  1. Image: admin-panel-be deployment

    kubectl -n flink set image deployment/admin-panel-be \
      admin-panel-be=socprimecom/detectflow:admin-panel-be.v2.1.0

    set image immediately triggers a rollout — no separate restart needed.

  2. Image: admin-panel-ui deployment

    kubectl -n flink set image deployment/admin-panel-ui \
      admin-panel-ui=socprimecom/detectflow:admin-panel-ui.v2.1.0

  3. Fields in ConfigMap admin-panel-be-config

    Three keys (FLINK_IMAGE, AGENT_JOB_IMAGE, CORRELATION_JOB_IMAGE) feed the Flink jobs launched by admin-panel-be, and are also consumed by the admin-panel-be deployment itself via envFrom: configMapRef (not injected per-key), so the change is only picked up by a new pod after restart.

    kubectl -n flink patch configmap admin-panel-be-config --type merge -p '
    data:
      FLINK_IMAGE: "socprimecom/detectflow:flink-sigma-detector.v2.1.0"
      AGENT_JOB_IMAGE: "socprimecom/detectflow:df-agent.v2.1.0"
      CORRELATION_JOB_IMAGE: "socprimecom/detectflow:df-correlation.v2.1.0"
    '

    ConfigMap changes do not auto-trigger a rollout — restart explicitly:

    kubectl -n flink rollout restart deployment/admin-panel-be

    Step 1 already restarts this same deployment. If applying both changes together, one rollout restart after the patch is enough; running it twice is harmless.

  4. Verification

    kubectl -n flink rollout status deployment/admin-panel-be

    kubectl -n flink rollout status deployment/admin-panel-ui

    kubectl -n flink get deployment admin-panel-be -o jsonpath='{.spec.template.spec.containers[0].image}'

    kubectl -n flink get deployment admin-panel-ui -o jsonpath='{.spec.template.spec.containers[0].image}'

  5. Rollback

    kubectl -n flink rollout undo deployment/admin-panel-be

    kubectl -n flink rollout undo deployment/admin-panel-ui

    ConfigMap rollback must be done manually (patch back to the .v2.0.0 values + rollout restart) — rollout undo does not version ConfigMaps.

Appendix A

In terms of calculating required resources, the following components of DetectFlow should be considered, with individual requirements for each component summed up to estimate the total amount:

  • Admin Panel is the control center to manage event matching processes. This component requires at least 2 CPUs and 4 GB RAM

  • Task Manager, the default component to control the operation of pipelines (based on match nodes, the Apache Flink technology). This component requires at least 1 CPU and 4 GB RAM

  • Match Node, a pipeline controlled via Admin Panel that matches detection rules to Apache Kafka topic events. The number of Match Nodes (pipelines) is custom, and you should estimate the required Kubernetes cluster resources based on the number of pipelines you're going to create, the data you're processing (EPS), and how many Sigma rules you'll apply. Reach out to SOC Prime for a detailed resource calculation.

Related Articles
Did this answer your question?