Environment Preparation
This chapter explains how to set up the ClickHouse or OpenSearch 3.7.0 cluster and the Kafka service that the logging components use: nodes and disks, the operators, the cluster, its accounts, and the Kafka password, user, ACLs, and topics.
Follow the steps in order and complete each verification.
Before You Start
Make sure you have:
- Administrator access to the cluster that runs the logging components.
- The nodes and disks planned in Step 1.
- The operator packages available in the platform marketplace:
clickhouse-operator, the Alauda Kafka operator, and opensearch-operator.
Use Log Component Capacity Planning with the tables below to choose the scale, and place the workloads on dedicated nodes as described in Planning Infra Nodes for Logging Storage.
Run every command in this chapter from a host that has kubectl access to the cluster. Each YAML block is a file you save and then apply; the text below each block gives the file name and the kubectl apply -f command.
Step 0: Choose the Target and the Scale
Choose the target (ClickHouse or OpenSearch) and the scale.
ClickHouse profiles
The CPU and memory values are the container limits for each ClickHouse pod.
Use the single-node profile for evaluation only. Start production at the three-node profile, and move to six or nine nodes when a single shard no longer fits.
Kafka
Set up three brokers with a 2C/4G limit each, plus the three controllers that the manifest in Step 4 runs at 1C/2G. Size the broker volumes by retention and throughput.
OpenSearch profiles
The CPU and memory values are per-node limits.
Do not size below the smallest profile, and use the large-scale profiles once a single node pool can no longer serve the data volume. If your measured storage is weaker than 6,000 IOPS and 250 MB/s read/write, size up.
Disk
Provide dedicated SSD storage with at least 6,000 IOPS and 250 MB/s read/write, and size the volumes for your retention: 7 days for most logs, 30 days for Kubernetes logs, 180 days for events and audits, and 540 days for metering. The examples below use 200 Gi per ClickHouse pod and per Kafka broker, 20 Gi per Kafka controller, and size the OpenSearch master and data pools separately.
Step 1: Nodes and Disks
-
Select the nodes that will run the storage cluster. Do not co-locate them with business workloads.
-
Label them as infra nodes and add the matching taint, following Planning Infra Nodes for Logging Storage. The manifests below select node-role.kubernetes.io/infra and tolerate that taint; if your cluster uses a different key, make the same change in every manifest.
-
Attach a dedicated SSD to every node as a persistent path:
- On the traditional operating system layout, use
/cpaas/data/....
- On Alauda OS nodes only
/var/cpaas is writable, so use /var/cpaas/data/....
-
Make sure the path survives node re-provisioning.
-
Create the directories the storage pods use and set their ownership. The examples use the traditional layout; on Alauda OS nodes replace /cpaas with /var/cpaas.
# ClickHouse runs as uid 101
sudo mkdir -p /cpaas/data/clickhouse
sudo chown -R 101:101 /cpaas/data/clickhouse
# Kafka runs as uid 1001
sudo mkdir -p /cpaas/data/kafka
sudo chown -R 1001:1001 /cpaas/data/kafka
# OpenSearch runs as uid 1000
sudo mkdir -p /cpaas/data/opensearch
sudo chown -R 1000:1000 /cpaas/data/opensearch
-
Decide how the volumes are provisioned:
For static local volumes, create one StorageClass and one PV per pod, then write the StorageClass name into <your-storage-class> in the manifests below. The number of PVs is the sum of the pods you plan: ClickHouse shardsCount × replicasCount, Kafka replicas + controller.replicas, and OpenSearch the sum of the node pool replicas.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cpaas-local
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: cpaas-clickhouse-0
spec:
capacity:
storage: 200Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: cpaas-local
local:
path: /cpaas/data/clickhouse
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values: ["<node-ip>"]
Save the YAML as local-storage.yaml and apply it:
kubectl apply -f local-storage.yaml
Repeat the PersistentVolume part for every pod, with a distinct metadata.name, a distinct local.path, and the IP of the node that pod runs on. When a node hosts more than one pod of the same component, give each pod its own directory, for example /cpaas/data/clickhouse-0 and /cpaas/data/clickhouse-1, created with the ownership from step 5.
Step 2: Install the Operators
Install the three operators from the platform marketplace. Every storage and messaging resource below is created in cpaas-system, so each operator must be able to reconcile resources in that namespace.
-
Install only the operators that are missing. If one is already installed on the cluster, for example by an earlier release, keep it and do not install a second copy: two copies of the same operator write to the same cpaas-system resources. Check its watch scope instead and widen it if needed.
-
Do not create an OperatorGroup in cpaas-system. The platform already owns one there, and a second OperatorGroup makes the platform reject every Subscription in that namespace, including its own.
-
For kafka-system and opensearch-operator, the OperatorGroup must have no spec.targetNamespaces. If an OperatorGroup scoped to its own namespace already exists, remove the field and wait for the operator pod to restart.
kubectl -n kafka-system patch operatorgroup kafka-system \
--type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]'
kubectl -n opensearch-operator patch operatorgroup opensearch-operator \
--type=json -p='[{"op":"remove","path":"/spec/targetNamespaces"}]'
Verify each operator before you continue:
# ClickHouse
kubectl get crd clickhouseinstallations.clickhouse.altinity.com
kubectl -n cpaas-system get deploy clickhouse-operator
# Kafka
kubectl get crd rdskafkas.middleware.alauda.io
kubectl -n kafka-system get deploy strimzi-cluster-operator
# OpenSearch (only when the target is OpenSearch)
kubectl get crd opensearchclusters.opensearch.opster.io
kubectl -n opensearch-operator get deploy opensearch-operator-controller-manager
WARNING
An operator that does not watch cpaas-system ignores the resources below silently: no status, no events, and no pods. Confirm the deployments are ready and that the Kafka and OpenSearch OperatorGroups reach all namespaces before you continue.
Step 3: Create the ClickHouse Cluster
Skip this step when the target is OpenSearch.
3.1 Create the admin password Secret
kubectl -n cpaas-system create secret generic clickhouse-basic-auth \
--from-literal=password="$(openssl rand -hex 16)"
3.2 Create the Keeper client Service (three nodes and above)
Every profile needs a Keeper, including the single-node profile, because the logging components create ReplicatedMergeTree tables. The single-node profile runs the Keeper inside its ClickHouse pod through the keeper_server/* settings in 3.3, so skip this step for it.
For three nodes and above, the ClickHouse pods themselves form the Keeper quorum: every ClickHouse pod is also a Keeper member. ClickHouse reaches that quorum through a headless Service that selects all ready pods of the installation.
Save the YAML as cpaas-clickhouse-keeper-service.yaml and apply it:
apiVersion: v1
kind: Service
metadata:
name: cpaas-clickhouse-keeper
namespace: cpaas-system
spec:
clusterIP: None
type: ClusterIP
ports:
- name: keeper
port: 9181
protocol: TCP
targetPort: 9181
selector:
clickhouse.altinity.com/chi: cpaas-clickhouse
clickhouse.altinity.com/namespace: cpaas-system
clickhouse.altinity.com/ready: "yes"
clickhouse.altinity.com/role: keeper
kubectl apply -f cpaas-clickhouse-keeper-service.yaml
The chi, namespace and ready labels are set by the operator. The role: keeper label comes from the pod template in 3.3.
3.3 Create the ClickHouseInstallation
Keep the cluster name replicated for the logging components, and set shardsCount and replicasCount from the Step 0 profile. Apply one of the two manifests below, depending on the profile you chose.
Replace <clickhouse-server-image> with the ClickHouse server image published with the platform middleware packages, for example registry.alauda.cn:60070/middleware/clickhouse-server:v25.8.16.34-61a7880e.
Single node. The Keeper runs inside the ClickHouse pod through the keeper_server/* settings.
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
name: cpaas-clickhouse
namespace: cpaas-system
spec:
configuration:
users:
# The admin password comes from the Secret created above.
admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password
admin/networks/ip:
- "0.0.0.0/0"
- "::/0"
admin/grants/query:
- GRANT ALL ON *.* WITH GRANT OPTION
profiles:
default/allow_nondeterministic_mutations: "1"
default/allow_unrestricted_reads_from_keeper: "1"
default/max_execution_time: 120
default/max_estimated_execution_time: 120
clusters:
- name: replicated # Reused in the connection Secret
templates:
podTemplate: pod-template
dataVolumeClaimTemplate: data-volumeclaim-template
layout:
shardsCount: 1 # From the profile: 1, 1, 2, or 3
replicasCount: 1 # The example runs one pod; use 3 for three nodes and above
settings:
default_database: observability # Reused in the connection Secret
merge_tree/materialize_ttl_recalculate_only: "1"
# Self-observability system tables grow without bound and eventually fill the volume.
asynchronous_metric_log/database: system
asynchronous_metric_log/table: asynchronous_metric_log
asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE"
metric_log/database: system
metric_log/table: metric_log
metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE"
trace_log/database: system
trace_log/table: trace_log
trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE"
# Co-located Keeper for the single-node profile. For three nodes and above,
# use the manifest below instead, which runs Keeper in every ClickHouse pod.
keeper_server/tcp_port: "9181"
keeper_server/server_id: "1"
keeper_server/log_storage_path: /var/lib/clickhouse/coordination/log
keeper_server/snapshot_storage_path: /var/lib/clickhouse/coordination/snapshots
keeper_server/coordination_settings/operation_timeout_ms: "10000"
keeper_server/coordination_settings/session_timeout_ms: "30000"
keeper_server/raft_configuration/server/id: "1"
keeper_server/raft_configuration/server/hostname: localhost
keeper_server/raft_configuration/server/port: "9234"
zookeeper:
nodes:
- host: localhost
port: 9181
defaults:
templates:
podTemplate: pod-template
dataVolumeClaimTemplate: data-volumeclaim-template
serviceTemplate: service-template
templates:
podTemplates:
- name: pod-template
podDistribution:
- scope: Shard
topologyKey: kubernetes.io/hostname
type: ShardAntiAffinity
spec:
nodeSelector:
node-role.kubernetes.io/infra: "" # The label you set in Step 1
tolerations:
- key: node-role.kubernetes.io/infra # The taint you set in Step 1
operator: Exists
effect: NoSchedule
containers:
- name: clickhouse
image: <clickhouse-server-image>
ports:
- name: http
containerPort: 8123
- name: client
containerPort: 9000
- name: interserver
containerPort: 9009
- name: keeper
containerPort: 9181
- name: raft
containerPort: 9234
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "2" # From the profile
memory: 4Gi # From the profile
volumeMounts:
- name: data-volumeclaim-template
mountPath: /var/lib/clickhouse
serviceTemplates:
- name: service-template
spec:
ports:
- name: http
port: 8123
- name: tcp
port: 9000
type: ClusterIP
volumeClaimTemplates:
- name: data-volumeclaim-template
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
storageClassName: <your-storage-class> # From Step 1
Save the YAML as cpaas-clickhouse.yaml and apply it:
kubectl apply -f cpaas-clickhouse.yaml
Three nodes and above. Use this manifest instead of the one above. The Keeper runs inside every ClickHouse pod, so the pods form the quorum among themselves and the installation stays a single ClickHouseInstallation.
The static Keeper configuration is injected through the cluster files and pulls in a generated file with include_from. The identity-dependent part — server_id and the member list — is generated per pod by an init container into an in-memory emptyDir. Keep SHARDS_COUNT and REPLICAS_COUNT in that init container equal to layout.shardsCount and layout.replicasCount, otherwise the member list is incomplete and the quorum never forms.
The readiness probe checks the Raft port. This is required: the default HTTP probe only succeeds after ClickHouse is serving, and ClickHouse does not finish starting until the Keeper quorum exists, so the operator would wait for the first replica forever and never create the remaining ones.
The Keeper path sits under /var/lib/clickhouse, which is the mounted data volume, so the Keeper log and snapshots live on the persistent volume together with the ClickHouse data. Do not move it outside that mount: Keeper state kept in the container filesystem is lost whenever the pod restarts.
The wait-for-self-dns init container waits until the pod resolves its own headless service name. Without it, a pod that starts before its DNS record is published initialises its distributed DDL worker against an unresolved hostname and then never retries: CREATE TABLE ... ON CLUSTER succeeds on the other replicas, and that replica silently misses the statement.
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
name: cpaas-clickhouse
namespace: cpaas-system
spec:
configuration:
users:
# The admin password comes from the Secret created above.
admin/k8s_secret_password: cpaas-system/clickhouse-basic-auth/password
admin/networks/ip:
- "0.0.0.0/0"
- "::/0"
admin/grants/query:
- GRANT ALL ON *.* WITH GRANT OPTION
profiles:
default/allow_nondeterministic_mutations: "1"
default/allow_unrestricted_reads_from_keeper: "1"
default/max_execution_time: 120
default/max_estimated_execution_time: 120
zookeeper:
nodes:
- host: cpaas-clickhouse-keeper # The Service created in 3.2
port: 9181
settings:
default_database: observability # Reused in the connection Secret
merge_tree/materialize_ttl_recalculate_only: "1"
# Self-observability system tables grow without bound and eventually fill the volume.
asynchronous_metric_log/database: system
asynchronous_metric_log/table: asynchronous_metric_log
asynchronous_metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE"
metric_log/database: system
metric_log/table: metric_log
metric_log/ttl: "event_date + INTERVAL 7 DAY DELETE"
trace_log/database: system
trace_log/table: trace_log
trace_log/ttl: "event_date + INTERVAL 7 DAY DELETE"
clusters:
- name: replicated # Reused in the connection Secret
templates:
podTemplate: pod-template
dataVolumeClaimTemplate: data-volumeclaim-template
layout:
shardsCount: 1 # From the profile: 1, 2, or 3
replicasCount: 3 # 3 for three nodes and above
shards:
- files:
keeper_config.xml: |
<clickhouse>
<include_from>/tmp/clickhouse/keeper_dynamic_configuration.xml</include_from>
<keeper_server incl="keeper_server">
<path>/var/lib/clickhouse/coordination</path>
<tcp_port>9181</tcp_port>
<four_letter_word_white_list>*</four_letter_word_white_list>
<coordination_settings>
<raft_logs_level>information</raft_logs_level>
</coordination_settings>
</keeper_server>
</clickhouse>
defaults:
templates:
podTemplate: pod-template
dataVolumeClaimTemplate: data-volumeclaim-template
serviceTemplate: service-template
templates:
podTemplates:
- name: pod-template
podDistribution:
- scope: Shard
topologyKey: kubernetes.io/hostname
type: ShardAntiAffinity
metadata:
labels:
clickhouse.altinity.com/role: keeper # Selected by the Service in 3.2
spec:
nodeSelector:
node-role.kubernetes.io/infra: "" # The label you set in Step 1
tolerations:
- key: node-role.kubernetes.io/infra # The taint you set in Step 1
operator: Exists
effect: NoSchedule
containers:
- name: clickhouse
image: <clickhouse-server-image>
env:
- name: RAFT_PORT
value: "9444"
ports:
- name: http
containerPort: 8123
- name: client
containerPort: 9000
- name: interserver
containerPort: 9009
- name: ch-keeper
containerPort: 9181
- name: raft
containerPort: 9444
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "2" # From the profile
memory: 4Gi # From the profile
volumeMounts:
- name: data-volumeclaim-template
mountPath: /var/lib/clickhouse
- name: keeper-dynamic-config
mountPath: /tmp/clickhouse
readinessProbe:
tcpSocket:
port: 9444
initialDelaySeconds: 10
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
initContainers:
- name: wait-for-self-dns
image: <clickhouse-server-image>
command:
- /bin/bash
- -c
- |
SELF="$(hostname -s).$(hostname -d)"
MY_IP="$(hostname -i)"
for i in $(seq 1 60); do
RESOLVED=$(getent hosts "$SELF" 2>/dev/null | awk '{print $1}' | head -1)
if [ "$RESOLVED" = "$MY_IP" ]; then
echo "resolved $SELF to $MY_IP after ${i}s"
exit 0
fi
sleep 2
done
echo "still unresolved after 120s, continuing"
exit 0
- name: keeper-config-initializer
image: <clickhouse-server-image>
env:
- name: RAFT_PORT
value: "9444"
- name: SHARDS_COUNT
value: "1" # Keep equal to layout.shardsCount
- name: REPLICAS_COUNT
value: "3" # Keep equal to layout.replicasCount
command:
- /bin/bash
- -c
- |
set -euo pipefail
OUT="/tmp/config/keeper_dynamic_configuration.xml"
HOST=$(hostname -s)
DOMAIN=$(hostname -d)
if [[ $HOST =~ (.*)-([0-9]+)-([0-9]+)-([0-9]+)$ ]]; then
SHARD=${BASH_REMATCH[2]}
REPLICA=${BASH_REMATCH[3]}
else
echo "Failed to parse shard/replica from hostname $HOST"; exit 1
fi
if [[ $DOMAIN =~ ^(.*)-([0-9]+)-([0-9]+)\.(.*)$ ]]; then
DOMAIN_NAME=${BASH_REMATCH[1]}
DOMAIN_SUFFIX=.${BASH_REMATCH[4]}
else
echo "Failed to parse domain $DOMAIN"; exit 1
fi
MY_ID=$((SHARD * REPLICAS_COUNT + REPLICA + 1))
KEEPER_ID=1
{
echo "<clickhouse>"
echo " <keeper_server>"
echo " <server_id>${MY_ID}</server_id>"
echo " <raft_configuration>"
for (( i=0; i<SHARDS_COUNT; i++ )); do
for (( j=0; j<REPLICAS_COUNT; j++ )); do
echo " <server>"
echo " <id>${KEEPER_ID}</id>"
echo " <hostname>${DOMAIN_NAME}-${i}-${j}${DOMAIN_SUFFIX}</hostname>"
echo " <port>${RAFT_PORT}</port>"
echo " </server>"
KEEPER_ID=$((KEEPER_ID + 1))
done
done
echo " </raft_configuration>"
echo " </keeper_server>"
echo "</clickhouse>"
} > "$OUT"
echo "Keeper dynamic configuration generated for server_id=${MY_ID}"
volumeMounts:
- name: keeper-dynamic-config
mountPath: /tmp/config
volumes:
- name: keeper-dynamic-config
emptyDir:
medium: Memory
serviceTemplates:
- name: service-template
spec:
ports:
- name: http
port: 8123
- name: tcp
port: 9000
type: ClusterIP
volumeClaimTemplates:
- name: data-volumeclaim-template
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
storageClassName: <your-storage-class> # From Step 1
Save the YAML as cpaas-clickhouse.yaml and apply it:
kubectl apply -f cpaas-clickhouse.yaml
default_database: observability makes ClickHouse create the observability database at startup, so there is no database to create here.
3.4 Wait for the cluster
kubectl -n cpaas-system get clickhouseinstallation cpaas-clickhouse \
-o jsonpath='{.status.status}{"\n"}' # re-run until it reports: Completed
kubectl -n cpaas-system get pod -l clickhouse.altinity.com/chi=cpaas-clickhouse
kubectl -n cpaas-system get pvc -l clickhouse.altinity.com/chi=cpaas-clickhouse
kubectl -n cpaas-system get svc -l clickhouse.altinity.com/chi=cpaas-clickhouse
All ClickHouse pods must be Running and ready, and every claim must be Bound. A StorageClass name that does not exist or cannot bind produces no pods and no error, so check the claims rather than the ClickHouseInstallation status alone.
For three nodes and above, confirm the Keeper quorum before you continue. The Keeper runs inside the ClickHouse pods, so check one pod per replica:
kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state
kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-1-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state
kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-2-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 9181 -q mntr | grep zk_server_state
The three lines must report one leader and two follower. Also confirm that ClickHouse reads the quorum through the Service from 3.2:
kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -- \
clickhouse-client -q "SELECT host, port FROM system.zookeeper_connection FORMAT TSV"
# expect: cpaas-clickhouse-keeper 9181
3.5 Create the account used by the logging components
The account must be able to create and alter tables in the target database. Scope it to that database.
Run these on the host. They generate the password, then run the SQL inside the ClickHouse pod. The account name needs backticks because it contains a hyphen, and SYSTEM DROP DNS CACHE is required by the retention cleaner. Record the printed password: the connection details in Step 6 need it, and 3.6 reuses the LOG_PASSWORD variable.
CH_POD="$(kubectl -n cpaas-system get pod \
-l clickhouse.altinity.com/chi=cpaas-clickhouse \
-o jsonpath='{.items[0].metadata.name}')"
ADMIN_PASSWORD="$(kubectl -n cpaas-system get secret clickhouse-basic-auth \
-o jsonpath='{.data.password}' | base64 -d)"
LOG_PASSWORD="$(openssl rand -hex 16)"
echo "platform-logging password: $LOG_PASSWORD"
kubectl -n cpaas-system exec -i "$CH_POD" -- clickhouse-client \
--user admin --password "$ADMIN_PASSWORD" --multiquery <<SQL
CREATE USER \`platform-logging\` ON CLUSTER 'replicated' IDENTIFIED BY '$LOG_PASSWORD';
GRANT ALL ON observability.* TO \`platform-logging\` ON CLUSTER 'replicated';
GRANT SYSTEM DROP DNS CACHE ON *.* TO \`platform-logging\`;
SQL
3.6 Verify
Run these in the same shell, so that $CH_POD and $LOG_PASSWORD from 3.5 are still set:
kubectl -n cpaas-system exec "$CH_POD" -- \
clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \
--query "CREATE TABLE observability.__perm_check (a UInt8) ENGINE = Memory"
kubectl -n cpaas-system exec "$CH_POD" -- \
clickhouse-client --user platform-logging --password "$LOG_PASSWORD" \
--query "DROP TABLE observability.__perm_check"
Both commands must succeed; a failed create means the account cannot manage the schema.
Record these values:
Step 4: Create the Kafka Service
4.1 Create the SASL password Secret
The password must be at least 32 characters on Alauda OS nodes or other FIPS-enabled hosts.
kubectl -n cpaas-system create secret generic platform-logging-password \
--from-literal=password="$(openssl rand -hex 16)"
4.2 Create the broker cluster
apiVersion: middleware.alauda.io/v1
kind: RdsKafka
metadata:
name: cpaas-kafka
namespace: cpaas-system
spec:
mode: KRaft
version: 4.2.0 # Minimum supported by the Alauda Kafka operator
replicas: 3
resources:
limits: { cpu: "2", memory: 4Gi } # From the profile
requests: { cpu: 500m, memory: 2Gi }
storage:
size: 200Gi
class: <your-storage-class>
deleteClaim: false
controller:
replicas: 3
roles: ["controller"] # Required: without it the node pool is rejected
template:
pod:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/infra # The label you set in Step 1
operator: Exists
tolerations:
- key: node-role.kubernetes.io/infra # The taint you set in Step 1
operator: Exists
effect: NoSchedule
resources:
limits: { cpu: "1", memory: 2Gi }
requests: { cpu: 100m, memory: 512Mi }
storage:
size: 20Gi
class: <your-storage-class>
deleteClaim: false
kafka:
template:
pod:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/infra # The label you set in Step 1
operator: Exists
tolerations:
- key: node-role.kubernetes.io/infra # The taint you set in Step 1
operator: Exists
effect: NoSchedule
listeners:
plain:
authentication:
type: scram-sha-512
tls:
authentication:
type: scram-sha-512
authorization:
type: simple
config:
auto.create.topics.enable: "false"
default.replication.factor: "3"
min.insync.replicas: "2"
offsets.topic.replication.factor: "3"
transaction.state.log.replication.factor: "3"
transaction.state.log.min.isr: "2"
log.retention.hours: "48"
unclean.leader.election.enable: "false"
message.max.bytes: "10485760"
replica.fetch.max.bytes: "10485760"
socket.request.max.bytes: "104857600"
entityOperator:
topicOperator: {} # Required: it creates the topics in Step 4.5
userOperator: {} # Required: it creates the SASL user in Step 4.4
Save the YAML as cpaas-kafka.yaml and apply it:
kubectl apply -f cpaas-kafka.yaml
Do not omit these settings:
4.3 Wait for the broker cluster
kubectl -n cpaas-system get rdsKafka cpaas-kafka \
-o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}'
kubectl -n cpaas-system get pod -l strimzi.io/cluster=cpaas-kafka
Wait until the Ready condition is True and all broker pods are Running.
Confirm the required settings reached the brokers:
KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \
-l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \
-o jsonpath='{.items[0].metadata.name}')"
kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- \
grep -E "^message\.max\.bytes|^replica\.fetch\.max\.bytes" /tmp/strimzi.properties
Both values must be 10485760, otherwise stop. The broker applies these settings as static configuration, so kafka-configs.sh --describe does not show them.
4.4 Create the SASL user and its ACLs
The logging components use a single account with access to the three topics, the consumer groups, and the broker metadata.
apiVersion: middleware.alauda.io/v1
kind: RdsKafkaUser
metadata:
name: platform-logging
namespace: cpaas-system
labels:
middleware.alauda.io/cluster: cpaas-kafka
spec:
authentication:
type: scram-sha-512
password:
valueFrom:
secretKeyRef:
name: platform-logging-password
key: password
authorization:
type: simple
acls:
# The three topics
- host: "*"
operation: All
resource: { type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal }
- host: "*"
operation: All
resource: { type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal }
- host: "*"
operation: All
resource: { type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal }
# The consumer groups used by the log pipeline
- host: "*"
operation: All
resource: { type: group, name: alauda_log, patternType: literal }
- host: "*"
operation: All
resource: { type: group, name: alauda_event, patternType: literal }
- host: "*"
operation: All
resource: { type: group, name: alauda_audit, patternType: literal }
# The consumer group prefix used by LogForward
- host: "*"
operation: All
resource: { type: group, name: "logforward-", patternType: prefix }
# The consumer group prefix used by the log query service
- host: "*"
operation: All
resource: { type: group, name: "razor-", patternType: prefix }
# Broker metadata
- host: "*"
operation: All
resource: { type: cluster, name: kafka-cluster, patternType: literal }
Save the YAML as platform-logging-user.yaml and apply it:
kubectl apply -f platform-logging-user.yaml
All nine entries are required. operation: All covers the read and describe permissions these entries need. Verify:
kubectl -n cpaas-system get rdskafkauser platform-logging \
-o jsonpath='{.status.phase}{"\n"}' # expect: Active
kubectl -n cpaas-system get secret platform-logging
The group names use underscores (alauda_log), while the topic names use uppercase letters and underscores (ALAUDA_LOG_TOPIC).
4.5 Create the three topics
apiVersion: middleware.alauda.io/v1
kind: RdsTopic
metadata:
name: alauda-log-topic
namespace: cpaas-system
labels:
middleware.alauda.io/cluster: cpaas-kafka
spec:
topicName: ALAUDA_LOG_TOPIC # Broker-side name; must match the ACL and the connection Secret
partitions: 30 # Upper bound for consumer parallelism
replicas: 3
config:
retention.ms: "172800000" # 48 hours
segment.bytes: "1073741824"
min.insync.replicas: "2"
compression.type: producer
max.message.bytes: "10485760"
---
apiVersion: middleware.alauda.io/v1
kind: RdsTopic
metadata:
name: alauda-event-topic
namespace: cpaas-system
labels:
middleware.alauda.io/cluster: cpaas-kafka
spec:
topicName: ALAUDA_EVENT_TOPIC
partitions: 30
replicas: 3
config:
retention.ms: "172800000"
segment.bytes: "1073741824"
min.insync.replicas: "2"
compression.type: producer
max.message.bytes: "10485760"
---
apiVersion: middleware.alauda.io/v1
kind: RdsTopic
metadata:
name: alauda-audit-topic
namespace: cpaas-system
labels:
middleware.alauda.io/cluster: cpaas-kafka
spec:
topicName: ALAUDA_AUDIT_TOPIC
partitions: 30
replicas: 3
config:
retention.ms: "172800000"
segment.bytes: "1073741824"
min.insync.replicas: "2"
compression.type: producer
max.message.bytes: "10485760"
Save the YAML as alauda-topics.yaml and apply it:
kubectl apply -f alauda-topics.yaml
The resource name must be a valid DNS name. spec.topicName is the broker-side name and must match the ACL entries above.
The example uses a 48-hour retention. Time-based retention alone does not cap disk usage: a burst of traffic can fill the broker volume before the window expires. Either size the broker volumes for the peak rate over the retention window, or add retention.bytes to each topic. retention.bytes applies per partition, so the broker volumes must hold partitions × retention.bytes.
4.6 Verify the Kafka service end to end
Build the client properties inside the broker pod from the user Secret. The broker listeners require SASL, so every command below needs them.
KAFKA_BROKER_POD="$(kubectl -n cpaas-system get pod \
-l strimzi.io/cluster=cpaas-kafka,strimzi.io/pool-name=broker \
-o jsonpath='{.items[0].metadata.name}')"
kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- \
sh -c 'cat > /tmp/logging-client.properties' <<EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=$(kubectl -n cpaas-system get secret platform-logging \
-o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d)
EOF
Confirm the topics exist on the brokers:
kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- bin/kafka-topics.sh \
--bootstrap-server localhost:9092 --command-config /tmp/logging-client.properties --list
Then produce and consume one record with the logging account. The consumer group must be one of the groups the ACLs allow; the console consumer otherwise creates a random group and the broker rejects it.
kubectl -n cpaas-system exec -i "$KAFKA_BROKER_POD" -c kafka -- bin/kafka-console-producer.sh \
--bootstrap-server localhost:9092 \
--command-config /tmp/logging-client.properties \
--topic ALAUDA_LOG_TOPIC <<< "kafka-check"
kubectl -n cpaas-system exec "$KAFKA_BROKER_POD" -c kafka -- bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--command-config /tmp/logging-client.properties \
--topic ALAUDA_LOG_TOPIC --group alauda_log \
--from-beginning --max-messages 1 --timeout-ms 10000
Record these values:
Step 5: Create the OpenSearch Cluster
Skip this step when the target is ClickHouse.
5.1 Create the cluster
Set the node pools from the Step 0 profile. The example is 3 + 5: three master nodes and five data nodes. For the small-scale profiles, use a single pool with roles: [cluster_manager, data] and replicas: 3 or 5.
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
name: cpaas-opensearch
namespace: cpaas-system
spec:
general:
serviceName: cpaas-opensearch
httpPort: 9200
version: 3.7.0
security:
tls:
http:
generate: true
transport:
generate: true
perNode: true
nodePools:
- component: masters
replicas: 3
diskSize: 100Gi
roles:
- cluster_manager
nodeSelector:
node-role.kubernetes.io/infra: "" # The label you set in Step 1
tolerations:
- key: node-role.kubernetes.io/infra # The taint you set in Step 1
operator: Exists
effect: NoSchedule
persistence:
pvc:
accessModes:
- ReadWriteOnce
storageClass: <your-storage-class> # From Step 1
resources:
limits:
cpu: "2" # From the profile
memory: 4Gi # From the profile
requests:
cpu: "1"
memory: 2Gi
- component: data
replicas: 5
diskSize: 800Gi
roles:
- data
- ingest
nodeSelector:
node-role.kubernetes.io/infra: "" # The label you set in Step 1
tolerations:
- key: node-role.kubernetes.io/infra # The taint you set in Step 1
operator: Exists
effect: NoSchedule
persistence:
pvc:
accessModes:
- ReadWriteOnce
storageClass: <your-storage-class> # From Step 1
resources:
limits:
cpu: "8" # From the profile
memory: 16Gi # From the profile
requests:
cpu: "2"
memory: 8Gi
dashboards:
replicas: 0
Save the YAML as cpaas-opensearch.yaml and apply it:
kubectl apply -f cpaas-opensearch.yaml
Wait for the cluster to be healthy:
kubectl -n cpaas-system get opensearchcluster cpaas-opensearch \
-o jsonpath='{.status.health}{"\n"}' # expect: green
The health reports unknown and then yellow while the nodes start and the shards initialize. Wait for green.
5.2 Optional: install the Chinese analyzer plugin
analysis-ik is optional. Without it the standard analyzer is used and Chinese text is not segmented; with it Chinese text is segmented.
Configure the plugin in the OpenSearchCluster spec. The operator passes every pluginsList entry to opensearch-plugin install each time a node starts, so the plugin survives pod restarts and node replacement. Installing it by hand inside a running container does not survive a restart, because the plugin is written outside the data volume.
To create the cluster with the plugin from the start, add both fields to cpaas-opensearch.yaml from 5.1 before you apply it:
spec:
general:
pluginsList:
- "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"
bootstrap:
pluginsList:
- "https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"
If you already created the cluster without the plugin, patch it instead. The operator rolls the nodes to install the plugin:
kubectl -n cpaas-system patch opensearchcluster cpaas-opensearch --type=merge -p '{"spec":{"general":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]},"bootstrap":{"pluginsList":["https://release.infinilabs.com/analysis-ik/stable/opensearch-analysis-ik-3.7.0.zip"]}}}'
The plugin version must match spec.general.version. A URL that returns 404 stops every node from starting. On a cluster without external network access, host the zip on an internal server and use that URL.
Verify that every node has the plugin:
for p in $(kubectl -n cpaas-system get pod -l opster.io/opensearch-cluster=cpaas-opensearch -o jsonpath='{.items[*].metadata.name}'); do echo -n "$p: "; kubectl -n cpaas-system exec $p -c opensearch -- bin/opensearch-plugin list | grep -c '^analysis-ik'; done
Each node must print 1. Then confirm the analyzer segments Chinese text. Replace <opensearch-endpoint> with the cluster service address, for example https://cpaas-opensearch.cpaas-system.svc:9200:
curl -sk -u "admin:<admin-password>" -X POST "https://<opensearch-endpoint>/_analyze" \
-H 'Content-Type: application/json' \
-d '{"analyzer":"ik_smart","text":"自然语言处理技术"}'
The response must contain several tokens, for example 自然语言, 处理, 技术.
5.3 Create the account used by the logging components
The account must read and write the log indices, install the index templates the logging components create, and manage their lifecycle policies. The security plugin is enabled, so create the role, the user, and the role mapping through its REST API. Replace <admin-password> with the password of the security plugin administrator and <password> with a value of at least 32 characters.
OS="https://<opensearch-endpoint>" # the cluster service, for example https://cpaas-opensearch.cpaas-system.svc:9200
curl -sk -u "admin:<admin-password>" -X PUT "$OS/_plugins/_security/api/roles/log_storage_writer" \
-H 'Content-Type: application/json' -d '{
"cluster_permissions": [
"cluster:monitor/*",
"cluster:admin/opendistro/ism/policy/*",
"indices:admin/index_template/put",
"indices:admin/index_template/get",
"indices:admin/template/put",
"indices:admin/template/get"
],
"index_permissions": [{
"index_patterns": ["log-*", "event-*", "audit-*", "meter-*"],
"allowed_actions": [
"indices:admin/create",
"indices:admin/mapping/put",
"indices:data/write/*",
"indices:data/read/*"
]
}]
}'
curl -sk -u "admin:<admin-password>" -X PUT "$OS/_plugins/_security/api/internalusers/platform-logging" \
-H 'Content-Type: application/json' \
-d '{"password":"<password>","backend_roles":[]}'
curl -sk -u "admin:<admin-password>" -X PUT "$OS/_plugins/_security/api/rolesmapping/log_storage_writer" \
-H 'Content-Type: application/json' -d '{"users":["platform-logging"]}'
The template and policy permissions are cluster-level. Declaring indices:admin/index_template/put and indices:admin/template/put under index_permissions does not grant them, and the logging components then fail to install their index templates. Without the role mapping the account is authenticated but has no permissions at all.
Verify as the new account:
curl -sk -u "platform-logging:<password>" -X PUT "$OS/_index_template/perm-check" \
-H 'Content-Type: application/json' \
-d '{"index_patterns":["log-perm-check-*"],"template":{"settings":{"number_of_shards":1}}}'
curl -sk -u "platform-logging:<password>" -X POST "$OS/log-perm-check/_doc" \
-H 'Content-Type: application/json' -d '{"check":1}'
curl -sk -u "platform-logging:<password>" "$OS/_index_template/perm-check"
All three must succeed. A security_exception with no permissions for [...] means the role or the role mapping is incomplete. Clean up with the administrator account:
curl -sk -u "admin:<admin-password>" -X DELETE "$OS/_index_template/perm-check"
curl -sk -u "admin:<admin-password>" -X DELETE "$OS/log-perm-check"
Record these values:
The platform applies index templates with one shard and one replica. If your HA policy needs different values, apply a composable template with a higher priority and confirm the result with GET /_index_template. Existing indices keep the settings they were created with.
Step 6: Record the Connection Details
Record every value below.
Environment Checklist
Fix any failed check before the logging components connect to this storage.
