Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Scale-in ops of Pulsar cluster is always in Running status #8567

Open
tianyue86 opened this issue Dec 3, 2024 · 0 comments
Open

[BUG]Scale-in ops of Pulsar cluster is always in Running status #8567

tianyue86 opened this issue Dec 3, 2024 · 0 comments
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@tianyue86
Copy link

Describe the env

Kubernetes: v1.31.1-aliyun.1
KubeBlocks: 1.0.0-beta.10
kbcli: 1.0.0-beta.3/4

To Reproduce
Steps to reproduce the behavior:

  1. Create pulsar cluster by the following yaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: pulsar-baamxk
  namespace: default
  labels:
    app.kubernetes.io/instance: pulsar-baamxk
  annotations:
    resource.kubeblocks.io/ignore-constraint: "true"
    kubeblocks.io/enabled-pod-ordinal-svc: broker
    "kubeblocks.io/extra-env": '{"KB_PULSAR_BROKER_NODEPORT": "false"}'
spec:
  terminationPolicy: DoNotTerminate
  services:
    - name: broker-bootstrap
      serviceName: broker-bootstrap
      componentSelector: broker
      spec:
        type: ClusterIP
        ports:
          - name: pulsar
            port: 6650
            targetPort: 6650
          - name: http
            port: 80
            targetPort: 8080
          - name: kafka-client
            port: 9092
            targetPort: 9092
    - name: zookeeper
      serviceName: zookeeper
      componentSelector: zookeeper
      spec:
        type: ClusterIP
        ports:
          - name: client
            port: 2181
            targetPort: 2181
  componentSpecs:
    - name: proxy
      componentDef: pulsar-proxy
      replicas: 3
      resources:
        limits:
          cpu: 
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies-recovery
      componentDef: pulsar-bookies-recovery
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: broker
      componentDef: pulsar-broker
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies
      componentDef: pulsar-bookkeeper      
      replicas: 4
      resources:
        limits:
          cpu: 
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: ledgers
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: journal
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: zookeeper
      componentDef: pulsar-zookeeper
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. Scale-in it as below
kbcli cluster scale-in pulsar-xnhlfw --auto-approve --force=true --components bookies --replicas 3 --namespace default --namespace default
OpsRequest pulsar-xnhlfw-horizontalscaling-q7mhl created successfully, you can view the progress:
	kbcli cluster describe-ops pulsar-xnhlfw-horizontalscaling-q7mhl -n default
  1. Check ops status - always in Running status
kbcli cluster list-ops pulsar-xnhlfw --status all  --namespace default
NAME                                    NAMESPACE   TYPE                CLUSTER         COMPONENT   STATUS    PROGRESS   CREATED-TIME                 
pulsar-xnhlfw-horizontalscaling-q7mhl   default     HorizontalScaling   pulsar-xnhlfw   bookies     Running   0/3        Dec 03,2024 12:33 UTC+0800
  1. describe ops
tianyue@apeclouds-MacBook-Pro kbcli % k describe ops pulsar-xnhlfw-horizontalscaling-q7mhl
Name:         pulsar-xnhlfw-horizontalscaling-q7mhl
Namespace:    default
Labels:       app.kubernetes.io/instance=pulsar-xnhlfw
              app.kubernetes.io/managed-by=kubeblocks
              operations.kubeblocks.io/ops-type=HorizontalScaling
Annotations:  <none>
API Version:  operations.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2024-12-03T04:33:04Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  pulsar-xnhlfw-horizontalscaling-
  Generation:     2
  Owner References:
    API Version:     apps.kubeblocks.io/v1
    Kind:            Cluster
    Name:            pulsar-xnhlfw
    UID:             2ed0a167-7168-45a3-bda3-0f0f657ad463
  Resource Version:  5980100
  UID:               5fc98ea1-c0ba-4416-8a6a-3567626f8629
Spec:
  Cluster Name:      pulsar-xnhlfw
  Enqueue On Force:  false
  Force:             true
  Horizontal Scaling:
    Component Name:  bookies
    Scale In:
      Replica Changes:             3
  Pre Condition Deadline Seconds:  0
  Type:                            HorizontalScaling
Status:
  Cluster Generation:  9
  Components:
    Bookies:
      Phase:  Running
      Progress Details:
        Group:       bookies/Delete
        Message:     wait to delete "Pod/pulsar-xnhlfw-bookies-1" in Component: bookies
        Object Key:  Pod/pulsar-xnhlfw-bookies-1
        Status:      Pending
        Group:       bookies/Delete
        Message:     wait to delete "Pod/pulsar-xnhlfw-bookies-2" in Component: bookies
        Object Key:  Pod/pulsar-xnhlfw-bookies-2
        Status:      Pending
        Group:       bookies/Delete
        Message:     wait to delete "Pod/pulsar-xnhlfw-bookies-3" in Component: bookies
        Object Key:  Pod/pulsar-xnhlfw-bookies-3
        Status:      Pending
  Conditions:
    Last Transition Time:  2024-12-03T04:33:04Z
    Message:               wait for the controller to process the OpsRequest: pulsar-xnhlfw-horizontalscaling-q7mhl in Cluster: pulsar-xnhlfw
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2024-12-03T04:33:04Z
    Message:               OpsRequest: pulsar-xnhlfw-horizontalscaling-q7mhl is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2024-12-03T04:33:04Z
    Message:               Start to horizontal scale replicas in Cluster: pulsar-xnhlfw
    Reason:                HorizontalScalingStarted
    Status:                True
    Type:                  HorizontalScaling
  Last Configuration:
    Components:
      Bookies:
        Replicas:   4
  Phase:            Running
  Progress:         0/3
  Start Timestamp:  2024-12-03T04:33:04Z
Events:
  Type    Reason                    Age                From                    Message
  ----    ------                    ----               ----                    -------
  Normal  WaitForProgressing        11m                ops-request-controller  wait for the controller to process the OpsRequest: pulsar-xnhlfw-horizontalscaling-q7mhl in Cluster: pulsar-xnhlfw
  Normal  ValidateOpsRequestPassed  11m (x2 over 11m)  ops-request-controller  OpsRequest: pulsar-xnhlfw-horizontalscaling-q7mhl is validated
  Normal  HorizontalScalingStarted  11m (x2 over 11m)  ops-request-controller  Start to horizontal scale replicas in Cluster: pulsar-xnhlfw

Attach logs for investigation:

report-kubeblocks-2024-12-03-12-53-35.zip

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@tianyue86 tianyue86 added the kind/bug Something isn't working label Dec 3, 2024
@tianyue86 tianyue86 added this to the Release 1.0.0 milestone Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants