PVE API error HTTP 599 response - Reason: Too many redirections

[eminaktas]

Describe the bug

When vm creation fails - the error I got to complete: error retrieving task status: received an HTTP 599 response - Reason: Too many redirections, re-plan and re-apply fails to apply the replace in the sate for virtual_machine. VM is already created on first try but vm stuck at stopped status. However, the tf provider wasn't able to recover it from that state. I am already using GCS to store the state so that way I am generating the plan where I left off.

  # module.test[0].module.workers[1].proxmox_virtual_environment_vm.virtual_machine is tainted, so it must be replaced
+/- resource "proxmox_virtual_environment_vm" "virtual_machine" {
      ~ id                      = "2005" -> (known after apply)
      ~ ipv4_addresses          = [] -> (known after apply)
      ~ ipv6_addresses          = [] -> (known after apply)
      ~ mac_addresses           = [
          - "BC:24:11:FE:2C:1B",
        ] -> (known after apply)
        name                    = "k8s-worker-2-test"
      ~ network_interface_names = [] -> (known after apply)
      ~ started                 = false -> true
        tags                    = [
            "terraform",
            "worker",
        ]
        # (22 unchanged attributes hidden)

      ~ cpu {
          - flags        = [] -> null
            # (8 unchanged attributes hidden)
        }

      ~ disk {
          ~ path_in_datastore = "vm-2005-disk-5" -> (known after apply)
            # (12 unchanged attributes hidden)
        }
      ~ disk {
          ~ path_in_datastore = "vm-2005-disk-0" -> (known after apply)
            # (11 unchanged attributes hidden)
        }
      ~ disk {
          ~ path_in_datastore = "vm-2005-disk-1" -> (known after apply)
            # (11 unchanged attributes hidden)
        }
      ~ disk {
          ~ path_in_datastore = "vm-2005-disk-2" -> (known after apply)
            # (11 unchanged attributes hidden)
        }
      ~ disk {
          ~ path_in_datastore = "vm-2005-disk-3" -> (known after apply)
            # (11 unchanged attributes hidden)
        }
      ~ disk {
          ~ path_in_datastore = "vm-2005-disk-4" -> (known after apply)
            # (11 unchanged attributes hidden)
        }

      ~ initialization {
          - interface         = "ide2" -> null
          ~ upgrade           = false -> (known after apply)
            # (2 unchanged attributes hidden)

            # (2 unchanged blocks hidden)
        }

      ~ network_device {
          - disconnected = false -> null
          ~ mac_address  = "BC:24:11:FE:2C:1B" -> (known after apply)
            # (8 unchanged attributes hidden)
        }

        # (4 unchanged blocks hidden)
    }

Plan: 8 to add, 0 to change, 1 to destroy.

11512e5d15c9:/project/test# tofu apply tofu.tfplan 
Acquiring state lock. This may take a few moments...
module.test[0].module.workers[1].proxmox_virtual_environment_vm.virtual_machine: Creating...
╷
│ Error: error creating VM: received an HTTP 500 response - Reason: unable to create VM 2005 - VM 2005 already exists on node 'xxx'
│ 
│   with module.test[0].module.workers[1].proxmox_virtual_environment_vm.virtual_machine,
│   on .terraform/modules/test/modules/vm/main.tf line 1, in resource "proxmox_virtual_environment_vm" "virtual_machine":
│    1: resource "proxmox_virtual_environment_vm" "virtual_machine" {
│

To Reproduce
Steps to reproduce the behavior:

1. Create a VM with terraform/tofu
2. See failure creation the VM due to any reason during creating and let the tofu to store the state
3. Plan again
4. See the replace in the VM resource
5. Run Apply
6. See error

Expected behavior
Expected to see that it recovers the VM from its latest state.

• Single or clustered Proxmox: Single
• Proxmox version: 8.2.2
• Provider version (ideally it should be the latest version): v0.60.0
• Terraform/OpenTofu version: OpenTofu v1.7.2
• OS (where you run Terraform/OpenTofu from): Alpine Linux v3.18
• Debug logs (TF_LOG=DEBUG terraform apply): apply.log

[vanillaSprinkles]

it is marked as tainted; you can explicitly un-taint it (tofu untaint <tf-addr>); you can also run a state rm <tf-addr> (state remove) and import it. did you explicitly mark the item as tainted and are trying to have TF nuke the vm-id and re-create it?

[eminaktas]

it is marked as tainted; you can explicitly un-taint it (tofu untaint <tf-addr>); you can also run a state rm <tf-addr> (state remove) and import it. did you explicitly mark the item as tainted and are trying to have TF nuke the vm-id and re-create it?

I can get rid of the situation by removing the VM (if it is newly created). However, I don't think your suggestion would work since the VM is already created. Because on next try, it is trying to re-create the VM.

No, I don't mark the item as tainted but when it fails, flow taints the item.

[sn0ja]

I am able to reproduce this issue.

[bpg]

The PVE API error HTTP 599 response - Reason: Too many redirections is the root cause. It disrupts the provider's flow during VM creation (which is not an atomic operation), leaving resources in an inconsistent state.

While we can implement better error-handling logic—such as storing an incomplete VM resource in the state so that subsequent "apply" actions update it rather than creating a new one—this approach introduces additional challenges in handling those updates.

The root cause of the HTTP 599 error is not clear to me. I have encountered it several times in different cases, and it is always hard to pinpoint. Usually, a re-run passes successfully. I suspect it could be due to errors in the authentication flow (e.g., the provider trying to re-use an expired auth token) or some rate-limiting from the PVE API. This definitely needs deeper investigation.

[Blefish]

I've been able to get 599 when creating several virtual machines in one go. They are first cloned off of a template in one node and started on a different node.

Usually cloning goes through and VM seems to be made, but is never started due to previous error.

[bpg]

This seems to be an upstream issue: https://lore.proxmox.com/pve-devel/8392cb10-bec5-402d-bab1-6a1351049370@proxmox.com/t/#u