Teardown conflict with task scheduler

[vintagesucks]

Hi! I noticed that my "Sites" navigation on Forge contained some sites that should have been deleted already.

In fact, the site overview for that server no longer even shows the site, indicating that something has happened:

This happens on every teardown, but only when I include FORGE_JOB_SCHEDULER=true in my environment.

To narrow it down, I reduced the pipeline in TearDownCommand to this:

->through([
  FindServer::class,
  FindSiteOrFail::class,
  RemoveTaskScheduler::class,
  DestroySite::class,
])

Only when RemoveTaskScheduler is included here does the above bug occur. I think Forge is silently failing to actually remove the site if the scheduled job is still being removed. If I add something like sleep(10) before DestroySite, everything is deleted as expected:

->through([
  FindServer::class,
  FindSiteOrFail::class,
  RemoveTaskScheduler::class,
  function (ForgeService $service, Closure $next) {
    sleep(10);
    return $next($service);
  },
  DestroySite::class,
])

I am not sure if this is a bug in the Forge API/SDK or if Harbor is somehow supposed to wait for the job to actually be deleted, but this is currently resulting in many orphaned sites on my Forge server.

Complete environment from my local test:

SUBDOMAIN_NAME="test"
FORGE_DOMAIN="<removed>"
FORGE_GIT_BRANCH="main"
FORGE_GIT_REPOSITORY="<removed>"
FORGE_JOB_SCHEDULER=true
FORGE_PHP_VERSION="php82"
FORGE_SERVER="<removed>"
FORGE_TOKEN="<removed>"
FORGE_DEPLOY_SCRIPT="cd $FORGE_SITE_PATH"

Command I can use to reproduce the problem locally:

php ./harbor provision && php ./harbor teardown

This will return Environment teardown successful! All provisioned resources have been removed. and the site is partially removed from some parts of Forge, but remains in the site navigation and, more importantly, on the actual server.

[shawnhooper]

I've had this same problem. I haven't yet dug as deeply as @vintagesucks has into it. I have had to login to the server a few times and delete sites manually to clear up space. Sometimes in the file system directly, as they were already removed from the Sites dashboard.

Maybe related, I've also seen my GitHub account run of out deployment keys.

[Hannsre]

I've come across this as well. A new PR Site will be spawned without issues usually, but on teardown it won't be removed completely.
Haven't dug as deep yet as well, but I can't see any remnants of the preview Site in the forge Dashboard. But if I check via the Forge API, the removed site is still there in Status removing:

    "site": {
        "id": (removed),
        "server_id": (removed),
        "name": "pr-Id.example.com",
        "aliases": [],
        "directory": "/public",
        "wildcards": false,
        "status": "removing",

I can delete this site using the forge API manually then, but this shouldn't be necessary.

I can also confirm that setting FORGE_JOB_SCHEDULER to False will solve this issue for me.

Edit/Addition: I just noticed that for me this only happens in a PHP 8.3 environment. We also use Harbor for a different Project that's currently on PHP 8.2 which does not show the same issue. Here Teardown works as intended with FORGE_JOB_SCHEDULER set to true.

[samshupac]

I'm also experiencing this same issue.
This solution of setting FORGE_JOB_SCHEDULER: false isn't ideal as it means the temporary environments we create can't be used to test features fully.

FYI @mehrancodes

[RealMrHex]

Hey @vintagesucks, thanks for your report and explanations.
I'm working on this issue and ill update u all asap.

[samshupac]

Hey @RealMrHex / @mehrancodes - is there possibly an update on this? thanks