Batch Support for Ubuntu 18.04 Ending

[alfpark]

Azure Batch will retire support for Canonical UbuntuServer 18.04-LTS to follow the publisher ending standard support in April 2023. If your workload utilizes Batch pools based on this OS, either via Marketplace images (directly from Canonical or derived from by other publishers) or your own custom images, then you will need to take action. Please migrate your workload to a publisher supported version of Canonical UbuntuServer such as 22.04-LTS. On 2023-05-30, Azure Batch will remove support for this compute node agent. Afterwards, create pool and resize up operations may fail. Customers who continue to use these pools past the indicated Batch support End of Life will be subject to potential security risks.

For more information, please see:

• Azure Batch VM Size and Image Guide
• Azure Batch Best Practices

[f-roche]

Hi @alfpark,

we were using this based image for a long time now so we need to migrate our pools to use the more recent 22.04-LTS version with Batch Shipyard but we are facing multiple issues:

1. replacing Ubuntu version in the pool.yaml is not working:
   
   

2. when looking at the available images using Azure Portal, it looks like the offer has been changed for this version:
   
   but when using 0001-com-ubuntu-server-jammy offer in the pool.yaml file it is still not working:
   

3. then I tried the latest ubuntu version from the microsoft-azure-batch publisher:
   
   and I've been able to create my pool with Shipyard... but since it is a native image our tasks cannot use the input_data feature anymore which is key in our pipeline (we are ingressing/egressing data from/to Azure blob containers and between Batch tasks)
   

So in regard to all these tests, what would you recommend to migrate our pools with the latest Ubuntu version AND non-native feature?

Thanks for your help.

PS: as a side question, will Microsoft develop and maintain a tool to replace Batch Shipyard? I recently read, with a lot of surprise, that this tool won't be maintained anymore, but I don't see any equivalent in Microsoft Azure Batch toolkits to deploy jobs with Docker support and data ingress/egress so easily. Does everyone have to create its own tool using only Batch SDKs?

[alfpark]

@f-roche, apologies for the disruption. Unfortunately, Batch Shipyard is no longer actively maintained. There is a possibility it can be revived later but given that there hasn't been active development on the project for many years the project archival reflects the proper state.

To potentially address your issue, please see: Azure/batch-shipyard#363

There was a commit that unblocked non-native 20.04 LTS, but this commit is currently only in develop. You can try taking the develop branch commits and seeing if that resolves your issue (at least for 20.04 LTS, there was no support tested for 22.04 LTS). Hopefully this provides you enough of a working bridge so that you can evaluate offboarding options from Batch Shipyard.

[f-roche]

Hi @alfpark, thanks for sharing this information, indeed the develop branch is compatible with Ubuntu 20.04-lts, so we are migrating to this image. But this would only be temporary if batch-shipyard has been retired I guess.

Last question, which tool and command line did you use to generate your release of batch-shipyard into a single executable file for Windows?
I tried with pyinstaller --onefile shipyard.py but I am still having a missing DLL dependency to python310.dll.

Best

[alfpark]

@f-roche, you can get some hints on how to build into a single executable by examining the Azure DevOps pipeline: https://github.com/Azure/batch-shipyard/blob/master/.vsts/pipeline.yml