diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..1e26c92 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,14 @@ +# Changelog + +All notable changes to this package will be documented in this file. + +The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) +and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). + +## [0.1.0] - 2024-09-04 + +Initial release + +Features: +- Source code of `bulk_upload_cli` +- Source code of `bulk_download_script` diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..ca665c4 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,5 @@ +Unity Cloud Python SDK copyright © 2023 Unity Technologies SF + +Licensed under the Unity Terms of Service ( see https://unity.com/legal/terms-of-service.). + +Unless expressly provided otherwise, the Software under this license is made available strictly on an "AS IS" BASIS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. Please review the license for details on these and other terms and conditions. \ No newline at end of file diff --git a/README.md b/README.md index 355d488..d72f46c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,20 @@ -# unity-cloud-python-sdk-samples -Public slack channel: [#uc-cs-am-python-sdk](https://unity.slack.com/messages/C04R01SGG68/)
-[View this project in Backstage](https://backstage.corp.unity3d.com/catalog/default/component/unity-cloud-python-sdk-samples)
-# Converting to public repository -Any and all Unity software of any description (including components) (1) whose source is to be made available other than under a Unity source code license or (2) in respect of which a public announcement is to be made concerning its inner workings, may be licensed and released only upon the prior approval of Legal. -The process for that is to access, complete, and submit this [FORM](https://airtable.com/appj757BYrNIMuTBI/shriEdWiQuxWmJOku). +# Python SDK samples + +This repository exposes a few samples that demonstrate the use of Python SDK in real life use-cases: + +- [Bulk Upload CLI](./bulk_upload_cli/README.md) +- [Bulk Download script](./bulk_download_script/README.md) + +> **Note**: This repository does not accept pull requests, review requests, or any other GitHub-hosted issue management requests. + +## Licenses + +The bulk Python SDK samples are made available under the [Unity ToS license](./LICENSE.md). + +## See also + +- [Unity Cloud Python SDK documentation](https://docs.unity.com/cloud/en-us/asset-manager/python-sdk) + +## Tell us what you think! + +Thank you for taking a look at the project! To help us improve and provide greater value, please consider providing feedback in our [Help & Support page](https://cloud.unity.com/home/dashboard-support). Thank you! \ No newline at end of file diff --git a/bulk_download_script/README.md b/bulk_download_script/README.md new file mode 100644 index 0000000..808d969 --- /dev/null +++ b/bulk_download_script/README.md @@ -0,0 +1,58 @@ +# Bulk download script + +The sample script demonstrates how to use Python SDK to download assets from Unity Cloud Asset Manager. + +To connect and find support, join the [Help & Support page](https://cloud.unity.com/home/dashboard-support)! + +## Table of contents +- [Bulk download script](#bulk-download-script) + - [Table of contents](#table-of-contents) + - [Prerequisites](#prerequisites) + - [System requirements](#system-requirements) + - [Licenses](#licenses) + - [How do I run the sample ?](#how-do-i-run-the-sample-) + - [1. Edit the `bulk_download.py` script with your requirements](#1-edit-the--bulk_downloadpy-script-with-your-requirements) + - [2. Run the script](#2-run-the-script) + - [See also](#see-also) + - [Tell us what you think!](#tell-us-what-you-think) + +## Prerequisites + +### System requirements + +To run the script, you need: +- Python installed on your machine +- An up-to-date Python SDK wheel installed ( > 0.5.0). +- The right permissions to use Asset Manager. See [Get Started with Asset Manager](https://docs.unity.com/cloud/en-us/asset-manager/get-started) for more details. +- A source project with assets manager enable and assets already uploaded in it. + +### Licenses + +The bulk download sample script is made available under the [Unity ToS license](../LICENSE.md). + +## How do I run the sample ? + +To run the sample, follow these steps: + +### 1. Edit the `bulk_download.py` script with your requirements + +In the `main` conditional section, you must edit some information to link the sample to your project. + +- org_id: Your organization id. +- project_id: Your project id. +- download_directory: must be edited with the path where the assets will be downloaded. +- overwrite: When set to `True`, the script will overwrite the files in the download directory if they already exist. Otherwise, it will skip the download. +- include_filter/exclude_filter/any_filter: This dictionary contains the search criteria to fetch the assets. Some example are written in comments, otherwise please refer to [the Python SDK documentation](https://docs.unity.com/cloud/en-us/asset-manager/python-sdk/manage-assets#create-filter-for-a-search-query) to learn how to use search criteria. +- collections: This list contains the collections to fetch the assets from. Leave it empty to search through all the assets in the project. + +### 2. Run the script + +With you favorite command line tool, run `python bulk_download.py` in this folder. + +## See also + +- [Unity Cloud Python SDK documentation](https://docs.unity.com/cloud/en-us/asset-manager/python-sdk) + +## Tell us what you think! + +Thank you for taking a look at the project! To help us improve and provide greater value, please consider providing feedback in our [Help & Support page](https://cloud.unity.com/home/dashboard-support). Thank you! \ No newline at end of file diff --git a/bulk_download_script/__init__.py b/bulk_download_script/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bulk_download_script/bulk_download.py b/bulk_download_script/bulk_download.py new file mode 100644 index 0000000..1b9c873 --- /dev/null +++ b/bulk_download_script/bulk_download.py @@ -0,0 +1,65 @@ +from concurrent.futures.thread import ThreadPoolExecutor + +import unity_cloud as uc +from pathlib import PurePath, Path +from unity_cloud.models import * + + +def login_with_user_account(): + uc.identity.user_login.use() + auth_state = uc.identity.user_login.get_authentication_state() + if auth_state != uc.identity.user_login.Authentication_State.LOGGED_IN: + uc.identity.user_login.login() + + +def download_asset(organization_id: str, project_id: str, asset: Asset, download_path: str, overwrite: bool = False): + dataset = uc.assets.get_dataset_list(organization_id, project_id, asset.id, asset.version)[0] + asset_files = uc.assets.get_file_list(organization_id, project_id, asset.id, asset.version, dataset.id) + + with ThreadPoolExecutor(max_workers=10) as executor: + for file in asset_files: + file_download_info = FileDownloadInformation(organization_id, project_id, asset.id, asset.version, + dataset.id, file.path, PurePath(download_path)) + + target_file = Path(download_path) / file.path + + if not overwrite and target_file.exists(): + print(f"Skipping download of {file.path} as it already exists", flush=True) + continue + + print(f"Downloading file: {file.path}", flush=True) + executor.submit(uc.assets.download_file, file_download_info) + + +def download_assets(assets: [Asset], org_id: str, project_id: str, download_path: str, overwrite: bool = False): + for asset in assets: + print(f"Downloading files for asset: {asset.name}", flush=True) + download_asset(org_id, project_id, asset, download_path, overwrite) + + +if __name__ == '__main__': + + uc.initialize() + login_with_user_account() + + org_id = '' + project_id = '' + download_directory = 'C:\\path\\to\\download\\directory\\' + overwrite = False + + include_filter = dict() + + #to search by status uncomment one of the following lines + #include_filter[SearchableProperties.STATUS] = "Published" + #include_filter[SearchableProperties.STATUS] = "Draft" + + #to search by tags uncomment one of the following lines and replace with the tag you want to search for + #include_filter[SearchableProperties.TAGS] = ["""] + #include_filter[SearchableProperties.FILES_TAGS] = ["""] + + collections = [] + # collections = [''] + + assets = uc.assets.search_assets_in_projects(org_id=org_id, project_ids=[project_id], include_filter=include_filter, + collections=collections) + download_assets(assets, org_id, project_id, download_directory, overwrite=overwrite) diff --git a/bulk_upload_cli/README.md b/bulk_upload_cli/README.md new file mode 100644 index 0000000..7d3e074 --- /dev/null +++ b/bulk_upload_cli/README.md @@ -0,0 +1,74 @@ +# Bulk upload CLI + +The Bulk upload Command-Line Interface (CLI) is a cross-platform command-line tool to connect to Asset Manager and execute administrative commands. It allows you to create configuration files that you can save and run from a terminal. Using CLI, you can create and update assets in bulk from your local disk to Asset Manager based on several inputs to match your folder structure. This tool offers an interactive mode where you are prompted to provide the necessary information to create and save configuration files for future asset updates. + +To connect and find support, join the [Help & Support page](https://cloud.unity.com/home/dashboard-support)! + +## Table of contents +- [Bulk upload CLI](#bulk-upload-cli) + - [Table of contents](#table-of-contents) + - [Prerequisites](#prerequisites) + - [System requirements](#system-requirements) + - [Licenses](#licenses) + - [How do I...?](#how-do-i) + - [Install the tool](#install-the-tool) + - [Run the tool in interactive mode](#run-the-tool-in-interactive-mode) + - [Select the input method](#select-the-input-method) + - [See also](#see-also) + - [Tell us what you think!](#tell-us-what-you-think) + +## Prerequisites + +### System requirements + +To run the script, you need: +- Python installed on your machine +- An Asset Manager Contributor role on the project level or an Organization Owner role. For more information about roles, see [Roles and permissions](https://docs.unity.com/cloud/en-us/asset-manager/org-project-roles). You can upload up to 10 GB on the free tier of Unity Cloud. +- A Unity Cloud project with asset manager service enabled to upload assets. For more information on how to create a new project on Unity Cloud, see [Create a new project](https://docs.unity.com/cloud/en-us/asset-manager/new-asset-manager-project). +- An assigned seat if you are part of an entitled organization, that is, an organization with a Pro or Enterprise license. For more information, see the [Important notes](https://docs.unity.com/cloud/en-us/asset-manager/org-project-roles#project-level-roles) section. + +### Licenses + +The bulk download sample script is made available under the [Unity ToS license](../LICENSE.md). + +## How do I...? + +### Install the tool + +1. Navigate to the current folder with your terminal. +2. Run the following help command to install the tool: +* On Mac: `python3 bulk_cli.py --install` +* On Windows: `python bulk_cli.py --install` + +### Run the tool in interactive mode + +1. Navigate to the current folder with your terminal. +2. Run the following command: +* On Mac: `python3 bulk_cli.py --create` +* On Windows: `python bulk_cli.py --create` + +### Select the input method + +Select one of the three strategies as the input method for bulk asset creation: + +1. Answer the following CLI prompt: `Are you uploading assets from a Unity project?` +* Enter Yes if you upload files that are either: + * In a Unity project and have .meta files from the editor +or: + * In a Unity package, like content from the asset store + +2. If you answered No to the prompt in step 1, select either of the following under the `Select a strategy` prompt: + +* `group files by name`: Select this option if your assets are following a naming convention, for example, blueasset.fbx, blueasset.png. +![Using the group by name convention](./documentation/group-by-name.png) + +* `group files by folder`: Select this option if your assets are organized by folder, that is, all relevant files are in distinct folders. +![Using the group by folder convention](./documentation/group-by-folder.png) + +## See also + +- [Unity Cloud Python SDK documentation](https://docs.unity.com/cloud/en-us/asset-manager/python-sdk) + +## Tell us what you think! + +Thank you for taking a look at the project! To help us improve and provide greater value, please consider providing feedback in our [Help & Support page](https://cloud.unity.com/home/dashboard-support). Thank you! \ No newline at end of file diff --git a/bulk_upload_cli/bulk_cli.py b/bulk_upload_cli/bulk_cli.py new file mode 100644 index 0000000..f1a4f3c --- /dev/null +++ b/bulk_upload_cli/bulk_cli.py @@ -0,0 +1,103 @@ +import argparse +import platform +import json +from shared.utils import OperationSystem, download_wheel, pip_install_wheel, pip_install_other_libraries, \ + check_install_requirements, check_python_version +import os + +source_folder = "../Source" +wheels_path = os.curdir + "/wheels" + + +def read_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument("--install", action="store_true", help="Install the requirements for the tool") + parser.add_argument("--create", action="store_true", help="Bulk create assets in the cloud") + parser.add_argument("--config-write", action="store_true", + help="Write the configuration file instead of running the action. Use with --create.", default=False) + parser.add_argument("--config-select", action="store_true",help="Select a configuration file to run. Use with --create.", default=False) + parser.add_argument("--config", type=str, help="Path to the configuration file. Use with --create.", default=None) + + args = parser.parse_args() + return args + + +def get_current_os(): + system = platform.system() + if system == "Windows": + return OperationSystem.windows + elif system == "Linux": + return "linux" + elif system == "Darwin": # macOS + return OperationSystem.macos + else: + raise Exception("Unsupported operating system:" + system) + + +def install_requirements(): + current_os = get_current_os() + download_wheel(wheels_path, current_os, False) + pip_install_wheel(wheels_path, current_os) + pip_install_other_libraries() + + +def run_bulk_assets_creation(interactive=False, config=None, write_config=False, config_select=False): + + if config_select: + from bulk_upload import interactive_runner + interactive_runner.run_with_config_select() + elif interactive or write_config: + from bulk_upload import interactive_runner + interactive_runner.run(write_config=write_config) + else: + if config is None: + raise Exception("Configuration file must be provided when running in non-interactive mode.") + from bulk_upload import models, assets_uploader + creation_config = models.ProjectUploaderConfig() + with open(config, "r") as f: + creation_config.load_from_json(json.load(f)) + uploader = assets_uploader.ProjectUploader() + uploader.run(creation_config) + + +if __name__ == "__main__": + arguments = read_arguments() + + config = arguments.config + write_config = arguments.config_write + config_select = arguments.config_select + interactive = False + + if not check_python_version(): + print("Python version is not supported. Please use Python 3.9 or higher.") + exit(1) + + if arguments.install: + install_requirements() + print("\n\n\n") + print("===============================================") + print("Requirements installed.") + exit(0) + + if not check_install_requirements(): + print("It seems that the requirements are not installed. Please run the script with --install first") + exit(1) + + if not arguments.create and not arguments.install: + print("No action specified. Please always use --create.") + exit(1) + + if config is None and not write_config and not config_select: + print("No config options provided. Interactive mode will be used.") + interactive = True + + if config is not None and write_config: + raise Exception("Both --config and --write-config cannot be used at the same time.") + + if config is not None and not os.path.exists(config): + raise Exception("Configuration file not found.") + + if arguments.create: + run_bulk_assets_creation(interactive, config, write_config, config_select) + else: + print("No action specified. Please always use --create.") \ No newline at end of file diff --git a/bulk_upload_cli/bulk_upload/__init__.py b/bulk_upload_cli/bulk_upload/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bulk_upload_cli/bulk_upload/asset_gathering_strategies.py b/bulk_upload_cli/bulk_upload/asset_gathering_strategies.py new file mode 100644 index 0000000..853b941 --- /dev/null +++ b/bulk_upload_cli/bulk_upload/asset_gathering_strategies.py @@ -0,0 +1,257 @@ +import os +import tarfile +import re +import shutil + +from abc import ABC, abstractmethod +from glob import glob +from pathlib import PurePath, PurePosixPath +from bulk_upload.models import AssetInfo, FileInfo, ProjectUploaderConfig + + +def get_file_info(file: PurePath, root_folder: str) -> FileInfo: + return FileInfo(file, PurePosixPath(file.relative_to(root_folder))) + + +def get_meta_file(file: PurePath, root_folder: str) -> FileInfo: + return get_file_info(PurePath(f"{file}.meta"), root_folder) + + +def meta_file_exists(file: PurePath) -> bool: + return os.path.exists(f"{file}.meta") + + +class AssetGatheringStrategy(ABC): + @abstractmethod + def get_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]: + pass + + @abstractmethod + def clean_up(self): + pass + + +class SingleFileUnityProjectStrategy(AssetGatheringStrategy): + + def get_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]: + files = [] + + if len(config.file_extensions) == 0: + # If no file extensions are provided, we will use all files in the assets folder + files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))] + else: + for extension in config.file_extensions: + files.extend([y for x in os.walk(config.assets_path) for y in + glob(os.path.join(x[0], f'*.{extension}'))]) + + assets = dict() + for f in files: + if os.path.isdir(f): + continue + if f.endswith(".meta"): # meta file should not be considered as an asset alone + continue + + file_name = os.path.basename(f) + assets[file_name] = AssetInfo(file_name) + + file = get_file_info(PurePath(f), config.assets_path) + assets[file_name].files.append(file) + + if meta_file_exists(file.path): + meta_file = get_meta_file(file.path, config.assets_path) + assets[file_name].files.append(meta_file) + dependencies = [] + with open(file.path, 'r') as file_readable: + dependencies = get_dependencies_from_file(file_readable) + + with open(meta_file.path, 'r') as meta_file_readable: + meta_file_content = meta_file_readable.read() + assets[file_name].unity_id = get_unity_id_from_meta_file(meta_file_content) + dependencies.extend(get_dependencies_from_string(meta_file_content)) + + assets[file_name].dependencies.extend(list(set(dependencies))) + + return list(assets.values()) + + def clean_up(self): + remove_empty_file() + + +class NameGroupingStrategy(AssetGatheringStrategy): + + def get_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]: + files = [] + + if len(config.file_extensions) == 0: + # If no file extensions are provided, we will use all files in the assets folder + files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))] + else: + for extension in config.file_extensions: + files.extend([y for x in os.walk(config.assets_path) for y in + glob(os.path.join(x[0], f'*.{extension}'))]) + + assets = dict() + for f in files: + if os.path.isdir(f): + continue + if f.endswith(".meta"): # meta file should not be considered as an asset alone + continue + + base_name = os.path.splitext(os.path.basename(f))[0] + if not config.case_sensitive: + base_name = base_name.lower() + + if base_name not in assets: + assets[base_name] = AssetInfo(base_name) + + file = get_file_info(PurePath(f), config.assets_path) + assets[base_name].files.append(file) + if meta_file_exists(file.path): + assets[base_name].files.append(get_meta_file(file.path, config.assets_path)) + + for common_file in config.files_common_to_every_assets: + common_file = PurePath(common_file) + for asset in assets.values(): + asset.files.append(get_file_info(common_file, config.assets_path)) + if meta_file_exists(common_file): + asset.files.append(get_meta_file(common_file, config.assets_path)) + + return list(assets.values()) + + def clean_up(self): + pass + + +class FolderGroupingStrategy(AssetGatheringStrategy): + + def get_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]: + folders = [x[0] for x in os.walk(config.assets_path) if x[0] != config.assets_path] + + if len(folders) == 0: + print(f"No folders found in the assets path. Only the root folder will be considered as an asset") + folders = [config.assets_path] + + assets = dict() + for folder in folders: + asset_name = PurePath(folder).name + assets[asset_name] = AssetInfo(asset_name) + + if len(config.file_extensions) == 0: + # If no file extensions are provided, we will use all files in the folder + assets[asset_name].files = [get_file_info(PurePath(f), config.assets_path) for x in os.walk(folder) for f in glob(os.path.join(x[0], '*'))] + else: + for extension in config.file_extensions: + assets[asset_name].files.extend( + [get_file_info(PurePath(f), config.assets_path) for f in + glob(os.path.join(folder, f'*.{extension}'))]) + + for asset in assets.values(): + asset_files = asset.files.copy() + for file in asset_files: + if meta_file_exists(file.path) and get_meta_file(file.path, config.assets_path) not in asset.files: + asset.files.append(get_meta_file(file.path, config.assets_path)) + + return list(assets.values()) + + def clean_up(self): + pass + + +class UnityPackageStrategy(AssetGatheringStrategy): + + def get_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]: + if config.assets_path == "": + print("No unity package path provided. Please provide a unityPackagePath in the config file. Exiting...") + return + + config.tags.append(PurePath(config.assets_path).name.replace(".unitypackage", "")) + + assets = [] + os.makedirs("tempo", exist_ok=True) + with tarfile.open(config.assets_path, 'r:gz') as tar: + tar_names = tar.getnames() + for name in tar_names: + member = tar.getmember(name) + if member.isdir(): + asset_file_name = name + "/asset" + meta_file_name = name + "/asset.meta" + path_file = name + "/pathname" + if asset_file_name in tar_names and meta_file_name in tar_names and path_file in tar_names: + tar.extract(asset_file_name, path="tempo") + tar.extract(meta_file_name, path="tempo") + dependencies = self.get_dependencies_from_tar_file(tar, asset_file_name) + dependencies.extend(self.get_dependencies_from_tar_file(tar, meta_file_name)) + + asset_path = self.get_path_from_pathname_file(tar, path_file) + asset = AssetInfo(asset_path.name) + asset.dependencies = list(set(dependencies)) + asset_file = PurePath("tempo").joinpath(asset_file_name) + asset.files.append(FileInfo(asset_file, PurePosixPath(asset_path.__str__()))) + meta_file = PurePath("tempo").joinpath(meta_file_name) + asset.files.append(FileInfo(meta_file, PurePosixPath(asset_path.__str__() + ".meta"))) + asset.unity_id = PurePath(name).as_posix() + + preview_file = name + "/preview.png" + if preview_file in tar_names: + tar.extract(preview_file, path="tempo") + asset.files.append(FileInfo(PurePosixPath(PurePath("tempo").joinpath(preview_file).__str__()), + PurePosixPath("preview.png"))) + asset.preview_file = PurePosixPath("preview.png") + + assets.append(asset) + + return assets + + def clean_up(self): + shutil.rmtree("tempo") + remove_empty_file() + + @staticmethod + def get_dependencies_from_tar_file(tar, file) -> list: + file = tar.extractfile(file) + return get_dependencies_from_file(file) + + @staticmethod + def get_path_from_pathname_file(tar, path_file): + file = tar.extractfile(path_file) + if file: + return PurePath(file.read().decode('utf-8')) + return None + + +def get_dependencies_from_file(file) -> list: + try: + file_content = file.read() + # If the file is a string, we can directly use it, otherwise we need to decode it + if isinstance(file_content, str): + return get_dependencies_from_string(file_content) + return get_dependencies_from_string(file_content.decode('utf-8')) + except UnicodeDecodeError: + pass # Ignore non-UTF-8 encoded files, they do not contain dependency information + return [] + + +def get_dependencies_from_string(file_content: str) -> list: + guid_regex = r"fileID:.*guid: ([a-f0-9]{32})" + pattern = re.compile(guid_regex) + return pattern.findall(file_content) + + +def get_unity_id_from_meta_file(meta_file_content) -> str: + guid_regex = r"\nguid: ([a-f0-9]{32})" + pattern = re.compile(guid_regex) + return pattern.findall(meta_file_content)[0] + + +def get_empty_file() -> PurePath: + if not os.path.exists("empty-file.template"): + with open("empty-file.template", "w") as file: + file.write("") + return PurePath("empty-file.template") + + +def remove_empty_file(): + try: + os.remove("empty-file.template") + except FileNotFoundError: + pass \ No newline at end of file diff --git a/bulk_upload_cli/bulk_upload/assets_uploader.py b/bulk_upload_cli/bulk_upload/assets_uploader.py new file mode 100644 index 0000000..922a7d3 --- /dev/null +++ b/bulk_upload_cli/bulk_upload/assets_uploader.py @@ -0,0 +1,263 @@ +import json +import os + +import unity_cloud as uc +from bulk_upload.models import ProjectUploaderConfig, Strategy, FileInfo, AssetInfo +from bulk_upload.asset_gathering_strategies import * +from concurrent.futures import ThreadPoolExecutor, wait +from unity_cloud.models import * +from pathlib import PurePath, PurePosixPath + + +def login_with_user_account(): + uc.identity.user_login.use() + auth_state = uc.identity.user_login.get_authentication_state() + if auth_state != uc.identity.user_login.Authentication_State.LOGGED_IN: + uc.identity.user_login.login() + + +class ProjectUploader: + + def __init__(self): + self.config = None + self.futures = list() + + @staticmethod + def login(key_id=None, key=None): + uc.initialize() + + if key is not None and key_id != "" and key_id is not None and key != "": + uc.identity.service_account.use(key_id, key) + else: + print("Logging in with user account in progress", flush=True) + login_with_user_account() + + @staticmethod + def get_collections(org_id: str, project_id: str): + collections = uc.assets.list_collections(org_id, project_id) + return [collection.name for collection in collections] + + def run(self, config: ProjectUploaderConfig, skip_login=False): + self.config = config + + if not skip_login: + self.login(config.key_id, config.key) + + self.validate_config() + + strategy = None + if self.config.strategy == Strategy.NAME_GROUPING: + strategy = NameGroupingStrategy() + elif self.config.strategy == Strategy.FOLDER_GROUPING: + strategy = FolderGroupingStrategy() + elif self.config.strategy == Strategy.UNITY_PACKAGE: + strategy = UnityPackageStrategy() + elif self.config.strategy == Strategy.SINGLE_FILE_ASSET: + strategy = SingleFileUnityProjectStrategy() + + print("Gathering assets", flush=True) + + project_assets = uc.assets.get_asset_list(self.config.org_id, self.config.project_id) + + assets = strategy.get_assets(self.config) + if assets is None: + return + + for asset in assets: + for project_asset in project_assets: + if asset.name == project_asset.name or (asset.name.lower() == project_asset.name.lower() and not self.config.case_sensitive): + asset.am_id = project_asset.id + asset.version = project_asset.version + asset.already_in_cloud = True + asset.is_frozen_in_cloud = project_asset.is_frozen + break + + with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor: + for asset in assets: + if asset.am_id is None: + self.futures.append(executor.submit(self.create_asset, asset)) + elif asset.is_frozen_in_cloud: + self.futures.append(executor.submit(self.create_new_version, asset)) + + wait(self.futures) + self.futures = list() + + print("Setting asset dependencies", flush=True) + with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor: + for asset in assets: + self.futures.append(executor.submit(self.set_asset_references, asset, assets)) + + wait(self.futures) + self.futures = list() + + with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor: + for asset in assets: + if not asset.already_in_cloud or self.config.update_files: + self.futures.append(executor.submit(self.upload_asset_files, asset)) + else: + print(f"Skipping file upload for asset: {asset.name} because updateFiles is set to False", flush=True) + + wait(self.futures) + self.futures = list() + + print("Setting tags and collections for assets", flush=True) + with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor: + for asset in assets: + self.futures.append(executor.submit(self.set_metadata_and_collection, asset)) + + wait(self.futures) + + strategy.clean_up() + print("Done uploading assets") + + def validate_config(self): + print("Validating configuration..", flush=True) + metadata_keys = uc.assets.list_field_definitions(self.config.org_id, self.config.project_id) + for key in self.config.metadata.keys(): + if key not in metadata_keys: + print("Key: " + key + " is not a valid metadata key. It will be ignored.") + self.config.metadata.pop(key) + + def create_asset(self, asset: AssetInfo): + try: + print(f"Creating asset: {asset.name} in cloud", flush=True) + asset_creation = AssetCreation(name=asset.name, type= AssetType.OTHER if len(asset.files) == 0 else self.get_asset_type(asset.files[0].cloud_path)) + created_asset = uc.assets.create_asset(asset_creation, self.config.org_id, self.config.project_id) + asset.am_id = created_asset.id + asset.version = created_asset.version + + except Exception as e: + print(f'Failed to create asset: {asset.name}', flush=True) + print(e, flush=True) + + def create_new_version(self, asset: AssetInfo): + try: + print(f"Creating new version for asset: {asset.name}", flush=True) + new_version = uc.assets.create_unfrozen_asset_version(self.config.org_id, self.config.project_id, asset.am_id, asset.version) + asset.version = new_version.version + except Exception as e: + print(f'Failed to create new version for asset: {asset.name}', flush=True) + print(e, flush=True) + + def upload_asset_files(self, asset: AssetInfo): + try: + + dataset_id = uc.assets.get_dataset_list(self.config.org_id, self.config.project_id, asset.am_id, asset.version)[0].id + + if self.config.update_files: + self.delete_existing_files(asset, dataset_id) + + print(f"Uploading files for asset: {asset.name}", flush=True) + files_upload_futures = [] + with ThreadPoolExecutor(max_workers=5) as executor: + for file in asset.files: + files_upload_futures.append(executor.submit(self.upload_file, asset, dataset_id, file)) + + wait(files_upload_futures) + + except Exception as e: + print(f'Failed to upload files for asset: {asset.name}', flush=True) + print(e, flush=True) + + def delete_existing_files(self, asset: AssetInfo, dataset_id: str): + file_list = uc.assets.get_file_list(self.config.org_id, self.config.project_id, asset.am_id, asset.version, dataset_id) + for file in file_list: + uc.assets.remove_file(self.config.org_id, self.config.project_id, asset.am_id, asset.version, dataset_id, file.path) + + def upload_file(self, asset: AssetInfo, dataset_id: str, file: FileInfo): + try: + if not self.config.update_files: + file_in_cloud = None + try: + file_in_cloud = uc.assets.get_file(self.config.org_id, self.config.project_id, asset.am_id, + asset.version, dataset_id, file.cloud_path) + except Exception: + # do nothing file was not found, this is expected + pass + + if file_in_cloud is not None: + print(f"File already in cloud: {file.cloud_path}", flush=True) + return + + file_upload = FileUploadInformation(organization_id=self.config.org_id, project_id=self.config.project_id, + asset_id=asset.am_id, asset_version=asset.version, dataset_id=dataset_id, + upload_file_path=file.path, cloud_file_path=file.cloud_path) + uc.assets.upload_file(file_upload, disable_automatic_transformations=True) + + except Exception as e: + print(f'Failed to upload file: {file.path}', flush=True) + print(e, flush=True) + + def set_asset_references(self, asset: AssetInfo, assets: [AssetInfo]): + try: + for dependency in asset.dependencies: + for a in assets: + if a.unity_id == dependency: + uc.assets.add_asset_reference(self.config.org_id, self.config.project_id, asset.am_id, asset.version, target_asset_id=a.am_id, target_asset_version=a.version) + asset.files.append( + FileInfo(get_empty_file(), PurePosixPath(f"{a.am_id}_{a.version}.am4u_dep"))) + break + + except Exception as e: + print(f'Failed to set references for asset: {asset.name}', flush=True) + print(e, flush=True) + + def set_metadata_and_collection(self, asset: AssetInfo): + + asset_update = AssetUpdate(name=asset.name) + + if len(self.config.tags) > 0: + asset_update.tags = self.config.tags + + if self.config.metadata is not None and len(self.config.metadata) > 0: + asset_update.metadata = self.config.metadata + + if self.config.description is not None and self.config.description != "": + asset_update.description = self.config.description + + try: + uc.assets.update_asset(asset_update, self.config.org_id, self.config.project_id, asset.am_id, asset.version) + except Exception as e: + print(f'Failed to update asset: {asset.name}', flush=True) + print(e, flush=True) + + if self.config.collection is not None and self.config.collection != "": + uc.assets.link_assets_to_collection(self.config.org_id, self.config.project_id, self.config.collection, + [asset.am_id]) + + uc.assets.freeze_asset_version(self.config.org_id, self.config.project_id, asset.am_id, asset.version, + "new version") + + def get_asset_type(self, cloud_path: PurePosixPath) -> AssetType: + suffix = cloud_path.suffix.lower() + if suffix in [".fbx", ".obj", ".prefab"]: + return AssetType.MODEL_3D + elif suffix in [".png", ".apng", ".jpg", ".jpeg", ".bmp", ".tiff", ".gif", ".psd", ".tga", ".tif", + ".exr", ".webp", ".svg", "pjpeg", ".pjp", ".jfif", ".avif", ".ico", ".cur", ".ani"]: + return AssetType.ASSET_2D + elif suffix in [".mp4", ".webm", ".ogg", ".ogv", ".avi", ".mov", ".flv", ".mkv", ".m4v", ".3gp", + ".h264", ".h265", "wmv"]: + return AssetType.VIDEO + elif suffix in [".mp3", ".wav", ".ogg", ".aac"]: + return AssetType.AUDIO + elif suffix == ".cs": + return AssetType.SCRIPT + elif suffix in [".mat", ".shader"]: + return AssetType.MATERIAL + else: + return AssetType.OTHER + + def get_file_info(self, file: PurePath) -> FileInfo: + return FileInfo(file, PurePosixPath(file.relative_to(self.config.assets_root_folder))) + + def get_meta_file(self, file: PurePath) -> FileInfo: + return self.get_file_info(PurePath(f"{file}.meta")) + + +if __name__ == '__main__': + config = ProjectUploaderConfig() + with open("config.json") as f: + config.load_from_json(json.load(f)) + + uploader = ProjectUploader() + uploader.run(config) \ No newline at end of file diff --git a/bulk_upload_cli/bulk_upload/config.json b/bulk_upload_cli/bulk_upload/config.json new file mode 100644 index 0000000..636e75e --- /dev/null +++ b/bulk_upload_cli/bulk_upload/config.json @@ -0,0 +1,19 @@ +{ + "strategy": "", + "filesCommonToEveryAssets": [], + "assetsPath": "", + "assetFileExtensions": [], + "organizationId": "", + "projectId": "", + "serviceAccount": { + "keyId": "", + "key": "" + }, + "amountOfParallelUploads": 15, + "collectionToLinkAssetTo": "", + "tagsToApplyToAssets": [], + "assetNameCaseSensitive": false, + "metadataToApply": {}, + "updateFiles": true, + "description": "" +} \ No newline at end of file diff --git a/bulk_upload_cli/bulk_upload/interactive_runner.py b/bulk_upload_cli/bulk_upload/interactive_runner.py new file mode 100644 index 0000000..758d8cb --- /dev/null +++ b/bulk_upload_cli/bulk_upload/interactive_runner.py @@ -0,0 +1,331 @@ +import json +import os + +from InquirerPy import prompt +from InquirerPy.base.control import Choice + +from bulk_upload.models import ProjectUploaderConfig, Strategy +from bulk_upload.assets_uploader import ProjectUploader + +project_uploader = ProjectUploader() + + +def run(write_config: bool = False): + project_uploader = ProjectUploader() + key_id, key = ask_for_login() + project_uploader.login(key_id, key) + + questions = [ + { + "type": "confirm", + "message": "Are you uploading assets from a Unity project?", + "default": False + } + ] + + result = prompt(questions=questions) + if result[0]: + ask_unity_project_questions(write_config=write_config) + else: + ask_non_unity_project_questions(write_config=write_config) + + +def ask_unity_project_questions(write_config: bool = False): + questions = [ + { + "type": "list", + "message": "Where are the assets located?", + "choices": ["in a .unitypackage file", "in a folder", Choice(value=None, name="Exit")], + "default": 0, + } + ] + result = prompt(questions=questions) + + if result[0] == "in a .unitypackage file": + run_unity_package_strategy(write_config) + elif result[0] == "in a folder": + run_non_packaged_strategy(Strategy.SINGLE_FILE_ASSET, write_config) + + +def ask_non_unity_project_questions(write_config: bool = False): + questions = [ + { + "type": "list", + "message": "Select a strategy:", + "choices": ["group files by name", "group files by folder", Choice(value=None, name="Exit")], + "default": 0, + } + ] + result = prompt(questions=questions) + + if result[0] == "group files by name": + run_non_packaged_strategy(Strategy.NAME_GROUPING, write_config) + elif result[0] == "group files by folder": + run_non_packaged_strategy(Strategy.FOLDER_GROUPING, write_config) + + +def run_non_packaged_strategy(strategy: Strategy, write_config: bool = False): + config = ProjectUploaderConfig() + config.strategy = strategy + + questions = [ + { + "type": "input", + "message": "Enter the path to the root folder of the assets:", + "default": "" + }, + { + "type": "input", + "message": "Enter your organization ID:", + "default": "" + }, + { + "type": "input", + "message": "Enter your project ID:", + "default": "" + }, + { + "type": "confirm", + "message": "Would you like to update existing assets?", + "default": True + }, + { + "type": "input", + "message": "Enter the tags to apply to the assets (comma separated; leave empty to assign no tag):", + } + ] + + if strategy == Strategy.NAME_GROUPING or strategy == Strategy.SINGLE_FILE_ASSET: + questions.append({ + "type": "input", + "message": "Enter the file extensions to include (comma separated; leave empty to include everything in the search):", + "default": "" + }) + + if strategy == Strategy.NAME_GROUPING: + questions.append({ + "type": "confirm", + "message": "Is the asset name case sensitive?", + "default": False + }) + questions.append({ + "type": "input", + "message": "Enter the files that are common to every asset (comma separated; leave empty if there are none):", + "default": "" + }) + + result = prompt(questions=questions) + + config.assets_path = sanitize_string(result[0]) + config.org_id = result[1] + config.project_id = result[2] + config.update_files = result[3] + + config.tags = sanitize_tags(result[4]) + + if strategy == Strategy.NAME_GROUPING or strategy == Strategy.SINGLE_FILE_ASSET: + config.file_extensions = result[5].split(",") if result[5] != "" else [] + + if strategy == Strategy.NAME_GROUPING: + config.case_sensitive = result[6] + config.files_common_to_every_assets = result[7].split(",") if result[7] != "" else [] + + try: + collections = project_uploader.get_collections(config.org_id, config.project_id) + except Exception as e: + collections = [] + + collections.append(Choice(value=None, name="No collection")) + + questions = [ + { + "type": "list", + "message": "Select the collection you want to link the assets to.", + "choices": collections, + "default": 0, + }] + + result = prompt(questions=questions) + config.collection = result[0] if result[0] != "No collection" else "" + + if write_config: + write_config_file(config) + else: + questions = [{ + "type": "confirm", + "message": "Would you like to save the config of this run?", + "default": False + }] + + result = prompt(questions=questions) + + if result[0]: + write_config_file(config) + + project_uploader.run(config, True) + + +def run_unity_package_strategy(write_config: bool = False): + questions = [ + { + "type": "input", + "message": "Enter the path to the Unity package:", + "default": "" + }, + { + "type": "input", + "message": "Enter your organization ID:", + "default": "" + }, + { + "type": "input", + "message": "Enter your project ID:", + "default": "" + }, + { + "type": "confirm", + "message": "Would you like to update existing assets?", + "default": True + }, + { + "type": "input", + "message": "Enter the tags to apply to the assets (comma separated; leave empty to assign no tag):", + } + ] + + result = prompt(questions=questions) + + config = ProjectUploaderConfig() + config.strategy = Strategy.UNITY_PACKAGE + config.assets_path = sanitize_string(result[0]) + config.org_id = result[1] + config.project_id = result[2] + config.update_files = result[3] + config.tags = sanitize_tags(result[4]) + + try: + collections = project_uploader.get_collections(config.org_id, config.project_id) + except Exception as e: + collections = [] + + collections.append(Choice(value=None, name="No collection")) + + questions = [ + { + "type": "list", + "message": "Select the collection you want to link the assets to.", + "choices": collections, + "default": 0, + }] + + result = prompt(questions=questions) + config.collection = result[0] if result[0] != "No collection" else "" + + + if write_config: + write_config_file(config) + else: + questions = [{ + "type": "confirm", + "message": "Would you like to save the config of this run?", + "default": False + }] + + result = prompt(questions=questions) + + if result[0]: + write_config_file(config) + + project_uploader.run(config, True) + + +def ask_for_login(): + questions = [{ + "type": "list", + "message": "Choose authentication method?", + "choices": ["User login", "Service account"], + "default": 0, + }] + + result = prompt(questions=questions) + + if result[0] == "Service account": + questions = [ + { + "type": "input", + "message": "Enter your key ID:", + "default": "" + }, + { + "type": "password", + "message": "Enter your key:", + "default": "" + } + ] + + result = prompt(questions=questions) + + return result[0], result[1] + + return "", "" + + +def write_config_file(config: ProjectUploaderConfig): + questions = [ + { + "type": "input", + "message": "Enter the name to save the configuration file:", + "default": "" + } + ] + result = prompt(questions=questions) + file_name = result[0] if result[0].endswith(".json") else result[0] + ".json" + with open(file_name, "w") as f: + f.write(config.to_json()) + print("Configuration saved to", file_name) + + +def run_with_config_select(): + config_files = [f for f in os.listdir() if f.endswith(".json")] + if len(config_files) == 0: + print("No configuration files found in the current directory. Please create a configuration file first.") + return + + questions = [ + { + "type": "list", + "message": "Select a configuration file:", + "choices": config_files, + "default": 0 + } + ] + + result = prompt(questions=questions) + with open(result[0], "r") as f: + config = ProjectUploaderConfig() + config.load_from_json(json.load(f)) + project_uploader.run(config, False) + + +def sanitize_tags(tags: str) -> list[str]: + + tags = [tag for tag in tags.split(",") if tag != ""] + return_tags = [] + for tag in tags: + if tag == "": + continue + + tag = sanitize_string(tag) + return_tags.append(tag) + + return return_tags + + +def sanitize_string(value: str) -> str: + while value.startswith(" ") or value.endswith(" "): + if value.startswith(" "): + value = value[1:] + if value.endswith(" "): + value = value[:-1] + + return value \ No newline at end of file diff --git a/bulk_upload_cli/bulk_upload/models.py b/bulk_upload_cli/bulk_upload/models.py new file mode 100644 index 0000000..7d34dd3 --- /dev/null +++ b/bulk_upload_cli/bulk_upload/models.py @@ -0,0 +1,100 @@ +import json +from enum import Enum +from pathlib import PurePath, PurePosixPath + + +class Strategy(str, Enum): + SINGLE_FILE_ASSET = "singleFileAsset" + NAME_GROUPING = "nameGrouping" + FOLDER_GROUPING = "folderGrouping" + UNITY_PACKAGE = "unityPackage" + + +class ProjectUploaderConfig(object): + + def __init__(self): + self.strategy = Strategy.SINGLE_FILE_ASSET + self.files_common_to_every_assets = [] + self.assets_path = "" + self.file_extensions = [] + self.org_id = "" + self.project_id = "" + self.key_id = "" + self.key = "" + self.amount_of_parallel_uploads = 15 + self.collection = "" + self.tags = [] + self.case_sensitive = False + self.metadata = {} + self.update_files = False + self.description = "" + + def load_from_json(self, config_json: dict): + self.strategy = Strategy(config_json.get("strategy", "resource_type")) + self.files_common_to_every_assets = config_json.get("filesCommonToEveryAssets", []) + self.assets_path = config_json.get("assetsPath", "") + self.file_extensions = config_json.get("assetFileExtensions", []) + self.org_id = config_json.get("organizationId", "") + self.project_id = config_json.get("projectId", "") + self.key_id = config_json.get("serviceAccount", dict()).get("keyId", None) + self.key = config_json.get("serviceAccount", dict()).get("key", None) + self.amount_of_parallel_uploads = config_json.get("amountOfParallelUploads", 15) + self.collection = config_json.get("collectionToLinkAssetTo", None) + self.tags = config_json.get("tagsToApplyToAssets", []) + self.case_sensitive = config_json.get("assetNameCaseSensitive", False) + self.metadata = config_json.get("metadataToApply", {}) + self.update_files = config_json.get("updateFiles", False) + self.description = config_json.get("description", "") + + # remove the "." from the file extensions + self.file_extensions = [x[1:] if x.startswith(".") else x for x in self.file_extensions] + + # we remove meta file extension since they are included by default + if "meta" in self.file_extensions: + self.file_extensions.remove("meta") + if ".meta" in self.file_extensions: + self.file_extensions.remove(".meta") + + def to_json(self): + + asset_path = json.dumps(self.assets_path) + files_common_to_every_assets = [json.dumps(x) for x in self.files_common_to_every_assets] + return rf"""{{ + "strategy": "{self.strategy.value}", + "filesCommonToEveryAssets": {json.dumps(files_common_to_every_assets)}, + "assetsPath": {json.dumps(self.assets_path)}, + "assetFileExtensions": {json.dumps(self.file_extensions)}, + "organizationId": "{self.org_id}", + "projectId": "{self.project_id}", + "serviceAccount": {{ + "keyId": "{self.key_id}", + "key": "{self.key}" + }}, + "amountOfParallelUploads": {self.amount_of_parallel_uploads}, + "collectionToLinkAssetTo": "{self.collection}", + "tagsToApplyToAssets": {json.dumps(self.tags)}, + "assetNameCaseSensitive": {self.case_sensitive.__str__().lower()}, + "metadataToApply": {self.metadata}, + "updateFiles": {self.update_files.__str__().lower()}, + "description": {json.dumps(self.description)} +}} +""" + + +class FileInfo(object): + def __init__(self, path: PurePath, cloud_path: PurePosixPath): + self.path = path + self.cloud_path = cloud_path + + +class AssetInfo(object): + def __init__(self, name): + self.name = name + self.files = [] + self.am_id = None + self.unity_id = None + self.dependencies = [] + self.already_in_cloud = False + self.version = "" + self.preview_file = None + self.is_frozen_in_cloud = False \ No newline at end of file diff --git a/bulk_upload_cli/documentation/group-by-folder.png b/bulk_upload_cli/documentation/group-by-folder.png new file mode 100644 index 0000000..5028309 Binary files /dev/null and b/bulk_upload_cli/documentation/group-by-folder.png differ diff --git a/bulk_upload_cli/documentation/group-by-name.png b/bulk_upload_cli/documentation/group-by-name.png new file mode 100644 index 0000000..e313c62 Binary files /dev/null and b/bulk_upload_cli/documentation/group-by-name.png differ diff --git a/bulk_upload_cli/shared/__init__.py b/bulk_upload_cli/shared/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bulk_upload_cli/shared/utils.py b/bulk_upload_cli/shared/utils.py new file mode 100644 index 0000000..5227af6 --- /dev/null +++ b/bulk_upload_cli/shared/utils.py @@ -0,0 +1,205 @@ +import os.path +import shutil +import sys +import urllib.request +import re +import glob +import subprocess +from enum import Enum + +sdk_version = "0.10.0" + +class OperationSystem(Enum): + windows = 'windows' + macos = 'macos' + + +def get_platform_name(system: str, machine: str) -> str: + name: str + if system == "windows": + if machine == "amd64" or machine == "x86_64": + name = "win_amd64" + elif machine == "arm64": + name = "win_arm64" + elif system == "darwin": + name = "macosx_13_0_universal2" + else: + raise Exception(f"Unsupported configuration: {system}-{machine}") + return name + +sdk_version = "0.10.0" +protocol = "https://" +domain = "transformation.unity.com" +url_format = f"{protocol}{domain}/downloads/pythonsdks/release/{sdk_version}/unity_cloud-{sdk_version}-py3-none-{{0}}.whl" + +wheel_names = { + OperationSystem.macos: f"unity_cloud-{sdk_version}-py3-none-macosx_13_0_universal2.whl", + OperationSystem.windows: f"unity_cloud-{sdk_version}-py3-none-win_amd64.whl", +} + +operation_systems = { + OperationSystem.macos: url_format.format("macosx_13_0_universal2"), + OperationSystem.windows: url_format.format("win_amd64") +} + + +colors = { + "reset": "\x1b[0m", + "red": "\x1b[31m", + "green": "\x1b[32m", + "yellow": "\x1b[33m", + "cyan": "\x1b[36m", +} + + +def __log(color: str, msg: str): + print(f"{__c(color, msg)}") + + +def __is_windows(): + return sys.platform == "win32" and os.name == "nt" + + +def __c(color: str, msg: str) -> str: + if __is_windows(): + return msg + else: + return colors[color] + msg + colors["reset"] + + +def log_ok(msg: str): + __log("green", msg) + + +def log_warning(msg: str): + __log("yellow", f"WARNING: {msg}") + + +def log_error(msg: str): + __log("red", f"ERROR: {msg}") + + +def copy_wheels(source_folder: str, destination_folder: str, systems: list[OperationSystem], + skip_missing: bool) -> bool: + if not os.path.exists(source_folder): + return False + else: + os.makedirs(destination_folder, exist_ok=True) + for system in systems: + wheel_details = operation_systems[system] + for platform_name in wheel_details: + matching_files = glob.glob(f"{source_folder}/unity_cloud*-py3-none-{platform_name}.whl") + if len(matching_files) == 0: + msg = f"Could not find wheel file for {platform_name}" + if skip_missing: + log_warning(msg) + else: + log_error(msg) + return False + for source_file in matching_files: + destination_file = os.path.join(destination_folder, os.path.basename(source_file)) + if source_file != destination_file: + shutil.copy(source_file, destination_file) + print(f"\"{source_file}\" copied to \"{destination_file}\"") + return True + + +def __download_file(download_path: str, file_url: str) -> bool: + try: + response = urllib.request.urlopen(file_url) + except Exception as err: + log_error(f"Failed to download from {file_url}. Exception: {err}") + return False + + if response.code != 200: + log_error(f"Failed to download from {file_url}. Status code: {response.code}") + return False + + decoded_filename: str + if 'Content-Disposition' in response.headers: + content_disposition = response.headers['Content-Disposition'] + filename_match = re.search(r'filename\*=(?:UTF-8\'\'|utf-8\'\'|\'\'|"")?([^\'"]+)', content_disposition) + + if not filename_match: + log_error(f"Failed to download from {file_url}: Downloaded data has unexpected format") + return False + + utf8_encoded_filename = filename_match.group(1) + decoded_filename = utf8_encoded_filename + else: + decoded_filename = file_name = os.path.basename(file_url) + with open(os.path.join(download_path, decoded_filename), "wb") as file: + file.write(response.read()) + return True + + +def download_wheel(download_path: str, system: str, skip_missing: bool, + overwrite=False, write_log=True, ) -> bool: + os.makedirs(download_path, exist_ok=True) + if write_log: + print("Downloading unity-cloud wheel files...") + + wheel_name = operation_systems[system] + path = os.path.join(download_path, wheel_name) + if not os.path.exists(path) or overwrite: + if write_log: + print(f"Downloading wheel file for {wheel_name}...") + __download_file(download_path, wheel_name) + + else: + print(f"Skipping \"{path}\". The file already exists") + return True + + +def pip_install_wheel(download_path: str, system: str): + + #check if wheel is installed with a recent version + try: + version_check_command = [sys.executable, "-m", "pip", "show","unity_cloud"] + version_check_output = subprocess.run(version_check_command, check=True, capture_output=True, text=True) + version = re.search(r"Version: (\d+\.\d+\.\d+)", version_check_output.stdout).group(1) + if version >= sdk_version: + print(f"Unity Cloud SDK is already installed and up to date.") + return + + except Exception: + pass + + install_command = [sys.executable, "-m", "pip", "install", "wheel"] + try: + subprocess.run(install_command, check=True) + except subprocess.CalledProcessError: + sys.stderr.write(f"Failed to install wheel package\n") + return False + + wheel_name = wheel_names[system] + wheel_path = os.path.join(download_path, wheel_name) + install_command = [sys.executable, "-m", "pip", "install", wheel_path, "--force-reinstall"] + try: + subprocess.run(install_command, check=True) + print(f"Wheel {wheel_path} installed successfully.") + except subprocess.CalledProcessError: + sys.stderr.write(f"Failed to install wheel {wheel_path}\n") + + +def pip_install_other_libraries(): + install_command = [sys.executable, "-m", "pip", "install", "InquirerPy"] + try: + subprocess.run(install_command, check=True) + print(f"Other libraries installed successfully.") + except subprocess.CalledProcessError: + sys.stderr.write(f"Failed to install other libraries\n") + +def check_python_version(): + if sys.version_info < (3, 10): + return False + return True + +def check_install_requirements(): + try: + from InquirerPy import prompt + from InquirerPy.base.control import Choice + import unity_cloud + except ImportError: + return False + return True \ No newline at end of file diff --git a/bulk_upload_cli/test.json b/bulk_upload_cli/test.json new file mode 100644 index 0000000..dc0469b --- /dev/null +++ b/bulk_upload_cli/test.json @@ -0,0 +1,19 @@ +{ + "strategy": "unityPackage", + "filesCommonToEveryAssets": [], + "assetsPath": "C:\\UnitySrc\\test.unitypackage", + "assetFileExtensions": [], + "organizationId": "3574059373213", + "projectId": "29a62434-8899-455e-bf04-95f3c170a255", + "serviceAccount": { + "keyId": "", + "key": "" + }, + "amountOfParallelUploads": 15, + "collectionToLinkAssetTo": "None", + "tagsToApplyToAssets": [], + "assetNameCaseSensitive": false, + "metadataToApply": {}, + "updateFiles": true, + "description": "" +} diff --git a/catalog-info.yaml b/catalog-info.yaml index 3d15ed6..f2a79e8 100644 --- a/catalog-info.yaml +++ b/catalog-info.yaml @@ -3,14 +3,11 @@ apiVersion: backstage.io/v1alpha1 kind: Component metadata: annotations: - github.com/project-slug: Unity-Technologies/unity-cloud-python-sdk-samples - name: unity-cloud-python-sdk-samples + github.com/project-slug: unity/python-sdk-samples + name: python-sdk-samples description: "Various samples that show how to use the Unity Cloud Python SDK." labels: costcenter: "7054" - tags: - - planned-public - - enterprise links: - url: https://unity.slack.com/messages/C04R01SGG68/ title: "#uc-cs-am-python-sdk" @@ -18,4 +15,4 @@ metadata: spec: type: other lifecycle: experimental - owner: Unity-Technologies/unity-cloud-sdk-admin + owner: unity/digital-twins-sdk-admins diff --git a/release/SamplesRelease.md b/release/SamplesRelease.md new file mode 100644 index 0000000..13f4914 --- /dev/null +++ b/release/SamplesRelease.md @@ -0,0 +1,16 @@ +# Deploy samples to the public folder + +## Preparing the release branch + +1. Create a release branch in internal repo from develop. +2. Update changelog to bump the version of the sample. +3. Create a PR to merge this release branch into main in internal repo. Do not merge yet. +4. Share the release branch candidate with QA + + +## Publishing the release + +1. Make sure you have admin rights on the public repo. https://github.com/Unity-Technologies/unity-cloud-python-sdk-samples +2. Run the script `release\generate_release.py`. +3. In the public repo, publish a PR to the newly created branch. The description of the PR should be latest changelog. +4. Merge release PRs in both internal and public repo. \ No newline at end of file