[GUIDE]: How to update Python

[BobbyESP]

[GUIDE]: How to update Python

Hi guys, today I'm creating this issue to show you all the process I followed in order to build the Python package from source for this library.

⚠️THIS METHOD HASN'T BEEN REVIEWED AND IS STILL IN A PERSONAL INVESTIGATION STAGE. OTHER youtubedl-android MAINTAINERS MAY MODIFY THIS ISSUE IN ORDER TO CORRECT IT

We are about to update Python to the latest possible version available in the Termux repository nowadays - (C)Python 3.11

Prerequisites

• Docker (for creating a conatainer with all Termux available packages)
• Git (for retrieving the repository from GitHub)
• A Linux operating system / WSL (Windows Subsystem for Linux)

Steps

1. Clone the termux-packages repository

From here you are about to download all the available Termux packages, which contains the scripts and patches needed to build Python for Termux.

To do this, you have to enter the next commands in the Terminal:

git clone https://github.com/termux/termux-packages.git
cd termux-packages

2. Create the Python builder file

Now we are about to get all the needed binaries for creating our final Python zip for being used in youtubedl-android. The output of this will be files with the .deb extension, installable packages for Linux, the ones we are later going to de-package.

I have created for you a little bash script for building the binaries for ALL 4 possible architectures:

#!/bin/bash

# Define the architectures we want to build for
ARCHITECTURES=("aarch64" "arm" "i686" "x86_64")

# Define the fake user directories
FAKE_USER_PREFIX=/data/youtubedl-android/usr
FAKE_USER_HOME=/data/youtubedl-android/home

# Define the output directory base
OUTPUT_BASE_DIR=$(pwd)/output

# Loop over each architecture and build Python
for ARCH in "${ARCHITECTURES[@]}"; do
    echo "Building Python for architecture: $ARCH"
    
    # Set the environment variables for the current architecture
    export TERMUX_ARCH=$ARCH
    export TERMUX_PREFIX=$FAKE_USER_PREFIX
    export TERMUX_ANDROID_HOME=$FAKE_USER_HOME

    # Create a separate output folder for each architecture
    OUTPUT_DIR=${OUTPUT_BASE_DIR}/${ARCH}
    mkdir -p $OUTPUT_DIR

    # Run the build script for Python
    ./build-package.sh -a $ARCH -o $OUTPUT_DIR python

    if [ $? -ne 0 ]; then
        echo "Build failed for architecture: $ARCH"
        exit 1
    fi

    # Move files containing the architecture name or "all" to the OUTPUT_DIR
    echo "Moving files for architecture: $ARCH to $OUTPUT_DIR"
    find $OUTPUT_BASE_DIR -maxdepth 1 -type f \( -name "*$ARCH*" -o -name "*all*" \) -exec mv {} $OUTPUT_DIR/ \;

    echo "Build completed for architecture: $ARCH. Output is in $OUTPUT_DIR."
done

echo "Python build completed for all architectures."

Copy this whole content into a file named (for example) build-python.sh and give to it run permissions by running:

chmod +x ./build-python.sh

3. Build the Python package

Now we are about to build the Python packages by running the next commands (this will start a Docker container and all the processes will be running in there)

./scripts/run-docker.sh ./clean.sh
./scripts/run-docker.sh ./build-python.sh

Once this two commands finished running, there will be a folder called output in the same directory you ran the command; in there you will find 4 different folders, one for each built architecture (or less in case you deleted some from the architectures array in the build-python.sh file).

4. Merge packages into the .zip file

I have found the following packages (.deb) to be enough for youtube-dl to work, but this may vary depending on which yt-dlp functions you will use.

• python_3.11.9-5_aarch64.deb
• libandroid-posix-semaphore_0.1-3_aarch64.deb
• libandroid-support_29_aarch64.deb
• libffi_3.4.6-1_aarch64.deb
• zlib_1.3.1_aarch64.deb
• openssl_1:3.3.1_aarch64.deb
• ca-certificates_1:2024.07.02_all.deb
  ⚠️WARNING: Versions may vary depending on the date of building, but the process remains the same.

Now let's create the final usable Python package by de-packaging every .deb file that we want to be used. For this run the next commands:

⚠️WARNING: Replace "{arch}" with the wanted architecture

First of all, create in the output folder a file called deb-unpackage.sh, open it with some text editor and put the next content into it:

#!/bin/bash

# Define an array with the base names of the packages you want to match.
packages=(
    "python"
    "libandroid-support"
    "libffi"
    "zlib"
    "openssl"
    "ca-certificates"
)

# Flag to include "static" packages
include_static=false  # Set to true if you want to include "static" packages

# Variable to store the directory where .deb files are located
deb_dir=""

# Function to display usage information
usage() {
    echo "Usage: $0 [-s] -d <directory>"
    echo "  -s            Include 'static' packages"
    echo "  -d <directory>  Specify the directory where the .deb files are located"
    exit 1
}

# Process command-line arguments
while getopts "sd:" opt; do
    case $opt in
        s)
            include_static=true
            ;;
        d)
            deb_dir="$OPTARG"
            ;;
        \?)
            usage
            ;;
    esac
done

# Check if the deb_dir variable is empty (i.e., the directory wasn't provided)
if [ -z "$deb_dir" ]; then
    echo "Error: Directory not specified."
    usage
fi

# Loop through the array and match the filenames
for package in "${packages[@]}"; do
    if $include_static; then
        # Match both regular and static packages
        pattern="${deb_dir}/${package}*.deb"
    else
        # Match only non-static packages
        pattern="${deb_dir}/${package}[^-static]*.deb"
    fi

    for deb_file in $pattern; do
        if [[ -f $deb_file ]]; then
            echo "Extracting $deb_file"
            dpkg-deb -xv "$deb_file" .
        fi
    done
done

This script automates the extraction of .deb packages using dpkg-deb. It allows users to specify a set of package base names to be extracted, with an optional flag to include "static" versions of these packages (-s). Additionally, it supports specifying the directory where the .deb files are located by using the flag -d.

Now make it an executable by executing: chmod +x ./deb-unpackage.sh

You have to run this command once per architecture (I'm not doing an architecture-batch depackager because of unnecessary complexity).

An example of its usage (working on the output directory) is:

• For arm64-v8a

./deb-unpackage.sh -d ./aarch64

This will create inside the aarch64 folder a folder called data that contains the content of our Python update.

To finish we have to navigate to the desired architecture folder by using cd {arch} and execute the next command:

cd data/data/com.termux/files
cp usr/bin/python3.11 /tmp/libpython_{arch}.so
zip --symlinks -r /tmp/python3_11_{arch}.zip usr/lib usr/etc # ⚠️Remember to change the "{arch}" with the desired architecture

Now navigate to the /temp directory and you will find the built Python packages.

5. The Python Executable

In the jniLibs folder, you should notice two key files: libpython.zip.so and libpython.so.

• The libpython.zip.so file is the zipped package you created in the previous steps.
• The libpython.so file is extracted from the bin directory within the generated data folder (created in Step 4).

The code snippet above already handles copying the libpython.so file to the /tmp directory, along with the python3_11_{arch}.zip file.

Now, follow these final steps:

1. Rename the python3_11_{arch}.zip file to libpython.zip.so.
2. Rename the libpython_{arch}.so to libpython.so
3. Move the renamed files (libpython.zip.so and libpython.so) to the corresponding jniLibs subfolder. For example, if you are using the aarch64 architecture, move the files to the arm64-v8a folder.

6. jniLibs folder structure

Ensure that your jniLibs folder contains the following structure for each architecture. This ensures that the application can correctly locate the Python binaries.

In the jniLibs folder you should have 4 folders, each one named with the available CPU instructions architecture. Taking for example the arm64-v8a folder, you must have:

• libpython.zip.so - The generated zip file from the termux-packages repository
• libpython.so - The Python executable that has been copied automatically by the code snippet to the /tmp directory

[syedusama5556]

thanks for this I tried it and it worked I had some issues during the build on one Ubuntu machine due to permission issues so I added some updated commands in your method just to help others I have also made a fork and uploaded the latest libs if someone wants to download them directly

https://github.com/syedusama5556/youtubedl-android-updated/tree/master/library/src/main/jniLibs

[REVISED-GUIDE]: How to update Python

Hi guys, today I'm creating this issue to show you all the process I followed in order to build the Python package from source for this library.

⚠️THIS METHOD HASN'T BEEN REVIEWED AND IS STILL IN A PERSONAL INVESTIGATION STAGE. OTHER youtubedl-android MAINTAINERS MAY MODIFY THIS ISSUE IN ORDER TO CORRECT IT

We are about to update Python to the latest possible version available in the Termux repository nowadays - (C)Python 3.11

Prerequisites
Docker (for creating a conatainer with all Termux available packages)
Git (for retrieving the repository from GitHub)
A Linux operating system / WSL (Windows Subsystem for Linux)
Steps

1. Clone the termux-packages repository
   From here you are about to download all the available Termux packages, which contains the scripts and patches needed to build Python for Termux.

To do this, you have to enter the next commands in the Terminal:

git clone https://github.com/termux/termux-packages.git
cd termux-packages

2. Create the Python builder file
   Now we are about to get all the needed binaries for creating our final Python zip for being used in youtubedl-android. The output of this will be files with the .deb extension, installable packages for Linux, the ones we are later going to de-package.

I have created for you a little bash script for building the binaries for ALL 4 possible architectures:

#!/bin/bash

# Define the architectures we want to build for
# the architectures are used for the following devices:
# aarch64 - arm64-v8a (64-bit ARM) 
# arm - armeabi-v7a (32-bit ARM)
# i686 - x86 (32-bit x86)
# x86_64 - x86_64 (64-bit x86)

ARCHITECTURES=("aarch64" "arm" "i686" "x86_64")

# Define the fake user directories
FAKE_USER_PREFIX=/data/youtubedl-android/usr
FAKE_USER_HOME=/data/youtubedl-android/home

# Define the output directory base
OUTPUT_BASE_DIR="${PWD}/output"

# Loop over each architecture and build Python
for ARCH in "${ARCHITECTURES[@]}"; do
    echo "Building Python for architecture: $ARCH"
    
    # Set the environment variables for the current architecture
    export TERMUX_ARCH=$ARCH
    export TERMUX_PREFIX=$FAKE_USER_PREFIX
    export TERMUX_ANDROID_HOME=$FAKE_USER_HOME

    # Create a separate output folder for each architecture
    OUTPUT_DIR="${OUTPUT_BASE_DIR}/${ARCH}"
    mkdir -p "$OUTPUT_DIR"

    # Run the build script for Python
    ./build-package.sh -a "$ARCH" -o "$OUTPUT_DIR" python

    # Check if the build script executed successfully
    if [ $? -ne 0 ]; then
        echo "Build failed for architecture: $ARCH"
        exit 1
    fi

    # Move files containing the architecture name or "all" to the OUTPUT_DIR
    echo "Moving files for architecture: $ARCH to $OUTPUT_DIR"
    find "${OUTPUT_BASE_DIR}" -maxdepth 1 -type f \( -name "*$ARCH*" -o -name "*all*" \) -exec mv {} "$OUTPUT_DIR/" \;

    echo "Build completed for architecture: $ARCH. Output is in $OUTPUT_DIR."
done

echo "Python build completed for all architectures."

Copy this whole content into a file named (for example) build-python.sh and give to it run permissions by running:

chmod +x ./build-python.sh
cd ..
chmod 777 termux-packages
cd termux-packages
mkdir output
chmod 777 output

3. Build the Python package
   Now we are about to build the Python packages by running the next commands (this will start a Docker container and all the processes will be running in there)

Note: This process may take a while, depending on your computer's performance and internet speed.

./scripts/run-docker.sh ./clean.sh
./scripts/run-docker.sh ./build-python.sh

Once this two commands finished running, there will be a folder called output in the same directory you ran the command; in there you will find 4 different folders, one for each built architecture (or less in case you deleted some from the architectures array in the build-python.sh file). if incase they are not sorted based on architecture then follow below step else you can skip it

(optional) create a new move_accordingto_arch.sh file in the output directory then enter

#!/bin/bash

# Create directories for each architecture if they don't exist
mkdir -p aarch64 arm i686 x86_64

# Move files to their respective directories based on architecture
for arch in aarch64 arm i686 x86_64; do
    # Find and move files that end with the architecture type
    find . -type f -name "*_$arch.deb" -exec mv {} ./$arch/ \;
done

echo "Files have been moved to their respective folders."

make the script executeable and run it

chmod +x move_accordingto_arch.sh
./move_accordingto_arch.sh

now all will be sorted according to the ARCH now you can move to the next step.

4. Merge packages into the .zip file
   I have found the following packages (.deb) to be enough for youtube-dl to work, but this may vary depending on which yt-dlp functions you will use.

- python_3.11.9-5_aarch64.deb
- libandroid-posix-semaphore_0.1-3_aarch64.deb
- libandroid-support_29_aarch64.deb
- libffi_3.4.6-1_aarch64.deb
- zlib_1.3.1_aarch64.deb
- openssl_1:3.3.1_aarch64.deb
- ca-certificates_1:2024.07.02_all.deb

⚠️WARNING: Versions may vary depending on the date of building, but the process remains the same.

Now let's create the final usable Python package by de-packaging every .deb file that we want to be used. For this run the next commands:

⚠️WARNING: Replace "{arch}" with the wanted architecture

First of all, create in the output folder a file called deb-unpackage.sh, open it with some text editor and put the next content into it:

#!/bin/bash

# Define an array with the base names of the packages you want to match.
packages=(
    "python"
    "libandroid-support"
    "libffi"
    "zlib"
    "openssl"
    "ca-certificates"
)

# Flag to include "static" packages
include_static=false  # Set to true if you want to include "static" packages

# Variable to store the directory where .deb files are located
deb_dir=""

# Function to display usage information
usage() {
    echo "Usage: $0 [-s] -d <directory>"
    echo "  -s            Include 'static' packages"
    echo "  -d <directory>  Specify the directory where the .deb files are located"
    exit 1
}

# Process command-line arguments
while getopts "sd:" opt; do
    case $opt in
        s)
            include_static=true
            ;;
        d)
            deb_dir="$OPTARG"
            ;;
        \?)
            usage
            ;;
    esac
done

# Check if the deb_dir variable is empty (i.e., the directory wasn't provided)
if [ -z "$deb_dir" ]; then
    echo "Error: Directory not specified."
    usage
fi

# Loop through the array and match the filenames
for package in "${packages[@]}"; do
    if $include_static; then
        # Match both regular and static packages
        pattern="${deb_dir}/${package}*.deb"
    else
        # Match only non-static packages
        pattern="${deb_dir}/${package}[^-static]*.deb"
    fi

    for deb_file in $pattern; do
        if [[ -f $deb_file ]]; then
            echo "Extracting $deb_file"
            dpkg-deb -xv "$deb_file" .
        fi
    done
done

then

chmod +x deb-unpackage.sh

This script automates the extraction of .deb packages using dpkg-deb. It allows users to specify a set of package base names to be extracted, with an optional flag to include "static" versions of these packages (-s). Additionally, it supports specifying the directory where the .deb files are located by using the flag -d.

Now make it an executable by executing: chmod +x ./deb-unpackage.sh

You have to run this command once per architecture (I'm not doing an architecture-batch depackager because of unnecessary complexity).

An example of its usage (working on the output directory) is:

For arm64-v8a

./deb-unpackage.sh -d ./aarch64
./deb-unpackage.sh -d ./arm
./deb-unpackage.sh -d ./i686
./deb-unpackage.sh -d ./x86_64

This will create inside the aarch64 folder a folder called data that contains the content of our Python update.

To finish we have to navigate to the desired architecture folder by using cd {arch} and execute the next command:

cp {arch}/data/data/com.termux/files/usr/bin/python3.11 /tmp/libpython_{arch}.so
zip --symlinks -r /tmp/python3_11_{arch}.zip usr/lib usr/etc 

Now navigate to the /temp directory and you will find the built Python packages.

5. The Python Executable
   In the jniLibs folder, you should notice two key files: libpython.zip.so and libpython.so.

The libpython.zip.so file is the zipped package you created in the previous steps.
The libpython.so file is extracted from the bin directory within the generated data folder (created in Step 4).
The code snippet above already handles copying the libpython.so file to the /tmp directory, along with the python3_11_{arch}.zip file.

Now, follow these final steps:

Rename the python3_11_{arch}.zip file to libpython.zip.so.
Rename the libpython_{arch}.so to libpython.so
Move the renamed files (libpython.zip.so and libpython.so) to the corresponding jniLibs subfolder. For example, if you are using the aarch64 architecture, move the files to the arm64-v8a folder.
6. jniLibs folder structure
Ensure that your jniLibs folder contains the following structure for each architecture. This ensures that the application can correctly locate the Python binaries.

In the jniLibs folder you should have 4 folders, each one named with the available CPU instructions architecture. Taking for example the arm64-v8a folder, you must have:

libpython.zip.so - The generated zip file from the termux-packages repository
libpython.so - The Python executable that has been copied automatically by the code snippet to the /tmp directory

OR you can use the below script to automate step 5 and 6

#!/bin/bash

# Define the path to the original directory
original_dir=$(pwd)

# Define an array of architectures
architectures=("aarch64" "armv7" "x86_64" "i686")

# Clean up the /tmp directory
echo "Cleaning up /tmp directory..."
rm -f /tmp/python3_11_*.zip /tmp/libpython_*.so


# Loop through each architecture
for arch in "${architectures[@]}"; do
    # Create and change to the architecture-specific directory
    mkdir -p "${arch}"
    cd "${arch}" || exit

    # Print the current directory for debugging
    echo "Processing architecture: ${arch}"
    echo "In directory: $(pwd)"

    # Run the deb-unpackage.sh script using the original directory path
    if [ -f "${original_dir}/deb-unpackage.sh" ]; then
        echo "Found deb-unpackage.sh"
        "${original_dir}/deb-unpackage.sh" -d .
    else
        echo "Error: deb-unpackage.sh not found in the original directory"
        exit 1
    fi

    # Copy the Python executable
    cp -r data/data/com.termux/files/usr/bin/python3.11 /tmp/libpython_${arch}.so

    # Create a zip archive
    zip --symlinks -r /tmp/python3_11_${arch}.zip . -i usr/lib usr/etc

    # Create the jniLibs directory if it doesn't exist
    mkdir -p "${original_dir}/jniLibs/${arch}"

    # Move the renamed files to the corresponding jniLibs subfolder
    mv /tmp/python3_11_${arch}.zip "${original_dir}/jniLibs/${arch}/libpython.zip.so"
    mv /tmp/libpython_${arch}.so "${original_dir}/jniLibs/${arch}/libpython.so"

    # Return to the original directory
    cd "${original_dir}" || exit
done

[BobbyESP]

Nice @syedusama5556 ! Thanks for sharing it.

[SOCK-MAGIC]

It would be even better if we could use GitHub action to schedule releases

[BobbyESP]

It would be even better if we could use GitHub action to schedule releases

That's a pretty good idea. At some time I may create one