PL/Container supports the use of multiple images. While some images are provided by PL/Container itself, users have the flexibility to create and use their own custom-built images. This topic covers the following details:

• Prerequisites
• Loading the Base Container Image to Your Machine
• Using Dockerfile to Customize New Image
• Verifying the PL/Container Compatibility with the Container Image
• Adding a New Name to Your Container Image and Export
• Distributing the Image to Greenplum Cluster

Prerequisites

This document offers a practical demonstration on how to integrate PyTorch into an existing plcontainer-python3 image. The requirements for this process include:

• Docker installation that includes support for the docker image save and docker image load commands
• Internet access

Loading the Base Container Image to Your Machine

In this demonstration, we will use the official PL/Container Python3 image as our base container image. However, any container image can be utilized as a PL/Container image.

If you do not want to use the official Python3 image, you can skip this step.

1. Download the lateset Greenplum Procedural Languages > Pl/Container Image for Python3 from the Broadcom Support Portal. We've used version 2.1.4 as an example for this demostration.

2. Load the container image to your local machine.

   ➜ docker image load < plcontainer-python3-image-2.1.4-gp6.tar.gz
   21639b09744f: Loading layer [==================================================>]  65.51MB/65.51MB
   e9b2e396bb54: Loading layer [==================================================>]  3.584kB/3.584kB
   3a1e4239aec6: Loading layer [==================================================>]  3.072kB/3.072kB
   8aaf3ed0fba9: Loading layer [==================================================>]  6.716GB/6.716GB
   Loaded image: pivotaldata/plcontainer_python3_shared:devel
   

3. Check if the container image has been loaded correctly.

   ➜ docker images
   REPOSITORY                               TAG       IMAGE ID       CREATED         SIZE
   pivotaldata/plcontainer_python3_shared   devel     e4cda7f63158   6 months ago    6.71GB
   

Using Dockerfile to Customize New Image

1. Save the following code with a filename Dockerfile.

   FROM plcontainer_python3_shared:devel
   RUN pip3 install torch==1.10.2+cpu torchvision==0.11.3+cpu torchaudio==0.10.2+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html
   

2. Run the following command.

   docker build -t plcontainer_python3_with_torch:latest;
   

   Once the Docker build is complete, we will have a new Python3 image with Torch 1.10.2 installed.

Verifying the PL/Container Compatibility with the Container Image

1. Download the PL/Container installer.

   plcontainer-2.4.0-gp6-rhel7_x86_64.gppkg
   

2. Unzip the installer to get the PL/Container client binary.

   ➜ mkdir -p plcontainer; tar xzfv plcontainer-2.4.0-gp6-rhel7_x86_64.gppkg -C plcontainer
   ➜ cd plcontainer;
   ➜ rpm2cpio plcontainer-0.0-0.x86_64.rpm | cpio -idmv
   ➜ ls -l temp/bin/plcontainer_clients/
   .rwxr-xr-x 172k sa 13 Dec  2021 librcall.so
   .rwxr-xr-x 566k sa 13 Dec  2021 py3client
   .rwxr-xr-x  256 sa 13 Dec  2021 py3client.sh
   .rwxr-xr-x 541k sa 13 Dec  2021 pyclient
   .rwxr-xr-x  255 sa 13 Dec  2021 pyclient.sh
   .rwxr-xr-x 270k sa 13 Dec  2021 rclient
   .rwxr-xr-x  255 sa 13 Dec  2021 rclient.sh
   

3. Execute the client binary to verify whether this container image is compatible with the PL/Container client.

   ➜ docker run -it --rm -v $(readlink -f <path to client binary>/bin/plcontainer_clients):/clientdir pivotaldata/plcontainer_python3_with_torch:latest /clientdir/py3client.sh
   plcontainer log: pythonclient, unknown, unknown, -1, WARNING: USE_CONTAINER_NETWORK is not set, use default value "no".
   plcontainer log: pythonclient, unknown, unknown, -1, ERROR: Cannot bind the addr: No such file or directory
   

4. Due to some limitations, there will be an error message displayed even when the Container image is compatible with the PL/Container client. Run the following command to check for the error message. If there are errors that begin with plcontainer log, it indicates that the image is compatible with the PL/Container client. However, if the image is not compatible, you will receive messages similar to the following.

   ➜ docker run -it --rm -v $(readlink -f <path to client binary>/bin/plcontainer_clients):/clientdir pivotaldata/plcontainer_python3_with_torch:latest /clientdir/py3client.sh
   ./py3client: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
   

   If the container image is not compatible with the PL/Container client, you will need to rebuild the client from the source. The source code for this can be accessed from R Client and Python Client.

Adding a New Name to Your Container Image and Export

1. Run the following command.

   docker save plcontainer_python3_with_torch:latest | gzip > plcontainer_python3_with_torch.tar.gz
   

2. You now have the plcontainer_python3_with_torch.tar.gz file, which is the image that can be loaded into PL/Container.

Distributing the Image to Greenplum Cluster

1. Run the following command.

   plcontainer image-add -f plcontainer_python3_with_torch.tar.gz
   

   You can now proceed with the standard process to continue using PL/Container.