Working with data in an S3-compatible object store

Procedure

1. From the OpenShift AI dashboard, click Projects.
2. Click the name of the project that contains the workbench.
3. Click the Workbenches tab.

4. If the status of the workbench is Running, skip to the next step.

   If the status of the workbench is Stopped, in the Status column for the workbench, click Start.

   The Status column changes from Stopped to Starting when the workbench server is starting, and then to Running when the workbench has successfully started.

5. Click the open icon ( ) next to the workbench.

   Your Jupyter environment window opens.

6. On the toolbar, click the Git Clone icon and then select Clone a Repository.
7. In the Clone a repo dialog, enter the following URL https://github.com/opendatahub-io/odh-doc-examples.git and then click Clone.
8. In the file browser, select the newly-created odh-doc-examples folder.

9. Double-click the newly created storage folder.

   You see a Jupyter notebook named s3client_examples.ipynb.

10. Double-click the s3client_examples.ipynb file to launch the Jupyter notebook.

    The Jupyter notebook opens. You see code examples for the following tasks:

    • Installing Boto3 and required Boto3 libraries
    • Creating an S3 client session
    • Creating an S3 client connection
    • Listing files
    • Creating a bucket
    • Uploading a file to a bucket
    • Downloading a file from a bucket
    • Copying files between buckets
    • Deleting an object from a bucket
    • Deleting a bucket

11. In the Jupyter notebook, locate the following instructions to install Boto3 and its required libraries, and run the code cell:

    #Upgrade pip to the latest version
    !pip3 install --upgrade pip
    
    #Install Boto3
    !pip3 install boto3
    
    #Install Boto3 libraries
    import os
    import boto3
    from botocore.client import Config
    from boto3 import session
    
    #Check Boto3 version
    !pip3 show boto3

    Copy to Clipboard Toggle word wrap

    The instructions in the code cell update the Python Package Manager (pip) to the latest version, install Boto3 and its required libraries, and display the version of Boto3 installed.

12. Locate the following instructions to create an S3 client and session. Run the code cell.

    #Creating an S3 client
    #Define credentials
    key_id = os.environ.get('AWS_ACCESS_KEY_ID')
    secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
    endpoint = os.environ.get('AWS_S3_ENDPOINT')
    region = os.environ.get('AWS_DEFAULT_REGION')
    
    #Define client session
    session = Boto3.session.Session(aws_access_key_id=key_id,
    aws_secret_access_key=secret_key)
    
    #Define client connection
    s3_client = Boto3.client('s3', aws_access_key_id=key_id,
    aws_secret_access_key=secret_key,aws_session_token=None,
        config=Boto3.session.Config(signature_version='s3v4'),
                            endpoint_url=endpoint,
                            region_name=region)

    Copy to Clipboard Toggle word wrap

    The instructions in the code cell configure an S3 client and establish a session to your S3-compatible object store.

Procedure

1. In the Jupyter notebook, locate the following instructions that lists available buckets and then run the code cell.

   #List available buckets
   s3_client.list_buckets()

   Copy to Clipboard Toggle word wrap

   A successful response includes an HTTP request status code of 200 and a list of buckets, similar to the following output:

   'HTTPStatusCode': 200,
   'Buckets': [{'Name': 'aqs086-image-registry',
   'CreationDate': datetime.datetime(2024, 1, 16, 20, 21, 36, 244000, tzinfo=tzlocal( ))},

   Copy to Clipboard Toggle word wrap

2. Locate the instructions that prints only the names of available buckets and execute the code cell.

   #Print only names of available buckets
   for bucket in s3_client.list_buckets()[‘Buckets’]:
       print(bucket[‘Name’])

   Copy to Clipboard Toggle word wrap

   The output displays the names of the buckets, similar to the following example.

   aqs086-image-registry
   aqs087-image-registry
   aqs135-image-registry
   aqs246-image-registry

   Copy to Clipboard Toggle word wrap

Procedure

1. In the notebook file, locate the following instructions to create a bucket:

   #Create bucket                                                `
   s3_client.create_bucket(Bucket='<bucket_name>')

   Copy to Clipboard Toggle word wrap

2. Replace <name_of_the_bucket> with the name of the bucket that you want to create, as shown in the example, and then run the code cell.

   #Create bucket
   s3_client.create_bucket(Bucket='aqs43-image-registry')

   Copy to Clipboard Toggle word wrap

   The output displays an HTTP response status code of 200, indicating a successful request.

Procedure

1. In the Jupyter notebook, locate the following code for listing files.

   #List files
   #Replace <bucket_name> with the name of the bucket.
   bucket_name = ‘<bucket_name>’
   s3_client.list_objects_v2(Bucket=bucket_name)

   Copy to Clipboard Toggle word wrap

2. Replace <bucket_name> with the name of your own bucket, as shown in the example, and then run the code cell.

   #List files
   #Replace <bucket_name> with the name of the bucket.
   bucket_name = ‘aqs27-registry’
   s3_client.list_objects_v2(Bucket=bucket_name)

   Copy to Clipboard Toggle word wrap

   The output displays information about the files that are available in the specified bucket.

3. Locate the code cell that lists only the names of the files.

   #Print only names of files
   bucket_name = ‘<bucket_name>’
   for key in s3_client.list_objects_v2(Bucket=bucket_name)[‘Contents’]:
   	print(key[‘Key’])

   Copy to Clipboard Toggle word wrap

4. Replace <bucket_name> with the name of your bucket, as shown in the example, and run the code cell:

   #Print only names of files
   bucket_name = ‘aqs27-registry’
   for key in s3_client.list_objects_v2(Bucket=bucket_name)[‘Contents’]:
   print(key[‘Key’])

   Copy to Clipboard Toggle word wrap

   The output displays a list of file names that are available in the specified bucket.

5. Refine the previous query to specify a file path, by locating the following code cell:

   bucket_name = ‘<bucket_name>’
   for key in s3_client.list_objects_v2(Bucket=bucket_name,Prefix=’<start_of_file_path’)[‘Contents’]:
   print(key[‘Key’])

   Copy to Clipboard Toggle word wrap
6. Replace <bucket_name> and <start_of_file_path> with your own values and run the code cell.

Procedure

1. In the Jupyter notebook, locate the following instructions to download files from a bucket:

   #Download file from bucket
   #Replace the following values with your own:
   #<bucket_name>: The name of the bucket.
   #<object_name>: The name of the file to download. Must include full path to the file on the bucket.
   #<file_name>: The name of the file when downloaded.
   
   s3_client.download_file('<bucket_name>','<object_name>','<file_name>')

   Copy to Clipboard Toggle word wrap

2. Modify the code sample:

   1. Replace <bucket_name> with the name of the bucket that the file is located in…​ Replace <object_name> with the name of the file that you want to download.

   2. Replace <file_name> with the name and path that you want the file to be downloaded to, as shown in the example.

      s3_client.download_file('aqs086-image-registry',
                              'series35-image36-086.csv',
                              '\tmp\series35-image36-086.csv_old')

      Copy to Clipboard Toggle word wrap
3. Run the code cell.

Procedure

1. In the Jupyter notebook, locate the instructions to upload files to a bucket.

   #Upload file to bucket
   #Replace <file_name>, <bucket_name>, and <object_name> with your values.
   #<file_name>: Name of the file to upload. This value must include the full local path to the file on your workbench.
   #<bucket_name>: The name of the bucket to upload the file to.
   #<object_name>: The full key to use to save the file to the bucket.
   
   s3_client.upload_file('<file_name>', '<bucket_name>', '<object_name>')

   Copy to Clipboard Toggle word wrap

2. Replace <file_name>, <bucket_name> and <object_name> with your own values, as shown in the example, and then run the code cell.

   s3_client.upload_file('image-973-series123.csv', 'aqs973-image-registry', '/tmp/image-973-series124.csv')

   Copy to Clipboard Toggle word wrap

Procedure

1. In the notebook, locate the following instructions to copy files between buckets:

   #Copying files between buckets
   #Replace the placeholder values with your own.
   copy_source = {
       	'Bucket': '<bucket_name>',
       	'Key': '<key>'
   }
   s3_client.copy(copy_source, '<destination bucket>', '<destination_key>')

   Copy to Clipboard Toggle word wrap

2. Within the copy_source block, replace <bucket_name> with the name of the source bucket and <key> with the key of the source file, as shown in the example.

   copy_source = {
       	'Bucket': 'aqs086-image-registry',
       	'Key': 'series43-image12-086.csv'
   }

   Copy to Clipboard Toggle word wrap

3. Replace the <destination_bucket> with the name of the bucket to copy to, and <destination_key> with the name of the key to copy to, as shown in the example. Execute the code cell.

   s3_client.copy(copy_source, 'aqs971-image-registry', '/tmp/series43-image12-086.csv')

   Copy to Clipboard Toggle word wrap

Procedure

1. In the Jupyter notebook, locate the following instructions to delete files from a bucket:

   #Delete files from bucket
   s3_client.delete_object(Bucket='<bucket_name>', Key='<object_key>')

   Copy to Clipboard Toggle word wrap

2. Replace <bucket_name> with the name of your bucket and <key> with the key of the file you want to delete, as shown in the example. Run the code cell.

   #Delete object from bucket
   s3_client.delete_object(Bucket='aqs971-image-registry', Key='/tmp/series43-image12-086.csv')

   Copy to Clipboard Toggle word wrap

   The output displays a HTTP response status code of 204, which indicates that the request was successful.

Procedure

1. In the Jupyter notebook, locate the following instructions to delete a bucket:

   #Delete bucket
   s3_client.delete_bucket(Bucket='<bucket_name>')

   Copy to Clipboard Toggle word wrap

2. Replace <bucket_name> with the name of the bucket that you want to delete and run the code cell.

   #Delete bucket
   s3_client.delete_bucket(Bucket='aqs971-image-registry')

   Copy to Clipboard Toggle word wrap

   The output displays an HTTP response status code of 200, which indicates that the request was successful.

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

   oc login api.<cluster_name>.<cluster_domain>:6443 --web

   Copy to Clipboard Toggle word wrap

2. Retrieve the current OpenShift AI trusted CA configuration and store it in a new file:

   oc get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq -r '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crt

   Copy to Clipboard Toggle word wrap

3. Add the cluster’s kube-root-ca.crt ConfigMap to the OpenShift AI trusted CA configuration:

   oc get configmap kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" >> /tmp/my-custom-ca-bundles.crt

   Copy to Clipboard Toggle word wrap

4. Update the OpenShift AI trusted CA configuration to trust certificates issued by the certificate authorities in kube-root-ca.crt:

   oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' /tmp/my-custom-ca-bundles.crt)"'"}]'

   Copy to Clipboard Toggle word wrap