Setup MinIO with Docker

MinIO is an open-source High Performance Object Storage solution. It is API compatible with the Amazon S3 cloud storage service. One can use MinIO to setup high performance infrastructure for ML, analytics and application data workloads. MinIO is a good on-prem alternative to cloud object storage.

This article will provide a brief introduction to running MinIO on your machine and familiarise the reader with its’ basics. Please do not use these settings in production!

Continue reading to find out how to

1. set up a single node MinIO instance
2. generate access tokens
3. create a MinIO bucket
4. add files to MinIO bucket
5. retrieve files from MinIO bucket
6. generate pre-signed URLs to serve MinIO objects

Setting up MinIO

For setting up MinIO, we will be making use of Docker, specifically docker-compose. I have covered using docker-compose previously and highly recommend going through the article if you’re unfamiliar with it’s use.

Copy the contents of the code snippet below and create a file named docker-compose.yml in the directory where you want to store your MinIO objects.

version: '3'
services:
  minio:
    image: quay.io/minio/minio:RELEASE.2022-02-18T01-50-10Z
    volumes:
      - ./data:/data
    ports:
      - 9000:9000
      - 9001:9001
    environment:
      MINIO_ROOT_USER: 'minio_user'
      MINIO_ROOT_PASSWORD: 'minio_password'
      MINIO_ADDRESS: ':9000'
      MINIO_CONSOLE_ADDRESS: ':9001'
    command: minio server /data

The recipe above will setup a single node MinIO instance running on port 9001 for the MinIO Console and port 9000 for the MinIO API. It will also mount the data folder from inside the container to a data folder on your machine (in the same directory as the docker-compose.yml file). In case the container is stopped, all objects stored on the volume will persist.

Fire up a terminal and navigate to the directory of the docker-compose.yml file and execute the command below

docker-compose up -d

Docker will pull the MinIO image from DockerHub and setup the container. Provided there were no errors, you should be able to see your container running on Docker Desktop.

MinIO container running on Docker Desktop

Head on over to localhost:9001 and you should be greeted with the MinIO Console login screen.

MinIO Console Login

Congratulations! You just hosted a MinIO instance on your machine! Login to the MinIO Console and familiarise yourself with the UI.

MinIO Console Monitoring Metrics

Read more here if you’re interested in setting up MinIO in distributed mode.

Generate Access Tokens

To upload/retrieve files from MinIO programatically, we will be making use of the Python MinIO API.

Before we get to the code, we must generate an access token from the MinIO console. Head on over to the MinIO Console and navigate to Identity -> Service Accounts -> Create Service Accounts -> Create. You will then see a modal with your Access Key and Secret Key. Write these down, as this is the only time the secret will be displayed.

Access Key and Secret Key from MinIO Console

Next, install the Python minio library. We will then create a MinIO client in Python like so,

# Create client with access and secret key
client = Minio('http://localhost:9000/', 
               'minio_secret_key',  
               'minio_secret_code', 
                secure=False)

All MinIO API requests will be made through this client. Please note that we’re not using TLS for our connection and hence the secure parameter is set to False.

Creating a MinIO bucket

After establishing connection to MinIO, we can create a bucket programatically like so

# check if bucket already exists
found = client.bucket_exists("test_bucket")
# create bucket if it does not exist
if not found:
    client.make_bucket("test_bucket")
else:
    print("Bucket 'test_bucket' already exists")

We can also create a bucket using the MinIO Console. Navigate to Buckets -> Create Bucket.

Adding files to MinIO bucket

After creating a bucket in MinIO, we can add objects to it like so

# upload file to MinIO bucket
client.fput_object(
    "bucket_name", "file_name", "path_to_file_including_file_name",
)

We can also upload files to a bucket using the MinIO Console. Navigate to Buckets -> your desired bucket -> Browse -> Upload Files.

Retrieving files from MinIO bucket

We can retrieve files stored to a MinIO bucket like so

# retrieve file from MinIO bucket
bucketclient.fget_object("bucket_name", "file_name", "path_to_download_file_including_file_name")

One can also download the file directly from MinIO Console. Navigate to Buckets -> your desired bucket -> Browse -> Click on desired file -> Download.

Generating pre-signed URLs to serve MinIO objects

There can be use-cases where we may want to serve MinIO files via a URL instead of programatically retrieving the file from a bucket. We can generate a pre-signed URL like so,

# generate pre-signed URLs
image_url = client.get_presigned_url("GET", "bucket_name", 
                                    "file_name",
                                    expires=timedelta(days=7))

A pre-signed URL has a maximum expiration of 7 days.

This concludes this brief introduction to MinIO and using the MinIO Python API to upload/retrieve files from a bucket. If you’re interested to learn more about MinIO, I highly recommend going through their official documentation. Thank your for reading!

That’s all folks!