Guide to Integrating With Non-AWS Cloud Storage Services Supporting the S3 Protocol using Golang

17 Apr 2024

The Simple Storage Service (S3) protocol is a standard interface developed by Amazon Web Services (AWS) for storing and retrieving data in the cloud. It’s a key-value store, meaning that each piece of data (an object) is associated with a unique identifier (a key) that you use to access that data. The S3 protocol is crucial in cloud storage services for several reasons:

  1. Scalability
    S3 allows for virtually unlimited storage, making it ideal for applications that need to scale rapidly or handle large amounts of data.
  2. Durability and Availability
    S3 is designed for 99.999999999% (11 9’s) of durability, meaning the chances of losing data are extremely low. It also guarantees 99.99% availability, ensuring that your data is accessible when you need it.
  3. Security
    S3 provides robust security features, including encryption for data at rest and in transit, access control policies, and logging capabilities.
  4. Interoperability
    Many cloud storage providers, not just AWS, support the S3 protocol, making it a versatile choice for developers.
  5. Cost-Effective
    With S3, you only pay for the storage you use, making it a cost-effective solution for businesses of all sizes.

In summary, the S3 protocol is a reliable, secure, and scalable solution for storing data in the cloud, making it a cornerstone of modern cloud-based applications.

Setting Up Go Environment

install AWS SDK into our project

go get
go get
go get
go get
go get

Simple project to upload file

AWS SDK Wrapper

package s3client

import (

var (
 uploader *manager.Uploader
 once     sync.Once

func setup() {
 endpoint := ""
 accessKey := "XVXQ8Q3QGQ9QXQ8QXQ8Q"

 resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  return aws.Endpoint{
   URL:               endpoint,
   HostnameImmutable: true,
   Source:            aws.EndpointSourceCustom,
  }, nil
 awsConfig, err := config.LoadDefaultConfig(context.TODO(),
  config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
 if err != nil {
  log.Fatalf("failed to load config, %v", err)

 uploader = manager.NewUploader(s3.NewFromConfig(awsConfig)) // Initialize S3 client

func UploadFile(ctx context.Context, bucket, filename string, file io.Reader) (string, error) {
 result, err := uploader.Upload(ctx, &s3.PutObjectInput{
  Bucket: aws.String(bucket),
  Key:    aws.String(filename),
  Body:   file,
  ACL:    "public-read",
 if err != nil {
  log.Fatalf("failed to upload file, %v", err)

 return result.Location, nil

Why we need to wrap third-party library?

The purpose wrapper is to simplify the library’s interface, adapt it to fit our application’s specific needs, or add additional functionality. In this case we simplifies the process of uploading a file to S3 by providing a single UploadFile function that handles the setup and upload process. This makes it easier for the rest of our application to upload files to S3, as it only needs to call s3client.UploadFile with the appropriate parameters.

Anyway, let’s explain the code a little bit

We need to setup our project to connect with the cloud storage services, in the setupfunction there is 2 key steps that we need to highlight.

First, in resolver parameter we need to pass aws.EndpointSourceCustomto indicate we didn’t connect to AWS endpoint.

Secondly,config.WithRegion("auto")when load default config.

Then in UploadFile function we use once.Do(setup) to make sure setup function only called once.

Main Function

We will create simple server using chi, then create POST API to upload file to our S3 Cloud Storage. Also Hello World endpoint to make sure our server is working 😆.

package main

import (

func main() {
 r := chi.NewRouter()

 r.Get("/", func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello, World!"))

 r.Post("/", HandleUpload)
 http.ListenAndServe(":8080", r)

func HandleUpload(w http.ResponseWriter, r *http.Request) {
 file, header, err := r.FormFile("image")
 if err != nil {
  http.Error(w, err.Error(), http.StatusBadRequest)
 defer file.Close()

 // TODO: Do Validation

 // Upload file to S3
 filename := "suffix_" + header.Filename
 url, err := s3client.UploadFile(r.Context(), "bucket-name", filename, file)
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)


Let’s explain the HandleUploadfunction:

We will use r.FormFile("image") to get file from request, in other framework like mux and fiber we only get the header, so if we need to use header.Open() to get the file from header(CMIIW).

Then do some validation, after that we simply call our wrapper function s3client.UploadFile to upload to our cloud storage.


In this article we learn about S3 protocol, how to Integrate with Non-AWS Cloud Storage Service that supports S3 protocol, and a little bit about wrapper.

Hopefully, this article can be helpful. Don’t hesitate to give criticism and suggestions if you find any mistakes in this article.

Let’s connect on and


artikel berbahasa indonesia dapat ditemukan di