Uploading and deleting an entire directory to amazon s3 using Transfer Utility

Amazon S3 is a swiss army knife when it comes to cloud storage. There are simply a ton of ways you can use S3. To mention a few, data archiving, big data analytics, cloud storage, backup and recovery. One of the most common is the static hosting of websites. I want to show you how you can programmatically upload and delete an entire directory using the .NET apis for S3.

The short version

  1. Create a console application in visual studio
  2. Add the AWSSDK.S3 nuget package
  3. Create a class (S3TransferUtility) to manage uploading and deleting directories.
  4. Create a transfer request and call the UploadDirectory method using TransferUtility
  5. Use the File I/O APIs to delete the uploaded folder and files.

Uploading directories to S3

The AWSDK.S3 comes with a great utility called TransferUtility. Install via the following command in your console application

TransferUtility provides a simple API for uploading content to and downloading content from Amazon S3. It makes extensive use of Amazon S3 multipart uploads to achieve throughput, performance, and reliability. When uploading large files by specifying file paths instead of a stream, TransferUtility uses multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can increase throughput significantly.

To use the TransferUtility class simple initialize a new instance with your AWS access key and secret key.

TransferUtility transferUtility = new TransferUtility("[ACCESSKEY]", "[SECRETKEY]", RegionEndpoint.USWest2);

 

Uploading directories can be done by simply creating a new upload request and calling the method UploadDirectory. You can set the ACL permissions to PublicRead if you want the contents of your folder to be public.

 
/// <summary>
/// Upload specified Diretory to S3 bucket
/// </summary>
/// <param name="uploadDirectory"></param>
/// <param name="bucket"></param>
/// <returns></returns>
public bool SaveAsset(string uploadDirectory, string bucket)
{
    try
    {

        TransferUtilityUploadDirectoryRequest request = new TransferUtilityUploadDirectoryRequest
        {
            BucketName = bucket,
            Directory = uploadDirectory,
            SearchOption = System.IO.SearchOption.AllDirectories,
            CannedACL = S3CannedACL.PublicRead
        };
        _transferUtility.UploadDirectory(request);

        return true;
    }
    catch (Exception exception)
    {
        //Log Exception
        return false;
    }
}

Deleting a directory from S3

The S3 SDK also provides another set of APIs called File I/O. These APIs are useful for applications which want to treat S3 as a filesystem. It does this by mimicking the .NET base classes and FileInfo  DirectoryInfo with the new classes S3FileInfo and S3DirectoryInfo

 

/// <summary>
/// Delete Directory from S3
/// </summary>
/// <param name="uploadDirectory"></param>
/// <param name="bucket"></param>
/// <returns></returns>
public bool DeleteAsset(string bucket, string uploadDirectory)
{
    try
    {
        S3DirectoryInfo directoryToDelete = new S3DirectoryInfo(_client, bucket, uploadDirectory);

        var directoryFiles = directoryToDelete.EnumerateFiles();
        foreach (S3FileInfo file in directoryFiles)
        {
            S3FileInfo filetoDelete = new S3FileInfo(_client, bucket, file.FullName.Replace(bucket + ":\\", string.Empty));
            if (filetoDelete.Exists)
            {
                filetoDelete.Delete();
            }
        }


        if (directoryToDelete.Exists)
        {
            directoryToDelete.Delete(false);
            return true;
        }
        
    }
    catch (Exception exception)
    {
        //Log error

        return false;
    }
    return false;
}

Usage

 class Program
    {
        static void Main(string[] args)
        {
            var directoryToUpload = @"c:\\Dev\\site";
            var bucketName = "s3mediatransfers/transfers/site";

            //Upload Directory
            S3AssetTransferUtility transferUtility = new S3AssetTransferUtility();
            var uploadStatus = transferUtility.SaveAsset(directoryToUpload, bucketName);

            Console.WriteLine(string.Format("Upload to S3 Succeded : {0}", uploadStatus));

            //Delete Directory
            var deleteStatus = transferUtility.DeleteAsset("s3mediatransfers", "transfers\\site");
            Console.WriteLine(string.Format("Directory Deletion from S3 Succeded : {0}",deleteStatus));
        
        }
    }
}

Full code sample can be found here: https://github.com/samuelmensah/S3TransferUtility

references

  • https://aws.amazon.com/blogs/developer/the-three-different-apis-for-amazon-s3/
  • https://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/T_Amazon_S3_Transfer_TransferUtility.htm
  • https://www.nuget.org/packages/AWSSDK.S3/