Skyplane

Abstract

1. Introduction

Skyplane is a tool for blazingly fast bulk data transfers between object stores in the cloud. It provisions a fleet of VMs in the cloud to transfer data in parallel while using compression and bandwidth tiering to reduce cost.

Skyplane is an actively developed project. It will have SHARP EDGES . Please file an issue or ask the contributors via the #help channel on our Slack if you encounter bugs.

2. Objective

🔥 Blazing fast bulk data transfers between any cloud 🔥

This project demonstrates how Python, AWS can be applied to a real-world Blockchain & Cloud Computing problem.

4. Technology Stack

• Technical Talk
• NSDI '23 Paper

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/skyplane-project/skyplane.git
cd skyplane

$ pip install "skyplane[aws]"

# install support for other clouds as needed:

#   $ pip install "skyplane[azure]"
#   $ pip install "skyplane[gcp]"
#   $ pip install "skyplane[ibmcloud]"
#   $ pip install "skyplane[all]"

---> For AWS:
$ pip install awscli

---> For Google Cloud:
$ pip install gcloud

---> For Azure:
$ pip install azure

---> For AWS:
$ aws configure

---> For Google Cloud:
$ gcloud auth application-default login

---> For Azure:
$ az login

---> For IBM Cloud:
$ Follow IBM Cloud and create an account with the resource group.
Copy https://github.com/skyplane-project/skyplane/blob/main/skyplane/compute/ibmcloud/ibm_credentials.yaml.template
into `~/.bluemix/ibm_credentials` and fill your
IBM IAM key and credentials to your IBM Cloud object storage

---> For SCP:
$ # Create directory if required
$ mkdir -p ~/.scp
$ # Add the lines for "access_key", "secret_key", and "project_id" to scp_credential file
$ echo "access_key = <your_access_key>" >> ~/.scp/scp_credential
$ echo "secret_key = <your_secret_key>" >> ~/.scp/scp_credential
$ echo "project_id = <your_project_id>" >> ~/.scp/scp_credential

$ skyplane init

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Add a CI/CD pipeline with GitHub Actions
• Deploy to a public test network or cloud free tier
• Add monitoring and cost alerts

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. Why does this problem need a blockchain or cloud-native design?
2. Explain the smart contract / infrastructure components and how they interact.
3. How are gas costs or cloud costs kept under control?
4. How is the system secured (keys, IAM, access control)?
5. How would the solution scale to many more users?

9. Source Code & License

This project is developed by skyplane-project and published on GitHub under the Apache License 2.0. Please follow the license terms and credit the original author when you use or modify this code.