Mounting a Google Cloud Storage Bucket as a local disk
These days it's a pretty common usage pattern to store files of all sorts in a "storage as a service" offering such as Google Cloud Storage (GCS). It's simple enough to push and pull content to and from GCS but if, for example, you want to store your static website in GCS and serve it to the internet, what do you do? A simple solution might be to periodically sync the content of your GCS bucket to the local filesystem of your server(s) but that's got inherent lag due to the periodicity of the sync. Enter GCSFuse...
GCSFuse is an implementation of the Fuse filesystem which allows you to mount your GCS bucket on your server(s). That's pretty awesome as it means that functionally, you can treat your GCS bucket and the files (well, technically objects since GCS is an object store, not a filesystem) it contains as if they were on a physical or networked volume. Admittedly there's some lag in the transfer to and from GCS but for many read-only use cases (as a static website would be), that's only really an issue on the first access of a file as the file is cached by the operating system. You're currently reading content served in exactly this way.
If this sounds like a solution you might find useful, the good news is that it's incredibly simple to get started. I'm going to assume that your use case is like mine, i.e.:
- You only ever read files from your GCS bucket from the server(s) on which you'll mount your GCS bucket
- You have a relatively small number of small (i.e. KB not MB) objects you want to serve (few enough that the OS can cache them effectively
Follow the installation instructions for your OS. This essentially boils down to:
- Add the
yum installthe GCSFuse package
It really is that simple
All you need to do now is mount the volume in
read-only mode via your
fstab. We'll assume that:
- Your GCS bucket is called
- You want to mount the bucket at
- Your OS has a web server user whose user-id (
UID) and groupd-id (
MY-GCS-BUCKET with the name of your GCS bucket and you can run:
# Create the mount point in your local filesystem sudo mkdir /mnt/MY-GCS-BUCKET # Add an entry to your `fstab` to ensure the bucket is mounted even after reboot sudo echo "MY-GCS-BUCKET /mnt/MY-GCS-BUCKET gcsfuse ro,uid=33,gid=33,noatime,_netdev,noexec,user,implicit_dirs,allow_other 0 0" >> /etc/fstab # Mount the bucket sudo mount /mnt/MY-GCS-BUCKET
Explaining the mount options:
ro- mount the volume in
uid=33,gid=33- grant access to user-id and group-id 33 (usually the web serve user e.g.
noatime- don't update access times of files
_netdev- mark this as a network volume so the OS knows that the network needs to be available before it'll try to mount
noexec- don't allow execution of files in the mount
user- allow non-root users to mount the volume
implicit_dirs- treat object paths as if they were directories - this is critical and Fuse-specific
allow_other- allow non-root users to mount (Fuse-specific AFAIK)
You should now be able to configure your web server with a document root of the mount point (
/mnt/MY-GCS-BUCKET) and you're done!