Categories
General

Cloud storage cost calculation (backup/restore)

Related to my offsite-backup with restic project I recently wanted to choose an cloud-provider as backup target. But that seems not an easy decision. Besides the costs for storage and traffic things getting complicated when it comes to restore: Pricing differs here from $10 to $1300 for a single restore of 500GB!

Although I use mainly AWS for my business projects I consider to use google cloud storage for my private backup. Main reason for me is that the restore options for AWS S3 Glacier are not easy to predict and the costs could be very different.

Some notes on the calculation

General

  • To calculate the different tiers it is important to know the average archive size. Restic i.e. uses about 5MB chunks – therefore AWS Glacier with expedited access could be very, very expensive – on the other hand bulk access  might be cheap but could be very slow on restore. It would make sense with bigger archive sizes but not with backup tools like restic.
  • There might be some upload fees (number of requests) but they’re not considered here. Upload traffic is almost free at all providers

AWS S3

  • In the scope are AWS S3 with the tiers Standard, IA (infrequent access) and Glacier
  • Glacier has different download request times: Bulk (5-12 hours), Standard (3-5 hours), Expedited (1-5 Minutes)
  • IA has a minimum storage time of 30 days, Glacier has 90 days (normally no problem with backups)

Backblaze B2

  • almost the “cheapest” cloud storage provider, but as they’re based in the US the latency from/to Germany might be high (not tested).

Google Cloud Storage

  • In the scope are the tiers Regional, Nearline and Coldline
  • Nearline has a minimum storage time of 30 days, Cold-line has 90 days (normally no problem with backups)
  • Like AWS S3 Glacier, there are extra fees for requesting Nearline and Coldline data, but there are no “waiting times” on access!

Calculation

I’ve made a spreadsheet for the calculation. You can find an online version here (or embedded below). Or Download as Excel.

All information is supplied without guarantee and for my own private purpose. There might be additional fees that are not included here. Use at your own risk! Let me know if something might be wrong.

Update: Some months with running a weekly restic backup…

Monthly Google Invoice:

Restic cronjob output:

Files:         318 new,     3 changed, 242632 unmodified
Dirs:            0 new,     2 changed,     0 unmodified
Added to the repo: 1.333 GiB

processed 242953 files, 411.892 GiB in 21:29 snapshot abfef780 saved Applying Policy: keep the last 24 hourly, 7 daily, 4 weekly, 12 monthly, 3 yearly snapshots snapshots for (host [enterprise], paths [/data/data0/data]):

keep 29 snapshots:
ID        Time                 Host        Tags        Reasons           Paths
-----------------------------------------------------------------------------------------
1f95712e  2018-01-29 02:01:15  enterprise              monthly snapshot  /data/data0/data
8e12732d  2018-02-26 02:01:15  enterprise              monthly snapshot  /data/data0/data
fbd49a81  2018-03-26 02:01:14  enterprise              monthly snapshot  /data/data0/data
cfb98595  2018-04-23 02:01:14  enterprise              monthly snapshot  /data/data0/data
e54a1ef7  2018-05-28 02:01:12  enterprise              monthly snapshot  /data/data0/data
e01d934b  2018-06-11 02:01:14  enterprise              hourly snapshot   /data/data0/data
bbf7e884  2018-06-16 17:32:40  enterprise              hourly snapshot   /data/data0/data
38430fb6  2018-06-18 02:01:01  enterprise              hourly snapshot   /data/data0/data
a7923811  2018-06-25 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       monthly snapshot
47266d58  2018-07-02 02:01:01  enterprise              hourly snapshot   /data/data0/data
f7dee0c1  2018-07-09 02:01:02  enterprise              hourly snapshot   /data/data0/data
50eb4d73  2018-07-16 02:01:01  enterprise              hourly snapshot   /data/data0/data
b56838a2  2018-07-23 02:01:01  enterprise              hourly snapshot   /data/data0/data
69e6e24b  2018-07-30 02:01:02  enterprise              hourly snapshot   /data/data0/data
                                                       monthly snapshot
eb319f8e  2018-08-06 02:01:01  enterprise              hourly snapshot   /data/data0/data
4c617720  2018-08-13 02:01:01  enterprise              hourly snapshot   /data/data0/data
f2700244  2018-08-20 02:01:01  enterprise              hourly snapshot   /data/data0/data
c1831d23  2018-08-27 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       monthly snapshot
8dd873c6  2018-09-03 02:01:01  enterprise              hourly snapshot   /data/data0/data
e0590b89  2018-09-10 02:01:01  enterprise              hourly snapshot   /data/data0/data
c2c780d1  2018-09-17 02:01:01  enterprise              hourly snapshot   /data/data0/data
a3c2d6f4  2018-09-24 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       monthly snapshot
be9c14ad  2018-10-01 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
a9af7981  2018-10-08 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
074e76a4  2018-10-15 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
b56e1785  2018-10-22 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
                                                       weekly snapshot
de62c698  2018-10-29 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
                                                       weekly snapshot
                                                       monthly snapshot
8f6d26d5  2018-11-05 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
                                                       weekly snapshot
abfef780  2018-11-12 02:01:01  enterprise              hourly snapshot   /data/data0/data
                                                       daily snapshot
                                                       weekly snapshot
                                                       monthly snapshot
                                                       yearly snapshot
-----------------------------------------------------------------------------------------
29 snapshots

remove 1 snapshots:
ID        Time                 Host        Tags        Paths
-----------------------------------------------------------------------
6151d891  2018-06-04 02:01:32  enterprise              /data/data0/data
-----------------------------------------------------------------------
1 snapshots

1 snapshots have been removed, running prune counting files in repo building new index for repo [43:15] 100.00%  86295 / 86295 packs

repository contains 86295 packs (512873 blobs) with 410.911 GiB processed 512873 blobs: 0 duplicate blobs, 0B duplicate load all snapshots find data that is still in use for 29 snapshots [0:12] 100.00%  29 / 29 snapshots

found 512860 of 512873 data blobs still in use, removing 13 blobs will remove 0 invalid files will delete 1 packs and rewrite 2 packs, this frees 2.688 MiB [0:04] 100.00%  2 / 2 packs rewritten

counting files in repo
[33:14] 100.00%  86294 / 86294 packs

finding old index files
saved new indexes as [d36b4c0b bb72972f f6e803e2 e257d59d f2ae6af7 cbcf8f7a cd97a693 355bb778 5bf44fc7 87a01dc8 1b332542 e4dfb758 a592039d 1264aebb 169914b8 25b30161 8dc9eb92 51e7084e 22ce0c6a 7f5f498a c4169ede e98278a5 a7955efd 212c1e73 51f77666 b2682981 ada18575 244bb4ae 8ce840fd] remove 33 old index files [0:00] 100.00%  3 / 3 packs deleted

done

2 replies on “Cloud storage cost calculation (backup/restore)”

Thanks for the overview.
What is your experience on Class A/B operations caused by rsync / restic. Do you have data on how many files per GB would destroy the coldline pricing advantage, assuming a daily run of rsync / restic?

with my regualar (weekly) restic backup I can’t see a really increasing invoice – but to be honest currently there are not much purge operations. Will try to share details here later…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.