From Fedora Project Wiki
(fix typo) |
No edit summary |
||
| Line 8: | Line 8: | ||
* Need list of IP addresses for each region to populate MM. Would be nice if we could get that programmatically. | * Need list of IP addresses for each region to populate MM. Would be nice if we could get that programmatically. | ||
** https://forums.aws.amazon.com/ann.jspa?annID=1252 | ** https://forums.aws.amazon.com/ann.jspa?annID=1252 | ||
* Per FI meeting 20120216, suggest using secondary01 or a releng* box for the copying. secondary01 does not have /pub/epel or /pub/fedora currently mounted. | |||
* bucket names s3-mirror-<region>.fedoraproject.org allow for CNAME s3-mirror.fedoraproject.org to s3.amazon.com in our DNS | * bucket names s3-mirror-<region>.fedoraproject.org allow for CNAME s3-mirror.fedoraproject.org to s3.amazon.com in our DNS | ||
| Line 51: | Line 52: | ||
Open questions: | Open questions: | ||
* do we sync to one region, then COPY to others? If so, what tool? That'll cost $ for bandwidth. | * do we sync to one region, then COPY to others? If so, what tool? That'll cost $ for bandwidth. | ||
Proposed Excludes: | |||
<pre> | |||
source/ | |||
SRPMS/ | |||
debug/ | |||
beta/ | |||
ppc/ | |||
ppc64/ | |||
repoview/ | |||
Fedora/ | |||
Live/ | |||
isolinux/ | |||
images/ | |||
EFI/ | |||
drpms/ | |||
core/ | |||
extras/ | |||
LiveOS/ | |||
updates/8 | |||
updates/9 | |||
updates/10 | |||
updates/11 | |||
updates/12 | |||
updates/13 | |||
updates/14 | |||
updates/testing/8 | |||
updates/testing/9 | |||
updates/testing/10 | |||
updates/testing/11 | |||
updates/testing/12 | |||
updates/testing/13 | |||
updates/testing/14 | |||
releases/test/ | |||
</pre> | |||
Revision as of 17:22, 17 February 2012
Initial thoughts by Matt Domsch
- Use Reduced Redundancy Storage. All the content will be replicated easily.
- Use s3cmd sync to keep content in buckets in sync
- exclude ISOs
- exclude debuginfo? I think so.
- Use bucket policies to limit access to each region
- Need list of IP addresses for each region to populate MM. Would be nice if we could get that programmatically.
- Per FI meeting 20120216, suggest using secondary01 or a releng* box for the copying. secondary01 does not have /pub/epel or /pub/fedora currently mounted.
- bucket names s3-mirror-<region>.fedoraproject.org allow for CNAME s3-mirror.fedoraproject.org to s3.amazon.com in our DNS
| Region | Region Server | Bucket Name | CNAME |
| US Standard | s3-website-us-east-1.amazonaws.com | s3-mirror-us-east-1.fedoraproject.org | s3-mirror-us-east-1.fedoraproject.org CNAME s3-mirror-us-east-1.fedoraproject.org.s3-website-us-east-1.amazonaws.com |
| US West (Oregon) Region | s3-website-us-west-2.amazonaws.com | ||
| US West (Northern California) Region | s3-website-us-west-1.amazonaws.com | ||
| EU (Ireland) Region | s3-website-eu-west-1.amazonaws.com | ||
| Asia Pacific (Singapore) Region | s3-website-ap-southeast-1.amazonaws.com | ||
| Asia Pacific (Tokyo) Region | s3-website-ap-northeast-1.amazonaws.com | ||
| South America (Sao Paulo) Region | s3-website-sa-east-1.amazonaws.com |
Torrents:
- if we upload ISOs, we get .torrent links "for free".
- no tracker stats :-(
- Can't group multiple files together into a single torrent
- we're paying for outbound bandwidth
- bucket policies keeping traffic in a single region means we need separate buckets for torrent content
Costs:
- none for all uploads
- none for intra-region requests
- 0.093/GB/month for data, 200GB = $30-40/month/region. 7 Regions.
- no way guess number of GET requests. $40 assumes 10M requests, while $30/month assumes 1M requests.
Total: ~$280/month, or $3360/yr
Open questions:
- do we sync to one region, then COPY to others? If so, what tool? That'll cost $ for bandwidth.
Proposed Excludes:
source/ SRPMS/ debug/ beta/ ppc/ ppc64/ repoview/ Fedora/ Live/ isolinux/ images/ EFI/ drpms/ core/ extras/ LiveOS/ updates/8 updates/9 updates/10 updates/11 updates/12 updates/13 updates/14 updates/testing/8 updates/testing/9 updates/testing/10 updates/testing/11 updates/testing/12 updates/testing/13 updates/testing/14 releases/test/