A Scalable Large-File Transfer Service over HTTP Built on the CoDeeN content distribution network

News

* Video flash crowd break CoBlitz previous BW record (9/28/2007)
Aggregate CoBlitz transfer rate reached 3.3 Gbps!
("M" in the graph means "Gbps")

* Port 80 Demux is deployed. (6/15/2007)
Now you can drop ":3125" in the coblitz prefix, that is, you can use http://coblitz.codeen.org/ instead of http://coblitz.codeen.org:3125/.

* CoBlitz users mailing list is open!(2/13/2007)

* Fedora Core 6 is being distributed through CoBlitz!(10/24/2006)
The 15-minute average transmit bandwidth graph is shown below. CoBlitz achieved 1.44Gbps peak, and over 1Gbps sustained bandwidth during the first two days of the FC6 release.
("M" in the graph means "Gbps")

* Our CoBlitz paper was published at NSDI '06.

* Fedora Core 5 is being distributed through CoBlitz!(3/20/2006)
Here is CoMon's 15 minutes average of transmit bandwidth consumption of CoBlitz.
("k" in the graph means "Mbps", so CoBlitz uses 400 ~ 700Mbps aggregate bandwidth.)

* We are running a Fedora Core mirror using CoBlitz! (11/28/2005)

What
Is It?

CoBlitz provides a means to scalably serve large files over an HTTP content distribution network. It requires no modification of clients or servers, since all of the necessary support is located on the content distribution network itself. While it is built using the CoDeeN network running on PlanetLab, it does not require you to actively use CoDeeN or to join PlanetLab.

How Does
It Work?

You add the prefix http://coblitz.codeen.org/ to the URL you want to serve, and CoBlitz does the rest. The details can be found in our WORLDS'05 and NSDI'06 paper. CoBlitz uses the same underlying infrastructure as CoDeploy, but is being made available for public access. To give a high-level description of how it operates:

• When clients request a large file, they are really contacting a special agent that resides on the CDN node. This agent looks like a standard Web server.
• The agent converts the single request from the client into a stream of requests for smaller pieces (chunks) of the file. These requests are spread, in parallel, to other peer CDN nodes.
• These peers request the chunks from the origin server, using the byte-range support in HTTP. The peers not only send the request back to the original agent, but also cache their chunks.
• The agent reassembles the chunks and sends them back to the client in order, making it appear like one seamless download.

This approach has several benefits:

• As peers join/leave the CDN, only the missing parts of the large file need to be re-requested, instead of doing whole-file caching.
• Large files can be spread across the main memory of many nodes, reducing the memory pressure on any single node, and reducing the number of disk accesses needed to serve the file.
• Since we use HTTP as the underlying protocol, no changes are required to clients or servers. All CoBlitz support is on the CDN itself.

Access
Restrictions

CoBlitz currently has some usage restrictions while we are in beta testing. CoBlitz is not meant to be a general-purpose hosting service, and these restrictions limit it intentionally

• No porn or illegal software - do not use CoBlitz for pornography or software/music/video piracy. We reserve the right to make access logs public at any time or to share them at any time.
• File size limits - No files smaller than 100KB or larger than 20GB are served for the general public. Exceptions are provided for PlanetLab users and for US Educational sites.
• Content types - We are focusing on serving large files, like ISO images, PDFs, etc. We automatically change the content type of '.iso' files to be 'application/octet-stream'. We do not serve Web pages, images, videos, or audio files for the general public. These restrictions do not apply to files hosted at US Educational sites, or to any downloads initiated at PlanetLab-affiliated addresses.
• Exceptions - As noted above, the content type and size restrictions do not apply for content hosted at US Educational Sites or for content accessed by members of the PlanetLab community. In both cases, acceptable use policies are in place that provide mechanisms to handle any illegal transfers, so please do not abuse our policies

How to Use

You can easily make a CoBlitz link out of a regular link. The canonical form is

http://coblitz.codeen.org/Original_URL

Note that the original URL can either contain "http://" or not. (wget complains if you include "http://" in the orignal URL. So, when you use wget, please either use "\" in the second "http://" like "http://coblitz.codeen.org/http:\/\/original_url", or strip off the second "http://", like "http://coblitz.codeen.org/original_url".)

Example:
http://coblitz.codeen.org/www.cs.princeton.edu/~kyoungso/bigfile.zip

Warning:
wget(or similar tool) users should not include any option(or header) that disables caching. For example, specifying "--cache=off" will disable caching in CoBlitz and all the requests will go directly to the origin server, possibly causing DDoS attacks!

News(11/30/2005):
To work around this in our part, we have CoBlitz drop all Pragma headers at the time of receiving the request. But CoBlitz still supports the "max-age" directive in the Cache-Control header, so you can specify the maximum age of the contents. For example,

'Cache-Control: max-age=0' means you want CoBlitz to fetch the contents from the origin server. Please avoid it if not necessary.
'Cache-Control: max-age=300' means you are fine with the cached content as long as it is five minutes or less

Test
Files

CoBlitz serves as Fedora Core mirror for six different locations around the world. Please try out Fedora Core image downloading through CoBlitz.

CoBlitz Fedora Core mirror

Status

CoBlitz is currently in beta testing. While we strive to ensure availability of the service 24 hours a day, 7 days a week, we may encounter outages when the service is being upgraded. CoBlitz does involve a number of new components, and while we are confident enough in their development to allow open usage, we may still encounter bugs. We ask for your understanding, and equally importantly, your feedback about the service.

People

KyoungSoo Park
Vivek Pai

We may collectively be contacted at
princeton_codeen at slices.planet-lab.org