A Scalable Large-File Transfer Service over HTTP Built on the CoDeeN content distribution network

About

The CoBlitz system was a scalable content distribution network (CDN) on PlanetLab, which focused on transferring large files, such as video and software downloads. Its design was focused around breaking large files into smaller pieces, naming each piece with its own unique name, and then distributing these pieces among a set of cooperating Web proxies. This approach provided natural scalability, and unified the storage, processing power, and bandwidth of a set of cooperating machines.

At its peak, the CoBlitz CDN on PlanetLab ran on over 600 machines spread across over 200 PoPs, giving it a footprint larger than many commercial CDNs of its time. However, it was limited by the donated bandwidth of the PlanetLab network, which was typically less than 2-3 Gbps. The system used full-custom software written in C and C++, capable of handling tens of thousands of requests per second on modern hardware, but ran in only 100MB of memory per server on the PlanetLab system.

Status

The CoBlitz system that runs on PlanetLab is now gradually closing down, but may have intermittent periods of activity. It ran in mostly-continuous operation from 2004-2011 on the PlanetLab system.

CoBlitz was spun out of Princeton as a commercial entity, CoBlitz LLC, in 2007. Its first commercial customer was Polish Telecom, which uses it for a country-wide CDN. CoBlitz LLC was acquired by Verivue, Inc. in October 2010, where it formed the basis of the OneVantage CDN. That company was acquired by Akamai in November 2012, and CoBlitz will become part of the Aura Licensed CDN that will be used by network operators.

People

The CoBlitz system was developed at Princeton by KyoungSoo Park and Vivek Pai. At the time, KyoungSoo was a graduate student, and Vivek was his advisor. KyoungSoo is currently the Ewon Associate Professor of Electrical Engineering at KAIST, the leading technical university of Korea. Vivek returned to Princeton after a short leave after the acquisition of CoBlitz LLC.

Publications

These are the main publications that describe the CoBlitz system and its components.

Scale and Performance in the CoBlitz Large-File Distribution Service
KyoungSoo Park and Vivek S. Pai.
In Proceedings of the 3rd Symposium on Networked Systems Design and Implementation (NSDI '06).
San Jose, CA, May 2006

CoDNS: Improving DNS Performance and Reliability via Cooperative Lookups
KyoungSoo Park, Vivek S. Pai, Larry Peterson and Zhe Wang.
In Proceedings of the Sixth Symposium on Operating Systems Design and Implementation (OSDI '04).
San Francisco, CA, Dec 2004

CoMon: A Mostly-Scalable Monitoring System for PlanetLab
KyoungSoo Park and Vivek S. Pai
In Operating Systems Review, Vol 40, No 1, Jan 2006

Connection Conditioning: Architecture-Independent Support for Simple, Robust Servers
KyoungSoo Park and Vivek S. Pai.
In Proceedings of the 3rd Symposium on Networked Systems Design and Implementation (NSDI '06).
San Jose, CA, May 2006

CoDeploy: Deploying Large File Transfer on an HTTP Content Distribution Network
KyoungSoo Park and Vivek S. Pai.
In Proceedings of the First Workshop on Real, Large Distributed Systems (WORLDS '04).
San Francisco, CA, Dec 2004

(Re)Design Considerations for Scalable Large-File Content Distribution
Brian Biskeborn, Michael Golightly, KyoungSoo Park, and Vivek S. Pai.
In Proceedings of the Second Workshop on Real, Large Distributed Systems (WORLDS '05).
San Francisco, CA, Dec 2005

Old
News

* Video flash crowd break CoBlitz previous BW record (9/28/2007)
Aggregate CoBlitz transfer rate reached 3.3 Gbps!
("M" in the graph means "Gbps")

* Port 80 Demux is deployed. (6/15/2007)
Now you can drop ":3125" in the coblitz prefix, that is, you can use http://coblitz.codeen.org/ instead of http://coblitz.codeen.org:3125/.

* CoBlitz users mailing list is open!(2/13/2007)

* Fedora Core 6 is being distributed through CoBlitz!(10/24/2006)
The 15-minute average transmit bandwidth graph is shown below. CoBlitz achieved 1.44Gbps peak, and over 1Gbps sustained bandwidth during the first two days of the FC6 release.
("M" in the graph means "Gbps")

* Our CoBlitz paper was published at NSDI '06.

* Fedora Core 5 is being distributed through CoBlitz!(3/20/2006)
Here is CoMon's 15 minutes average of transmit bandwidth consumption of CoBlitz.
("k" in the graph means "Mbps", so CoBlitz uses 400 ~ 700Mbps aggregate bandwidth.)

* We are running a Fedora Core mirror using CoBlitz! (11/28/2005)

What
Is It?

CoBlitz provides a means to scalably serve large files over an HTTP content distribution network. It requires no modification of clients or servers, since all of the necessary support is located on the content distribution network itself. While it is built using the CoDeeN network running on PlanetLab, it does not require you to actively use CoDeeN or to join PlanetLab.

How Does
It Work?

You add the prefix http://coblitz.codeen.org/ to the URL you want to serve, and CoBlitz does the rest. The details can be found in our WORLDS'05 and NSDI'06 paper. CoBlitz uses the same underlying infrastructure as CoDeploy, but is being made available for public access. To give a high-level description of how it operates:

• When clients request a large file, they are really contacting a special agent that resides on the CDN node. This agent looks like a standard Web server.
• The agent converts the single request from the client into a stream of requests for smaller pieces (chunks) of the file. These requests are spread, in parallel, to other peer CDN nodes.
• These peers request the chunks from the origin server, using the byte-range support in HTTP. The peers not only send the request back to the original agent, but also cache their chunks.
• The agent reassembles the chunks and sends them back to the client in order, making it appear like one seamless download.

This approach has several benefits:

• As peers join/leave the CDN, only the missing parts of the large file need to be re-requested, instead of doing whole-file caching.
• Large files can be spread across the main memory of many nodes, reducing the memory pressure on any single node, and reducing the number of disk accesses needed to serve the file.
• Since we use HTTP as the underlying protocol, no changes are required to clients or servers. All CoBlitz support is on the CDN itself.

Access
Restrictions

CoBlitz currently has some usage restrictions while we are in beta testing. CoBlitz is not meant to be a general-purpose hosting service, and these restrictions limit it intentionally

• No porn or illegal software - do not use CoBlitz for pornography or software/music/video piracy. We reserve the right to make access logs public at any time or to share them at any time.
• File size limits - No files smaller than 100KB or larger than 20GB are served for the general public. Exceptions are provided for PlanetLab users and for US Educational sites.
• Content types - We are focusing on serving large files, like ISO images, PDFs, etc. We automatically change the content type of '.iso' files to be 'application/octet-stream'. We do not serve Web pages, images, videos, or audio files for the general public. These restrictions do not apply to files hosted at US Educational sites, or to any downloads initiated at PlanetLab-affiliated addresses.
• Exceptions - As noted above, the content type and size restrictions do not apply for content hosted at US Educational Sites or for content accessed by members of the PlanetLab community. In both cases, acceptable use policies are in place that provide mechanisms to handle any illegal transfers, so please do not abuse our policies

How to Use

You can easily make a CoBlitz link out of a regular link. The canonical form is

http://coblitz.codeen.org/Original_URL

Note that the original URL can either contain "http://" or not. (wget complains if you include "http://" in the orignal URL. So, when you use wget, please either use "\" in the second "http://" like "http://coblitz.codeen.org/http:\/\/original_url", or strip off the second "http://", like "http://coblitz.codeen.org/original_url".)

Example:
http://coblitz.codeen.org/www.cs.princeton.edu/~kyoungso/bigfile.zip

Warning:
wget(or similar tool) users should not include any option(or header) that disables caching. For example, specifying "--cache=off" will disable caching in CoBlitz and all the requests will go directly to the origin server, possibly causing DDoS attacks!

News(11/30/2005):
To work around this in our part, we have CoBlitz drop all Pragma headers at the time of receiving the request. But CoBlitz still supports the "max-age" directive in the Cache-Control header, so you can specify the maximum age of the contents. For example,

'Cache-Control: max-age=0' means you want CoBlitz to fetch the contents from the origin server. Please avoid it if not necessary.
'Cache-Control: max-age=300' means you are fine with the cached content as long as it is five minutes or less

Test
Files

CoBlitz serves as Fedora Core mirror for six different locations around the world. Please try out Fedora Core image downloading through CoBlitz.

CoBlitz Fedora Core mirror