So we herd u liek logz | Telecomix

Project Bluesmote

tl;dr - analyze & visualize Blue Coat censorship logs from Syria

VIDEO (5 MiN)

A short video. This would be a good place to start.

About these logs...

This data was recovered from public FTP servers in Syria over a period of six weeks in late 2011. The logs are from Blue Coat SG-9000 filtering proxies (aka "deep packet inspection") installed by Syrian ISPs and used to censor and surveil the Internet. Individual log lines are data about HTTP request/response from web browsers.

The logs were eventually leaked, with client IP redacted. The data is now public domain.

The data set is ~55GB in total compressed, and almost 1/2 TB uncompressed. That's a whole lotta lines. They provide unique, first hand evidence of what censorship looks like from the censor's perspective.

Data Sets

Mirrored from http://sec.tl/syrian-censorship-logs/ at 2011-12-12 06:44 Two versions of the same data. Blocks are easier to work with.

Blocks: are a series of tab-separated lines, with several columns. Each file is gzipped & about ~10MB. They are numbered like block-0000.log.gz up to 5274. The fields are described here. Unparseable lines removed, etc..

Raw logs: are the original, unaltered mirror of the leaked logs.

no there are no versions with the client IP intact, please stop asking.

WARNING

This mirror is hosted on Amazon Web Services. You can be certain that they keep logs too.

Download

PLEASE USE BITTORRENT THIS THING CO$T MONEY$

kthx. If you have never used bittorrent (wat), ╬╝Torrent is a simple client.

Amazon S3 & AWS

Bandwidth is free inside AWS s3://project-bluesmote/v1/blocks/

Blocks - HTTP

Individual block files, about 10 MB each. Handy to grab a random few for testing. If you want a larger amount, use these torrents.

Raw Logs - HTTP

Mirror of the original raw logs, as individual files. If you want the whole set, please use the torrent.

Torrents

Each block torrent is

  1. Blocks Torrent 0 - 5.5GB
  2. Blocks Torrent 1 - 5.5GB
  3. Blocks Torrent 2 - 5.5GB
  4. Blocks Torrent 3 - 5.5GB
  5. Blocks Torrent 4 - 5.5GB
  6. Blocks Torrent 5 - 5.5GB
  7. Blocks Torrent 6 - 5.5GB
  8. Blocks Torrent 7 - 5.5GB
  9. Blocks Torrent 8 - 5.5GB
  10. Blocks Torrent 9 - 5.5GB
  11. Raw Logs Torrent - 54GB

Code & Results

COMING SOON

Github: https://github.com/wearpants/bluesmote

Cluster

IRC: #bluesmote on irc.telecomix.org or use webchat. Tweet @telecomix with #bluesmote.

Kittehs!

We like cute cats.

The Telecomix Network | No copyright | Please copy us and make datalove