From 38fc10d3858be4a4161775a52e9bbb5b97c23d7e Mon Sep 17 00:00:00 2001 From: Dimitri Sokolyuk Date: Sat, 15 Dec 2018 14:43:12 +0100 Subject: rename docs --- doc/bittorrent-rfc.txt | 1456 ----------------------------------------------- doc/links.txt | 11 - doc/out.txt | 65 --- docs/bittorrent-rfc.txt | 1456 +++++++++++++++++++++++++++++++++++++++++++++++ docs/links.txt | 11 + docs/out.txt | 65 +++ 6 files changed, 1532 insertions(+), 1532 deletions(-) delete mode 100644 doc/bittorrent-rfc.txt delete mode 100644 doc/links.txt delete mode 100644 doc/out.txt create mode 100644 docs/bittorrent-rfc.txt create mode 100644 docs/links.txt create mode 100644 docs/out.txt diff --git a/doc/bittorrent-rfc.txt b/doc/bittorrent-rfc.txt deleted file mode 100644 index 38fc8ac..0000000 --- a/doc/bittorrent-rfc.txt +++ /dev/null @@ -1,1456 +0,0 @@ - - - -BitTorrent Protocol version 1.0 J. Fonseca -$Revision: 1.33 $ B. Reza - L. Fjeldsted - DIKU - April 2005 - - - BitTorrent Protocol -- BTP/1.0 - - -Abstract - - This document describes the BitTorrent Protocol version 1.0 referred - to as "BTP/1.0". The BitTorrent Protocol (BTP) is a protocol for - collaborative file distribution across the Internet and has been in - place on the Internet since 2002. It is best classified as a peer- - to-peer (P2P) protocol, although it also contains highly centralized - elements. BTP has already been implemented many times for different - platforms, and could well be said to be a mature protocol, although a - formal, detailed and complete description has so far been lacking. - - BTP was devised and implemented by Bram Cohen as a P2P replacement to - standard File Transfer Protocol (FTP) to be used in places where the - usage of an FTP implementation poses too much strain on the server in - terms of request-processing and sheer bandwidth. Normally a client - does not use her upload capacity while downloading a file. The BTP - approach capitalizes on this fact by having clients upload bits of - the data to each other. In comparison to FTP this adds huge - scalability and cost-management advantages. - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 1] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -Table of Contents - - 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 4 - 1.1 Extensions . . . . . . . . . . . . . . . . . . . . . . . . 4 - 1.2 Audience . . . . . . . . . . . . . . . . . . . . . . . . . 4 - 1.3 Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 - 1.4 Overall Operation . . . . . . . . . . . . . . . . . . . . 5 - 2. Bencoding . . . . . . . . . . . . . . . . . . . . . . . . 7 - 2.1 Scalar Types . . . . . . . . . . . . . . . . . . . . . . . 7 - 2.2 Compound Types . . . . . . . . . . . . . . . . . . . . . . 7 - 3. Pieces and Blocks . . . . . . . . . . . . . . . . . . . . 8 - 3.1 Pieces . . . . . . . . . . . . . . . . . . . . . . . . . . 8 - 3.2 Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . 8 - 4. The Metainfo File . . . . . . . . . . . . . . . . . . . . 10 - 4.1 The Structure of the Metainfo File . . . . . . . . . . . . 10 - 4.1.1 Single File Torrents . . . . . . . . . . . . . . . . . . . 10 - 4.1.2 Multi File Torrents . . . . . . . . . . . . . . . . . . . 11 - 5. The Tracker HTTP Protocol . . . . . . . . . . . . . . . . 13 - 5.1 Request . . . . . . . . . . . . . . . . . . . . . . . . . 13 - 5.2 Response . . . . . . . . . . . . . . . . . . . . . . . . . 14 - 6. The Peer Wire Protocol . . . . . . . . . . . . . . . . . . 16 - 6.1 Peer Wire Guidelines . . . . . . . . . . . . . . . . . . . 16 - 6.2 Handshaking . . . . . . . . . . . . . . . . . . . . . . . 16 - 6.3 Message Communication . . . . . . . . . . . . . . . . . . 18 - 6.3.1 Peer States . . . . . . . . . . . . . . . . . . . . . . . 18 - 6.3.2 Peer Wire Messages . . . . . . . . . . . . . . . . . . . . 19 - 6.3.3 Choke . . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.4 Unchoke . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.5 Interested . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.6 Uninterested . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.7 Have . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.8 Bitfield . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.9 Request . . . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.3.10 Piece . . . . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.3.11 Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.4 The End Game . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.5 Piece Selection Strategy . . . . . . . . . . . . . . . . . 22 - 6.6 Peer Selection Strategy . . . . . . . . . . . . . . . . . 22 - 7. Security Consideration . . . . . . . . . . . . . . . . . . 23 - 7.1 Tracker HTTP Protocol Issues . . . . . . . . . . . . . . . 23 - 7.2 Denial of Service Attacks on Trackers . . . . . . . . . . 23 - 7.3 Peer Identity Issues . . . . . . . . . . . . . . . . . . . 23 - 7.4 DNS Spoofing . . . . . . . . . . . . . . . . . . . . . . . 23 - 7.5 Issues with File and Directory Names . . . . . . . . . . . 24 - 7.6 Validating the Integrity of Data Exchanged Between Peers . 24 - 7.7 Transfer of Sensitive Information . . . . . . . . . . . . 25 - 8. IANA Considerations . . . . . . . . . . . . . . . . . . . 26 - 9. References . . . . . . . . . . . . . . . . . . . . . . . . 26 - - - -Fonseca, et al. [Page 2] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - Authors' Addresses . . . . . . . . . . . . . . . . . . . . 26 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 3] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -1. Introduction - - The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in RFC 2119 [3]. - - The File Transfer Protocol (FTP) [1] with the recent additions of - security extensions, still remains the standard for secure and - reliable transmission of large files over the Internet. However, its - highly centralized client-server approach also means, it is - inadequate for mass publication of files, where a single point may - expect to be requested by a critically large number of clients - simultaneously. To remedy this situation many organizations either - implement a cap on the number of simultaneous requests, or spread the - load on multiple mirror servers. Needless to say both approaches - have their drawbacks, and a solution that addresses these problems is - highly needed. - - The approach in BitTorrent Protocol (BTP) is to spread the load not - on mirror servers, but to the clients themselves by having them - upload bits of the file to each other while downloading it. Since - the clients usually do not utilize their upload capacity while - fetching a file, this approach does not put the clients in any - disadvantage. This has the added advantage that even small - organizations with limited resources can publish large files on the - Internet without having to invest in costly infrastructure. - -1.1 Extensions - - Since the introduction of BTP many modifications and extensions have - been proposed by individuals and community forums. To the extent - that these extensions have become part of what the BitTorrent - community considers best practice they have been included in this - document. However, many extensions have been omitted either because - they have been deemed to lack interoperability with existing - implementations, or because they are not regarded as being - sufficiently mature. - -1.2 Audience - - This document is aimed at developers who wish to implement BTP for a - particular platform. Also, system administrators and architects may - use this document to fully understand the implications of installing - an implementation of BTP. In particular, it is advised to study the - security implications in more detail, before installing an - implementation on a machine that also contains sensitive data. - Security implications are discussed in Section 7. - - - - -Fonseca, et al. [Page 4] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -1.3 Terminology - - Peer: A peer is a node in a network participating in file sharing. - It can simultaneously act both as a server and a client to other - nodes on the network. - - Neighboring peers: Peers to which a client has an active point to - point TCP connection. - - Client: A client is a user agent (UA) that acts as a peer on behalf - of a user. - - Torrent: A torrent is the term for the file (single-file torrent) or - group of files (multi-file torrent) the client is downloading. - - Swarm: A network of peers that actively operate on a given torrent. - - Seeder: A peer that has a complete copy of a torrent. - - Tracker: A tracker is a centralized server that holds information - about one or more torrents and associated swarms. It functions as - a gateway for peers into a swarm. - - Metainfo file: A text file that holds information about the torrent, - e.g. the URL of the tracker. It usually has the extension - .torrent. - - Peer ID: A 20-byte string that identifies the peer. How the peer ID - is obtained is outside the scope of this document, but a peer must - make sure that the peer ID it uses has a very high probability of - being unique in the swarm. - - Info hash: A SHA1 hash that uniquely identifies the torrent. It is - calculated from data in the metainfo file. - - -1.4 Overall Operation - - BTP consists of two logically distinct protocols, namely the Tracker - HTTP Protocol (THP), and the Peer Wire Protocol (PWP). THP defines a - method for contacting a tracker for the purposes of joining a swarm, - reporting progress etc. PWP defines a mechanism for communication - between peers, and is thus responsible for carrying out the actual - download and upload of the torrent. - - In order for a client to download a torrent the following steps must - be carried through: - - - - -Fonseca, et al. [Page 5] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 1. A metainfo file must be retrieved. - - 2. Instructions that will allow the client to contact other peers - must be periodically requested from the tracker using THP. - - 3. The torrent must be downloaded by connecting to peers in the - swarm and trading pieces using PWP. - - To publish a torrent the following steps must be taken: - - 1. A tracker must be set up. - - 2. A metainfo file pointing to the tracker and containing - information on the structure of the torrent must be produced and - published. - - 3. At least one seeder with access to the complete torrent must be - set up. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 6] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -2. Bencoding - - Bencoding encodes data in a platform independent way. In BTP/1.0 the - metainfo file and all responses from the tracker are encoded in the - bencoding format. The format specifies two scalar types (integers - and strings) and two compound types (lists and dictionaries). - - The Augmented BNF syntax [5] for bencoding format is: - - dictionary = "d" 1*(string anytype) "e" ; non-empty dictionary - list = "l" 1*anytype "e" ; non-empty list - integer = "i" signumber "e" - string = number ":" - anytype = dictionary / list / integer / string - signumber = "-" number / number - number = 1*DIGIT - CHAR = %x00-FF ; any 8-bit character - DIGIT = "0" / "1" / "2" / "3" / "4" / - "5" / "6" / "7" / "8" / "9" - -2.1 Scalar Types - - Integers are encoded by prefixing a string containing the base ten - representation of the integer with the letter "i" and postfixing it - with the letter "e". E.g. the integer 123 is encoded as i123e. - - Strings are encoded by prefixing the string content with the length - of the string followed by a colon. E.g. the string "announce" is - encoded as "8:announce". - -2.2 Compound Types - - The compound types provides a mean to structure elements of any - bencoding type. - - Lists are an arbitrary number of bencoded elements prefixed with the - letter "l" and postfixed with the letter "e". It follows that lists - can contain nested lists and dictionaries. For instance "li2e3:fooe" - defines a list containing the integer "2" and the string "foo". - - Dictionaries are an arbitrary number of key/value pairs delimited by - the letter "d" at the beginning and the letter "e" at the end. All - keys are bencoded strings while the associated value can be any - bencoded element. E.g. "d5:monthi4e4:name5:aprile" defines a - dictionary holding the associations: "month" => "4" and "name" => - "april". All dictionary keys MUST be sorted. - - - - - -Fonseca, et al. [Page 7] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -3. Pieces and Blocks - - This section describes how a torrent is organized in pieces and - blocks. The torrent is divided into one or more pieces. Each piece - represents a range of data which it is possible to verify using a - piece SHA1 hash. When distributing data over PWP pieces are divided - into one or more blocks, as shown in the following diagram: - - --------------------------------------- - | Piece #0 | Piece #1 | .. | Piece #N | - --------------------------------------- - _-' `-_ - _-' `-_ - ---------------------------- - | Block #0 | .. | Block #M | - ---------------------------- - -3.1 Pieces - - The number of pieces in the torrent is indicated in the metainfo - file. The size of each piece in the torrent remains fixed and can be - calculated using the following formula: - - fixed_piece_size = size_of_torrent / number_of_pieces - - where "/" is the integer division operator. Only the last piece of - the torrent is allowed to have fewer bytes than the fixed piece size. - - The size of a piece is determined by the publisher of the torrent. A - good recommendation is to use a piece size so that the metainfo file - does not exceed 70 kilobytes. - - For the sake of calculating the correct position of a piece within a - file, or files, the torrent is regarded as a single continuous byte - stream. In case the torrent consists of multiple files, it is to be - viewed as the concatenation of these files in the order of their - appearance in the metainfo file. Conceptually, the torrent is only - translated into files when all its pieces have been downloaded and - verified using their respective SHA1 values; although in practice an - implementation may choose a better approach in accordance with local - operating system and filesystem specific demands. - -3.2 Blocks - - The size of a block is an implementation defined value that is not - dependant on the fixed piece size. Once a fixed size is defined, the - number of blocks per piece can be calculated using the formula: - - - - -Fonseca, et al. [Page 8] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - number_of_blocks = (fixed_piece_size / fixed_block_size) - + !!(fixed_piece_size % fixed_block_size) - - where "%" denotes the modulus operator, and "!" the negation - operator. The negation operator is used to ensure that the last - factor only adds a value of 0 or 1 to the sum. Given the start - offset of the block its index within a piece can be calculated using - the formula: - - block_index = block_offset % fixed_block_size - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 9] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -4. The Metainfo File - - The metainfo file provides the client with information on the tracker - location as well as the torrent to be downloaded. Besides listing - which files will result from downloading the torrent, it also lists - how the client should split up and verify individual pieces making up - the complete torrent. - - In order for a client to recognize the metainfo file it SHOULD have - the extension .torrent and the associated the media type - "application/x-bittorrent". How the client retrieves the metainfo - file is beyond the scope of this document, however, the most user- - friendly approach is for a client to find the file on a web page, - click on it, and start the download immediately. This way, the - apparent complexity of BTP as opposed to FTP or HTTP transfer is - transparent to the user. - -4.1 The Structure of the Metainfo File - - The metainfo file contains a bencoded dictionary with the following - structure. A key below is REQUIRED unless otherwise noted. - - 'announce': This is a string value. It contains the announce URL of - the tracker. - - 'announce-list': This is an OPTIONAL list of string values. Each - value is a URL pointing to a backup tracker. This value is not - used in BTP/1.0. - - 'comment': This is an OPTIONAL string value that may contain any - comment by the author of the torrent. - - 'created by': This is an optional string value and may contain the - name and version of the program used to create the metainfo file. - - 'creation date': This is an OPTIONAL string value. It contains the - creation time of the torrent in standard Unix epoch format. - - 'info': This key points to a dictionary that contains information - about the files to download. The entries are explained in the - following sections. - - -4.1.1 Single File Torrents - - If the torrent only specifies one file, the info dictionary must have - the following keys: - - - - -Fonseca, et al. [Page 10] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'length': This is an integer value indicating the length of the file - in bytes. - - 'md5sum': This is an OPTIONAL value. If included it must be a string - of 32 characters corresponding to the MD5 sum of the file. This - value is not used in BTP/1.0. - - 'name': A string containing the name of the file. - - 'piece length': An integer indicating the number of bytes in each - piece. - - 'pieces': This is a string value containing the concatenation of the - 20-byte SHA1 hash value for all pieces in the torrent. For - example, the first 20 bytes of the string represent the SHA1 value - used to verify piece index 0. - - The complete file is derived by combining all the pieces into one - string of bytes. - -4.1.2 Multi File Torrents - - If the torrent specifies multiple files, the info dictionary must - have the following structure: - - 'files': This is a list of dictionaries. Each file in the torrent - has a dictionary associated to it having the following structure: - - 'length': This is an integer indicating the total length of the - file in bytes. - - 'md5sum': This is an OPTIONAL value. if included it must be a - string of 32 characters corresponding to the MD5 sum of the - file. This value is not used in BTP/1.0. - - 'path': This is a list of string elements that specify the path of - the file, relative to the topmost directory. The last element - in the list is the name of the file, and the elements preceding - it indicate the directory hierarchy in which this file is - situated. - - 'name': This is a string value. It contains the name of the top-most - directory in the file structure. - - 'piece length': This is an integer value. It contains the number of - bytes in each piece. - - - - - -Fonseca, et al. [Page 11] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'pieces': This is a string value. It must contain the concatenation - of all 20-byte SHA1 hash values that are used by BTP/1.0 to verify - each downloaded piece. The first 20 bytes of the string represent - the SHA1 value used to verify piece index 0. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 12] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -5. The Tracker HTTP Protocol - - The Tracker HTTP Protocol (THP) is a simple mechanism for introducing - peers to each other. A tracker is a HTTP service that must be - contacted by a peer in order to join a swarm. As such the tracker - constitutes the only centralized element in BTP/1.0. A tracker does - not by itself provide access to any downloadable data. A tracker - relies on peers sending regular requests. It may assume that a peer - is dead if it misses a request. - -5.1 Request - - To contact the tracker a peer MUST send a standard HTTP GET request - using the URL in the "announce" entry of the metainfo file. The GET - request must be parametrized as specified in the HTTP protocol. The - following parameters must be present in the request: - - 'info_hash': This is a REQUIRED 20-byte SHA1 hash value. In order to - obtain this value the peer must calculate the SHA1 of the value of - the "info" key in the metainfo file. - - 'peer_id': This is a REQUIRED string and must contain the 20-byte - self-designated ID of the peer. - - 'port': The port number that the peer is listening to for incoming - connections from other peers. BTP/1.0 does not specify a standard - port number, nor a port range to be used. This key is REQUIRED. - - 'uploaded': This is a base ten integer value. It denotes the total - amount of bytes that the peer has uploaded in the swarm since it - sent the "started" event to the tracker. This key is REQUIRED. - - 'downloaded': This is a base ten integer value. It denotes the total - amount of bytes that the peer has downloaded in the swarm since it - sent the "started" event to the tracker. This key is REQUIRED. - - 'left': This is a base ten integer value. It denotes the total - amount of bytes that the peer needs in this torrent in order to - complete its download. This key is REQUIRED. - - 'ip': This is an OPTIONAL value, and if present should indicate the - true, Internet-wide address of the peer, either in dotted quad - IPv4 format, hexadecimal IPv6 format, or a DNS name. - - 'numwant': This is an OPTIONAL value. If present, it should indicate - the number of peers that the local peer wants to receive from the - tracker. If not present, the tracker uses an implementation - defined value. - - - -Fonseca, et al. [Page 13] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'event': This parameter is OPTIONAL. If not specified, the request - is taken to be a regular periodic request. Otherwise, it MUST - have one of the three following values: - - 'started': The first HTTP GET request sent to the tracker MUST - have this value in the "event" parameter. - - 'stopped': This value SHOULD be sent to the tracker when the peer - is shutting down gracefully. - - 'completed': This value SHOULD be sent to the tracker when the - peer completes a download. The peer SHOULD NOT send this value - if it started up with the complete torrent. - - -5.2 Response - - Upon receiving the HTTP GET request, the tracker MUST respond with a - document having the "text/plain" MIME type. This document MUST - contain a bencoded dictionary with the following keys: - - 'failure reason': This key is OPTIONAL. If present, the dictionary - MUST NOT contain any other keys. The peer should interpret this - as if the attempt to join the torrent failed. The value is a - human readable string containing an error message with the failure - reason. - - 'interval': A peer must send regular HTTP GET requests to the tracker - to obtain an updated list of peers and update the tracker of its - status. The value of this key indicated the amount of time that a - peer should wait between to consecutive regular requests. This - key is REQUIRED. - - 'complete': This is an integer that indicates the number of seeders. - This key is OPTIONAL. - - 'incomplete': This is an integer that indicates the number of peers - downloading the torrent. This key is OPTIONAL. - - 'peers': This is a bencoded list of dictionaries containing a list of - peers that must be contacted in order to download a file. This - key is REQUIRED. It has the following structure: - - 'peer id': This is a REQUIRED string value containing the self- - designated ID of the peer. - - - - - - -Fonseca, et al. [Page 14] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'ip': This is a REQUIRED string value indicating the IP address of - the peer. This may be given as a dotted quad IPv4 format, - hexadecimal IPv6 format or DNS name. - - 'port': This is an integer value. It must contain the self- - designated port number of the peer. This key is REQUIRED. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 15] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -6. The Peer Wire Protocol - - The aim of the PWP, is to facilitate communication between - neighboring peers for the purpose of sharing file content. PWP - describes the steps taken by a peer after it has read in a metainfo - file and contacted a tracker to gather information about other peers - it may communicate with. PWP is layered on top of TCP and handles - all its communication using asynchronous messages. - -6.1 Peer Wire Guidelines - - PWP does not specify a standard algorithm for selecting elements from - a clients neighboring peers with whom to share pieces, although the - following guidelines are expected to be observed by any such - algorithm: - - The algorithm should not be constructed with the goal in mind to - reduce the amount of data uploaded compared to downloaded. At the - very least a peer should upload the same amount that it has - downloaded. - - The algorithm should not use a strict tit-for-tat schema when - dealing with remote peers that have just joined the swarm and thus - have no pieces to offer. - - The algorithm should make good use of both download and upload - bandwidth by putting a cap on the number of simultaneous - connection that actively send or receive data. By reducing the - number of active connections, TCP congestion can be avoided. - - The algorithm should pipeline data requests in order so saturate - active connections. - - The algorithm should be able to cooperate with peers that - implement a different algorithm. - - -6.2 Handshaking - - The local peer opens a port on which to listen for incoming - connections from remote peers. This port is then reported to the - tracker. As BTP/1.0 does not specify any standard port for listening - it is the sole responsibility of the implementation to select a port. - - Any remote peer wishing to communicate with the local peer must open - a TCP connection to this port and perform a handshake operation. The - handshake operation MUST be carried out before any other data is sent - from the remote peer. The local peer MUST NOT send any data back to - - - -Fonseca, et al. [Page 16] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - the remote peer before a well constructed handshake has been - recognized according to the rules below. If the handshake in any way - violates these rules the local peer MUST close the connection with - the remote peer. - - - - - - A handshake is a string of bytes with the following structure: - - ---------------------------------------------------------------- - | Name Length | Protocol Name | Reserved | Info Hash | Peer ID | - ---------------------------------------------------------------- - - Name Length: The unsigned value of the first byte indicates the - length of a character string containing the protocol name. In - BTP/1.0 this number is 19. The local peer knows its own protocol - name and hence also the length of it. If this length is different - than the value of this first byte, then the connection MUST be - dropped. - - Protocol Name: This is a character string which MUST contain the - exact name of the protocol in ASCII and have the same length as - given in the Name Length field. The protocol name is used to - identify to the local peer which version of BTP the remote peer - uses. In BTP/1.0 the name is 'BitTorrent protocol'. If this - string is different from the local peers own protocol name, then - the connection is to be dropped. - - Reserved: The next 8 bytes in the string are reserved for future - extensions and should be read without interpretation. - - Info Hash: The next 20 bytes in the string are to be interpreted as a - 20-byte SHA1 of the info key in the metainfo file. Presumably, - since both the local and the remote peer contacted the tracker as - a result of reading in the same .torrent file, the local peer will - recognize the info hash value and will be able to serve the remote - peer. If this is not the case, then the connection MUST be - dropped. This situation can arise if the local peer decides to no - longer serve the file in question for some reason. The info hash - may be used to enable the client to serve multiple torrents on the - same port. - - At this stage, if the connection has not been dropped, then the - local peer MUST send its own handshake back, which includes the - last step: - - - - -Fonseca, et al. [Page 17] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - Peer ID: The last 20 bytes of the handshake are to be interpreted as - the self-designated name of the peer. The local peer must use - this name to identify the connection hereafter. Thus, if this - name matches the local peers own ID name, the connection MUST be - dropped. Also, if any other peer has already identified itself to - the local peer using that same peer ID, the connection MUST be - dropped. - - In BTP/1.0 the handshake has a total of 68 bytes. - -6.3 Message Communication - - Following the PWP handshake both ends of the TCP channel may send - messages to each other in a completely asynchronous fashion. PWP - messages have the dual purpose of updating the state of neighboring - peers with regard to changes in the local peer, as well as - transferring data blocks between neighboring peers. - - PWP Messages fall into two different categories: - - State-oriented messages: These messages serve the sole purpose of - informing peers of changes in the state of neighboring peers. A - message of this type MUST be sent whenever a change occurs in a - peer's state, regardless of the state of other peers. The - following messages fall into this category: Interested, - Uninterested, Choked, Unchoked, Have and Bitfield. - - Data-oriented messages: These messages handle the requesting and - sending of data portions. The following messages fall into this - category: Request, Cancel and Piece. - - -6.3.1 Peer States - - For each end of a connection, a peer must maintain the following two - state flags: - - Choked: When true, this flag means that the choked peer is not - allowed to request data. - - Interested: When true, this flag means a peer is interested in - requesting data from another peer. This indicates that the peer - will start requesting blocks if it is unchoked. - - A choked peer MUST not send any data-oriented messages, but is free - to send any other message to the peer that has choked it. If a peer - chokes a remote peer, it MUST also discard any unanswered requests - for blocks previously received from the remote peer. - - - -Fonseca, et al. [Page 18] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - An unchoked peer is allowed to send data-oriented messages to the - remote peer. It is left to the implementation how many peers any - given peer may choose to choke or unchoke, and in what fashion. This - is done deliberately to allow peers to use different heuristics for - peer selection. - - An interested peer indicates to the remote peer that it must expect - to receive data-oriented messages as soon as it unchokes the - interested peer. It must be noted, that a peer must not assume a - remote peer is interested solely because it has pieces that the - remote peer is lacking. There may be valid reasons why a peer is not - interested in another peer other than data-based ones. - -6.3.2 Peer Wire Messages - - All integer members in PWP messages are encoded as a 4-byte big- - endian number. Furthermore, all index and offset members in PWP - messages are zero-based. - - A PWP message has the following structure: - - ----------------------------------------- - | Message Length | Message ID | Payload | - ----------------------------------------- - - Message Length: This is an integer which denotes the length of the - message, excluding the length part itself. If a message has no - payload, its size is 1. Messages of size 0 MAY be sent - periodically as keep-alive messages. Apart from the limit that - the four bytes impose on the message length, BTP does not specify - a maximum limit on this value. Thus an implementation MAY choose - to specify a different limit, and for instance disconnect a remote - peer that wishes to communicate using a message length that would - put too much strain on the local peer's resources. - - Message ID: This is a one byte value, indicating the type of the - message. BTP/1.0 specifies 9 different messages, as can be seen - further below. - - Payload: The payload is a variable length stream of bytes. - - If an incoming message in any way violates this structure then the - connection SHOULD be dropped. In particular the receiver SHOULD make - sure the message ID constitutes a valid message, and the payload - matches the the expected payload, as given below. - - For the purpose of compatibility with future protocol extensions the - client SHOULD ignore unknown messages. There may arise situations in - - - -Fonseca, et al. [Page 19] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - which a client may choose to drop a connection after receiving an - unknown message, either for security reasons, or because discarding - large unknown messages may be viewed as excessive waste. - - BTP/1.0 specifies the following messages: - -6.3.3 Choke - - This message has ID 0 and no payload. A peer sends this message to a - remote peer to inform the remote peer that it is being choked. - -6.3.4 Unchoke - - This message has ID 1 and no payload. A peer sends this message to a - remote peer to inform the remote peer that it is no longer being - choked. - -6.3.5 Interested - - This message has ID 2 and no payload. A peer sends this message to a - remote peer to inform the remote peer of its desire to request data. - -6.3.6 Uninterested - - This message has ID 3 and no payload. A peer sends this message to a - remote peer to inform it that it is not interested in any pieces from - the remote peer. - -6.3.7 Have - - This message has ID 4 and a payload of length 4. The payload is a - number denoting the index of a piece that the peer has successfully - downloaded and validated. A peer receiving this message must - validate the index and drop the connection if this index is not - within the expected bounds. Also, a peer receiving this message MUST - send an interested message to the sender if indeed it lacks the piece - announced. Further, it MAY also send a request for that piece. - -6.3.8 Bitfield - - This message has ID 5 and a variable payload length. The payload is - a bitfield representing the pieces that the sender has successfully - downloaded, with the high bit in the first byte corresponding to - piece index 0. If a bit is cleared it is to be interpreted as a - missing piece. A peer MUST send this message immediately after the - handshake operation, and MAY choose not to send it if it has no - pieces at all. This message MUST not be sent at any other time - during the communication. - - - -Fonseca, et al. [Page 20] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -6.3.9 Request - - This message has ID 6 and a payload of length 12. The payload is 3 - integer values indicating a block within a piece that the sender is - interested in downloading from the recipient. The recipient MUST - only send piece messages to a sender that has already requested it, - and only in accordance to the rules given above about the choke and - interested states. The payload has the following structure: - - --------------------------------------------- - | Piece Index | Block Offset | Block Length | - --------------------------------------------- - - -6.3.10 Piece - - This message has ID 7 and a variable length payload. The payload - holds 2 integers indicating from which piece and with what offset the - block data in the 3rd member is derived. Note, the data length is - implicit and can be calculated by subtracting 9 from the total - message length. The payload has the following structure: - - ------------------------------------------- - | Piece Index | Block Offset | Block Data | - ------------------------------------------- - - -6.3.11 Cancel - - This message has ID 8 and a payload of length 12. The payload is 3 - integer values indicating a block within a piece that the sender has - requested for, but is no longer interested in. The recipient MUST - erase the request information upon receiving this messages. The - payload has the following structure: - - --------------------------------------------- - | Piece Index | Block Offset | Block Length | - --------------------------------------------- - - -6.4 The End Game - - Towards the end of a download session, it may speed up the download - to send request messages for the remaining blocks to all the - neighboring peers. A client must issue cancel messages to all - pending requests sent to neighboring peers as soon as a piece is - downloaded successfully. This is referred to as the end game. - - - - -Fonseca, et al. [Page 21] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - A client usually sends requests for blocks in stages; sending - requests for newer blocks as replies for earlier requests are - received. The client enters the end game, when all remaining blocks - have been requested. - -6.5 Piece Selection Strategy - - BTP/1.0 does not force a particular order for selecting which pieces - to download. However, experience shows that downloading in rarest- - first order seems to lessen the wait time for pieces. To find the - rarest piece a client must calculate for each piece index the number - of times this index is true in the bitfield vectors of all the - neighboring peers. The piece with the lowest sum is then selected - for requesting. - -6.6 Peer Selection Strategy - - This section describes the choking algorithm recommended for - selecting neighboring peers with whom to exchange pieces. - Implementations are free to implement any strategy as long as the - guidelines in Section 6.1 are observed. - - After the initial handshake both ends of a connection set the Choked - flag to true and the Interested flag to false. - - All connections are periodically rated in terms of their ability to - provide the client with a better download rate. The rating may take - into account factors such as the remote peers willingness to maintain - an unchoked connection with the client over a certain period of time, - the remote peers upload rate to the client and other implementation - defined criteria. - - The peers are sorted according to their rating with regard to the - above mentioned scheme. Assume only 5 peers are allowed to download - at the same time. The peer selection algorithm will now unchoke as - many of the best rated peers as necessary so that exactly 5 of these - are interested. If one of the top rated peers at a later stage - becomes interested, then the peer selection algorithm will choke the - the worst unchoked peer. Notice that the worst unchoked peer is - always interested. - - The only lacking element from the above algorithm is the capability - to ensure that new peers can have a fair chance of downloading a - piece, even though they would evaluate poorly in the above schema. A - simple method is to make sure that a random peer is selected - periodically regardless of how it evaluates. Since this process is - repeated in a round robin manner, it ensures that ultimately even new - peers will have a chance of being unchoked. - - - -Fonseca, et al. [Page 22] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -7. Security Consideration - - This section examines security considerations for BTP/1.0.The - discussion does not include definitive solutions to the problems - revealed, though it does make some suggestions for reducing security - risks. - -7.1 Tracker HTTP Protocol Issues - - The use of the HTTP protocol for communication between the tracker - and the client makes BTP/1.0 vulnerable to the attacks mentioned in - the security consideration section of RFC 2616 [6]. - -7.2 Denial of Service Attacks on Trackers - - The nature of the tracker is to serve many clients. By mounting a - denial of service attack against the tracker the swarm attached to - the tracker can be starved. This type of attack is hard to defend - against, however, the metainfo file allows for multiple trackers to - be specified, making it possible to spread the load on a number of - trackers, and thus containing such an attack. - -7.3 Peer Identity Issues - - There is no strong authentication of clients when they contact the - tracker. The main option for trackers is to check peer ID and the IP - address of the client. The lack of authentication can be used to - mount an attack where a client can shut down another client if the - two clients are running on the same host and thus are sharing the - same IP address. In addition, a rogue peer may masquerade its - identity by using multiple peer IDs. Clients should there refrain - from taking the peer ID at face value. - -7.4 DNS Spoofing - - Clients using BTP/1.0 rely heavily on the Domain Name Service, which - can be used for both specifying the URI of the tracker and how to - contact a peer. Clients are thus generally prone to security attacks - based on the deliberate mis-association of IP addresses and DNS - names. Clients need to be cautious in assuming the continuing - validity of an IP address/DNS name association. - - In particular, BTP/1.0 clients SHOULD rely on their name resolver for - confirmation of an IP number/DNS name association, rather than - caching the result of previous host name lookups. If clients cache - the results of host name lookups in order to achieve a performance - improvement, they MUST observe the TTL information reported by DNS. - - - - -Fonseca, et al. [Page 23] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - If clients do not observe this rule, they could be spoofed when a - previously-accessed peers or trackers IP address changes. As network - renumbering is expected to become increasingly common according to - RFC 1900 [2], the possibility of this form of attack will grow. - Observing this requirement reduces this potential security - vulnerability. - -7.5 Issues with File and Directory Names - - The metainfo file provides a way to suggest a name of the downloaded - file for single-file torrents and the top-most directory for multi- - file torrents. This functionality is very like the Content- - Disposition header field documented in RFC 2183 [4] and the security - considerations mentioned in this RFC also apply to BitTorrent - clients. In short, BTP clients SHOULD verify that the suggested file - names in the metainfo file do not compromise services on the local - system. Furthermore, care must be taken for multi-file torrents to - validate that individual files are relative to the top-most directory - and that the paths do not contain path elements to the parent (that - is directory elevators such as ``..''), which can be used to place - files outside the top-most directory. - - Using UNIX as an example, some hazards would be: - - o Creating startup files (e.g., ".login"). - - o Creating or overwriting system files (e.g., "/etc/passwd"). - - o Overwriting any existing file. - - o Placing executable files into any command search path (e.g., - "~/bin/more"). - - o Sending the file to a pipe (e.g., "| sh"). - - It is very important to note that this is not an exhaustive list; it - is intended as a small set of examples only. Implementers must be - alert to the potential hazards on their target systems. In general, - the BTP client SHOULD NOT name or place files such that they will get - interpreted or executed without the user explicitly initiating the - action. - -7.6 Validating the Integrity of Data Exchanged Between Peers - - By default, all content served to the client from other peers should - be considered tainted and the client SHOULD validate the integrity of - the data before accepting it. The metainfo file contains information - for checking both individual pieces using SHA1, and optionally - - - -Fonseca, et al. [Page 24] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - individual files using MD5. SHA1, being the strongest of the two, is - preferred. Furthermore, sole reliance on whole-file checking can - potentially render otherwise valid pieces invalid, and should only be - considered for small files, to limit the amount of data being - discarded. - - Trusting the validity of the resulting file or files ends up being a - matter of trusting the content of the metainfo file. Ensuring the - validity of the metainfo file is beyond the scope of this document. - -7.7 Transfer of Sensitive Information - - Some clients include information about themselves when generating the - peer ID string. Clients should be aware that this information can - potentially be used to determine whether a specific client has a - exploitable security hole. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 25] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -8. IANA Considerations - - This document makes no request of IANA. - -9. References - - [1] Postel, J. and J. Reynolds, "File Transfer Protocol", STD 9, - RFC 959, October 1985. - - [2] Carpenter, B. and Y. Rekhter, "Renumbering Needs Work", - RFC 1900, February 1996. - - [3] Bradner, S., "Key words for use in RFCs to Indicate Requirement - Levels", BCP 14, RFC 2119, March 1997. - - [4] Troost, R., Dorner, S., and K. Moore, "Communicating - Presentation Information in Internet Messages: The Content- - Disposition Header Field", RFC 2183, August 1997. - - [5] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax - Specifications: ABNF", RFC 2234, November 1997. - - [6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., - Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- - HTTP/1.1", RFC 2616, June 1999. - - -Authors' Addresses - - Jonas Fonseca - DIKU - - Email: fonseca@diku.dk - - - Basim Reza - DIKU - - Email: basim@diku.dk - - - Lilja Fjeldsted - DIKU - - Email: lilja@diku.dk - - - - - - -Fonseca, et al. [Page 26] - diff --git a/doc/links.txt b/doc/links.txt deleted file mode 100644 index 55902ef..0000000 --- a/doc/links.txt +++ /dev/null @@ -1,11 +0,0 @@ -http://jonas.nitro.dk/bittorrent/bittorrent-rfc.html - -http://bittorrent.org/beps/bep_0003.html -http://bittorrent.org/beps/bep_0005.html - -https://wiki.theory.org/BitTorrentSpecification -http://wiki.vuze.com/w/Azureus_messaging_protocol - -Some random overview -http://www.kristenwidman.com/blog/33/how-to-write-a-bittorrent-client-part-1/ -http://www.kristenwidman.com/blog/71/how-to-write-a-bittorrent-client-part-2/ diff --git a/doc/out.txt b/doc/out.txt deleted file mode 100644 index 82bf6de..0000000 --- a/doc/out.txt +++ /dev/null @@ -1,65 +0,0 @@ -metainfo file.: OpenBSD_songs_ogg-2016-03-25-0127.torrent -info hash.....: 1aa5af9f6f533961a65169bdeeb801e611724d32 - -directory name: OpenBSD_songs_ogg -files.........: 38 - OpenBSD_songs_ogg/song30.ogg (2636244) 2.51 MB - OpenBSD_songs_ogg/song31.ogg (2381018) 2.27 MB - OpenBSD_songs_ogg/song32.ogg (2253880) 2.15 MB - OpenBSD_songs_ogg/song33.ogg (3238559) 3.09 MB - OpenBSD_songs_ogg/song34.ogg (5062603) 4.83 MB - OpenBSD_songs_ogg/song35.ogg (7191264) 6.86 MB - OpenBSD_songs_ogg/song36.ogg (5197630) 4.96 MB - OpenBSD_songs_ogg/song37.ogg (13937255) 13.29 MB - OpenBSD_songs_ogg/song38.ogg (5860681) 5.59 MB - OpenBSD_songs_ogg/song38b.ogg (5759105) 5.49 MB - OpenBSD_songs_ogg/song39.ogg (5954997) 5.68 MB - OpenBSD_songs_ogg/song40.ogg (3610740) 3.44 MB - OpenBSD_songs_ogg/song41.ogg (8382781) 7.99 MB - OpenBSD_songs_ogg/song42.ogg (6713047) 6.40 MB - OpenBSD_songs_ogg/song43.ogg (6834505) 6.52 MB - OpenBSD_songs_ogg/song44.ogg (4628329) 4.41 MB - OpenBSD_songs_ogg/song45.ogg (4737475) 4.52 MB - OpenBSD_songs_ogg/song46.ogg (3766867) 3.59 MB - OpenBSD_songs_ogg/song47.ogg (6586251) 6.28 MB - OpenBSD_songs_ogg/song48.ogg (3143847) 3.00 MB - OpenBSD_songs_ogg/song49.ogg (6006834) 5.73 MB - OpenBSD_songs_ogg/song50.ogg (4184951) 3.99 MB - OpenBSD_songs_ogg/song51.ogg (4216946) 4.02 MB - OpenBSD_songs_ogg/song52.ogg (4283099) 4.08 MB - OpenBSD_songs_ogg/song53.ogg (4635216) 4.42 MB - OpenBSD_songs_ogg/song54.ogg (3195265) 3.05 MB - OpenBSD_songs_ogg/song55.ogg (6160151) 5.87 MB - OpenBSD_songs_ogg/song56.ogg (5471902) 5.22 MB - OpenBSD_songs_ogg/song57.ogg (4057801) 3.87 MB - OpenBSD_songs_ogg/song58a.ogg (3252520) 3.10 MB - OpenBSD_songs_ogg/song58b.ogg (4376163) 4.17 MB - OpenBSD_songs_ogg/song58c.ogg (3595317) 3.43 MB - OpenBSD_songs_ogg/song58d.ogg (7013091) 6.69 MB - OpenBSD_songs_ogg/song59a.ogg (5766764) 5.50 MB - OpenBSD_songs_ogg/song59b.ogg (5317415) 5.07 MB - OpenBSD_songs_ogg/songsh.ogg (4890381) 4.66 MB - OpenBSD_songs_ogg/songsi.ogg (5694670) 5.43 MB - OpenBSD_songs_ogg/songty.ogg (6045645) 5.77 MB -archive size..: 196041209 (747 * 262144 + 219641) 186.96 MB - -announce url..: http://OpenBSD.somedomain.net:6969/announce -creation date.: Fri Mar 25 02:27:56 2016 -created by....: mktorrent 1.0 -comment.......: ogg files from OpenBSD/songs -Created by andrew fresh (andrew@afresh1.com) -http://OpenBSD.somedomain.net/ - - -metainfo file.: OpenBSD_5.9_amd64_install59.iso-2016-03-29-0449.torrent -info hash.....: e840038dea1998c39614dcd28594501df02bd32d - -file name.....: OpenBSD_5.9_amd64_install59.iso -file size.....: 233662464 (891 * 262144 + 92160) 222.84 MB - -announce url..: http://OpenBSD.somedomain.net:6969/announce -creation date.: Tue Mar 29 06:49:18 2016 -created by....: mktorrent 1.0 -comment.......: OpenBSD/5.9/amd64/install59.iso -Created by andrew fresh (andrew@afresh1.com) -http://OpenBSD.somedomain.net/ diff --git a/docs/bittorrent-rfc.txt b/docs/bittorrent-rfc.txt new file mode 100644 index 0000000..38fc8ac --- /dev/null +++ b/docs/bittorrent-rfc.txt @@ -0,0 +1,1456 @@ + + + +BitTorrent Protocol version 1.0 J. Fonseca +$Revision: 1.33 $ B. Reza + L. Fjeldsted + DIKU + April 2005 + + + BitTorrent Protocol -- BTP/1.0 + + +Abstract + + This document describes the BitTorrent Protocol version 1.0 referred + to as "BTP/1.0". The BitTorrent Protocol (BTP) is a protocol for + collaborative file distribution across the Internet and has been in + place on the Internet since 2002. It is best classified as a peer- + to-peer (P2P) protocol, although it also contains highly centralized + elements. BTP has already been implemented many times for different + platforms, and could well be said to be a mature protocol, although a + formal, detailed and complete description has so far been lacking. + + BTP was devised and implemented by Bram Cohen as a P2P replacement to + standard File Transfer Protocol (FTP) to be used in places where the + usage of an FTP implementation poses too much strain on the server in + terms of request-processing and sheer bandwidth. Normally a client + does not use her upload capacity while downloading a file. The BTP + approach capitalizes on this fact by having clients upload bits of + the data to each other. In comparison to FTP this adds huge + scalability and cost-management advantages. + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 1] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 4 + 1.1 Extensions . . . . . . . . . . . . . . . . . . . . . . . . 4 + 1.2 Audience . . . . . . . . . . . . . . . . . . . . . . . . . 4 + 1.3 Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 + 1.4 Overall Operation . . . . . . . . . . . . . . . . . . . . 5 + 2. Bencoding . . . . . . . . . . . . . . . . . . . . . . . . 7 + 2.1 Scalar Types . . . . . . . . . . . . . . . . . . . . . . . 7 + 2.2 Compound Types . . . . . . . . . . . . . . . . . . . . . . 7 + 3. Pieces and Blocks . . . . . . . . . . . . . . . . . . . . 8 + 3.1 Pieces . . . . . . . . . . . . . . . . . . . . . . . . . . 8 + 3.2 Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . 8 + 4. The Metainfo File . . . . . . . . . . . . . . . . . . . . 10 + 4.1 The Structure of the Metainfo File . . . . . . . . . . . . 10 + 4.1.1 Single File Torrents . . . . . . . . . . . . . . . . . . . 10 + 4.1.2 Multi File Torrents . . . . . . . . . . . . . . . . . . . 11 + 5. The Tracker HTTP Protocol . . . . . . . . . . . . . . . . 13 + 5.1 Request . . . . . . . . . . . . . . . . . . . . . . . . . 13 + 5.2 Response . . . . . . . . . . . . . . . . . . . . . . . . . 14 + 6. The Peer Wire Protocol . . . . . . . . . . . . . . . . . . 16 + 6.1 Peer Wire Guidelines . . . . . . . . . . . . . . . . . . . 16 + 6.2 Handshaking . . . . . . . . . . . . . . . . . . . . . . . 16 + 6.3 Message Communication . . . . . . . . . . . . . . . . . . 18 + 6.3.1 Peer States . . . . . . . . . . . . . . . . . . . . . . . 18 + 6.3.2 Peer Wire Messages . . . . . . . . . . . . . . . . . . . . 19 + 6.3.3 Choke . . . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6.3.4 Unchoke . . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6.3.5 Interested . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6.3.6 Uninterested . . . . . . . . . . . . . . . . . . . . . . . 20 + 6.3.7 Have . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6.3.8 Bitfield . . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6.3.9 Request . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 6.3.10 Piece . . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 6.3.11 Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 6.4 The End Game . . . . . . . . . . . . . . . . . . . . . . . 21 + 6.5 Piece Selection Strategy . . . . . . . . . . . . . . . . . 22 + 6.6 Peer Selection Strategy . . . . . . . . . . . . . . . . . 22 + 7. Security Consideration . . . . . . . . . . . . . . . . . . 23 + 7.1 Tracker HTTP Protocol Issues . . . . . . . . . . . . . . . 23 + 7.2 Denial of Service Attacks on Trackers . . . . . . . . . . 23 + 7.3 Peer Identity Issues . . . . . . . . . . . . . . . . . . . 23 + 7.4 DNS Spoofing . . . . . . . . . . . . . . . . . . . . . . . 23 + 7.5 Issues with File and Directory Names . . . . . . . . . . . 24 + 7.6 Validating the Integrity of Data Exchanged Between Peers . 24 + 7.7 Transfer of Sensitive Information . . . . . . . . . . . . 25 + 8. IANA Considerations . . . . . . . . . . . . . . . . . . . 26 + 9. References . . . . . . . . . . . . . . . . . . . . . . . . 26 + + + +Fonseca, et al. [Page 2] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + Authors' Addresses . . . . . . . . . . . . . . . . . . . . 26 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 3] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +1. Introduction + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119 [3]. + + The File Transfer Protocol (FTP) [1] with the recent additions of + security extensions, still remains the standard for secure and + reliable transmission of large files over the Internet. However, its + highly centralized client-server approach also means, it is + inadequate for mass publication of files, where a single point may + expect to be requested by a critically large number of clients + simultaneously. To remedy this situation many organizations either + implement a cap on the number of simultaneous requests, or spread the + load on multiple mirror servers. Needless to say both approaches + have their drawbacks, and a solution that addresses these problems is + highly needed. + + The approach in BitTorrent Protocol (BTP) is to spread the load not + on mirror servers, but to the clients themselves by having them + upload bits of the file to each other while downloading it. Since + the clients usually do not utilize their upload capacity while + fetching a file, this approach does not put the clients in any + disadvantage. This has the added advantage that even small + organizations with limited resources can publish large files on the + Internet without having to invest in costly infrastructure. + +1.1 Extensions + + Since the introduction of BTP many modifications and extensions have + been proposed by individuals and community forums. To the extent + that these extensions have become part of what the BitTorrent + community considers best practice they have been included in this + document. However, many extensions have been omitted either because + they have been deemed to lack interoperability with existing + implementations, or because they are not regarded as being + sufficiently mature. + +1.2 Audience + + This document is aimed at developers who wish to implement BTP for a + particular platform. Also, system administrators and architects may + use this document to fully understand the implications of installing + an implementation of BTP. In particular, it is advised to study the + security implications in more detail, before installing an + implementation on a machine that also contains sensitive data. + Security implications are discussed in Section 7. + + + + +Fonseca, et al. [Page 4] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +1.3 Terminology + + Peer: A peer is a node in a network participating in file sharing. + It can simultaneously act both as a server and a client to other + nodes on the network. + + Neighboring peers: Peers to which a client has an active point to + point TCP connection. + + Client: A client is a user agent (UA) that acts as a peer on behalf + of a user. + + Torrent: A torrent is the term for the file (single-file torrent) or + group of files (multi-file torrent) the client is downloading. + + Swarm: A network of peers that actively operate on a given torrent. + + Seeder: A peer that has a complete copy of a torrent. + + Tracker: A tracker is a centralized server that holds information + about one or more torrents and associated swarms. It functions as + a gateway for peers into a swarm. + + Metainfo file: A text file that holds information about the torrent, + e.g. the URL of the tracker. It usually has the extension + .torrent. + + Peer ID: A 20-byte string that identifies the peer. How the peer ID + is obtained is outside the scope of this document, but a peer must + make sure that the peer ID it uses has a very high probability of + being unique in the swarm. + + Info hash: A SHA1 hash that uniquely identifies the torrent. It is + calculated from data in the metainfo file. + + +1.4 Overall Operation + + BTP consists of two logically distinct protocols, namely the Tracker + HTTP Protocol (THP), and the Peer Wire Protocol (PWP). THP defines a + method for contacting a tracker for the purposes of joining a swarm, + reporting progress etc. PWP defines a mechanism for communication + between peers, and is thus responsible for carrying out the actual + download and upload of the torrent. + + In order for a client to download a torrent the following steps must + be carried through: + + + + +Fonseca, et al. [Page 5] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + 1. A metainfo file must be retrieved. + + 2. Instructions that will allow the client to contact other peers + must be periodically requested from the tracker using THP. + + 3. The torrent must be downloaded by connecting to peers in the + swarm and trading pieces using PWP. + + To publish a torrent the following steps must be taken: + + 1. A tracker must be set up. + + 2. A metainfo file pointing to the tracker and containing + information on the structure of the torrent must be produced and + published. + + 3. At least one seeder with access to the complete torrent must be + set up. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 6] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +2. Bencoding + + Bencoding encodes data in a platform independent way. In BTP/1.0 the + metainfo file and all responses from the tracker are encoded in the + bencoding format. The format specifies two scalar types (integers + and strings) and two compound types (lists and dictionaries). + + The Augmented BNF syntax [5] for bencoding format is: + + dictionary = "d" 1*(string anytype) "e" ; non-empty dictionary + list = "l" 1*anytype "e" ; non-empty list + integer = "i" signumber "e" + string = number ":" + anytype = dictionary / list / integer / string + signumber = "-" number / number + number = 1*DIGIT + CHAR = %x00-FF ; any 8-bit character + DIGIT = "0" / "1" / "2" / "3" / "4" / + "5" / "6" / "7" / "8" / "9" + +2.1 Scalar Types + + Integers are encoded by prefixing a string containing the base ten + representation of the integer with the letter "i" and postfixing it + with the letter "e". E.g. the integer 123 is encoded as i123e. + + Strings are encoded by prefixing the string content with the length + of the string followed by a colon. E.g. the string "announce" is + encoded as "8:announce". + +2.2 Compound Types + + The compound types provides a mean to structure elements of any + bencoding type. + + Lists are an arbitrary number of bencoded elements prefixed with the + letter "l" and postfixed with the letter "e". It follows that lists + can contain nested lists and dictionaries. For instance "li2e3:fooe" + defines a list containing the integer "2" and the string "foo". + + Dictionaries are an arbitrary number of key/value pairs delimited by + the letter "d" at the beginning and the letter "e" at the end. All + keys are bencoded strings while the associated value can be any + bencoded element. E.g. "d5:monthi4e4:name5:aprile" defines a + dictionary holding the associations: "month" => "4" and "name" => + "april". All dictionary keys MUST be sorted. + + + + + +Fonseca, et al. [Page 7] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +3. Pieces and Blocks + + This section describes how a torrent is organized in pieces and + blocks. The torrent is divided into one or more pieces. Each piece + represents a range of data which it is possible to verify using a + piece SHA1 hash. When distributing data over PWP pieces are divided + into one or more blocks, as shown in the following diagram: + + --------------------------------------- + | Piece #0 | Piece #1 | .. | Piece #N | + --------------------------------------- + _-' `-_ + _-' `-_ + ---------------------------- + | Block #0 | .. | Block #M | + ---------------------------- + +3.1 Pieces + + The number of pieces in the torrent is indicated in the metainfo + file. The size of each piece in the torrent remains fixed and can be + calculated using the following formula: + + fixed_piece_size = size_of_torrent / number_of_pieces + + where "/" is the integer division operator. Only the last piece of + the torrent is allowed to have fewer bytes than the fixed piece size. + + The size of a piece is determined by the publisher of the torrent. A + good recommendation is to use a piece size so that the metainfo file + does not exceed 70 kilobytes. + + For the sake of calculating the correct position of a piece within a + file, or files, the torrent is regarded as a single continuous byte + stream. In case the torrent consists of multiple files, it is to be + viewed as the concatenation of these files in the order of their + appearance in the metainfo file. Conceptually, the torrent is only + translated into files when all its pieces have been downloaded and + verified using their respective SHA1 values; although in practice an + implementation may choose a better approach in accordance with local + operating system and filesystem specific demands. + +3.2 Blocks + + The size of a block is an implementation defined value that is not + dependant on the fixed piece size. Once a fixed size is defined, the + number of blocks per piece can be calculated using the formula: + + + + +Fonseca, et al. [Page 8] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + number_of_blocks = (fixed_piece_size / fixed_block_size) + + !!(fixed_piece_size % fixed_block_size) + + where "%" denotes the modulus operator, and "!" the negation + operator. The negation operator is used to ensure that the last + factor only adds a value of 0 or 1 to the sum. Given the start + offset of the block its index within a piece can be calculated using + the formula: + + block_index = block_offset % fixed_block_size + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 9] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +4. The Metainfo File + + The metainfo file provides the client with information on the tracker + location as well as the torrent to be downloaded. Besides listing + which files will result from downloading the torrent, it also lists + how the client should split up and verify individual pieces making up + the complete torrent. + + In order for a client to recognize the metainfo file it SHOULD have + the extension .torrent and the associated the media type + "application/x-bittorrent". How the client retrieves the metainfo + file is beyond the scope of this document, however, the most user- + friendly approach is for a client to find the file on a web page, + click on it, and start the download immediately. This way, the + apparent complexity of BTP as opposed to FTP or HTTP transfer is + transparent to the user. + +4.1 The Structure of the Metainfo File + + The metainfo file contains a bencoded dictionary with the following + structure. A key below is REQUIRED unless otherwise noted. + + 'announce': This is a string value. It contains the announce URL of + the tracker. + + 'announce-list': This is an OPTIONAL list of string values. Each + value is a URL pointing to a backup tracker. This value is not + used in BTP/1.0. + + 'comment': This is an OPTIONAL string value that may contain any + comment by the author of the torrent. + + 'created by': This is an optional string value and may contain the + name and version of the program used to create the metainfo file. + + 'creation date': This is an OPTIONAL string value. It contains the + creation time of the torrent in standard Unix epoch format. + + 'info': This key points to a dictionary that contains information + about the files to download. The entries are explained in the + following sections. + + +4.1.1 Single File Torrents + + If the torrent only specifies one file, the info dictionary must have + the following keys: + + + + +Fonseca, et al. [Page 10] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + 'length': This is an integer value indicating the length of the file + in bytes. + + 'md5sum': This is an OPTIONAL value. If included it must be a string + of 32 characters corresponding to the MD5 sum of the file. This + value is not used in BTP/1.0. + + 'name': A string containing the name of the file. + + 'piece length': An integer indicating the number of bytes in each + piece. + + 'pieces': This is a string value containing the concatenation of the + 20-byte SHA1 hash value for all pieces in the torrent. For + example, the first 20 bytes of the string represent the SHA1 value + used to verify piece index 0. + + The complete file is derived by combining all the pieces into one + string of bytes. + +4.1.2 Multi File Torrents + + If the torrent specifies multiple files, the info dictionary must + have the following structure: + + 'files': This is a list of dictionaries. Each file in the torrent + has a dictionary associated to it having the following structure: + + 'length': This is an integer indicating the total length of the + file in bytes. + + 'md5sum': This is an OPTIONAL value. if included it must be a + string of 32 characters corresponding to the MD5 sum of the + file. This value is not used in BTP/1.0. + + 'path': This is a list of string elements that specify the path of + the file, relative to the topmost directory. The last element + in the list is the name of the file, and the elements preceding + it indicate the directory hierarchy in which this file is + situated. + + 'name': This is a string value. It contains the name of the top-most + directory in the file structure. + + 'piece length': This is an integer value. It contains the number of + bytes in each piece. + + + + + +Fonseca, et al. [Page 11] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + 'pieces': This is a string value. It must contain the concatenation + of all 20-byte SHA1 hash values that are used by BTP/1.0 to verify + each downloaded piece. The first 20 bytes of the string represent + the SHA1 value used to verify piece index 0. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 12] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +5. The Tracker HTTP Protocol + + The Tracker HTTP Protocol (THP) is a simple mechanism for introducing + peers to each other. A tracker is a HTTP service that must be + contacted by a peer in order to join a swarm. As such the tracker + constitutes the only centralized element in BTP/1.0. A tracker does + not by itself provide access to any downloadable data. A tracker + relies on peers sending regular requests. It may assume that a peer + is dead if it misses a request. + +5.1 Request + + To contact the tracker a peer MUST send a standard HTTP GET request + using the URL in the "announce" entry of the metainfo file. The GET + request must be parametrized as specified in the HTTP protocol. The + following parameters must be present in the request: + + 'info_hash': This is a REQUIRED 20-byte SHA1 hash value. In order to + obtain this value the peer must calculate the SHA1 of the value of + the "info" key in the metainfo file. + + 'peer_id': This is a REQUIRED string and must contain the 20-byte + self-designated ID of the peer. + + 'port': The port number that the peer is listening to for incoming + connections from other peers. BTP/1.0 does not specify a standard + port number, nor a port range to be used. This key is REQUIRED. + + 'uploaded': This is a base ten integer value. It denotes the total + amount of bytes that the peer has uploaded in the swarm since it + sent the "started" event to the tracker. This key is REQUIRED. + + 'downloaded': This is a base ten integer value. It denotes the total + amount of bytes that the peer has downloaded in the swarm since it + sent the "started" event to the tracker. This key is REQUIRED. + + 'left': This is a base ten integer value. It denotes the total + amount of bytes that the peer needs in this torrent in order to + complete its download. This key is REQUIRED. + + 'ip': This is an OPTIONAL value, and if present should indicate the + true, Internet-wide address of the peer, either in dotted quad + IPv4 format, hexadecimal IPv6 format, or a DNS name. + + 'numwant': This is an OPTIONAL value. If present, it should indicate + the number of peers that the local peer wants to receive from the + tracker. If not present, the tracker uses an implementation + defined value. + + + +Fonseca, et al. [Page 13] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + 'event': This parameter is OPTIONAL. If not specified, the request + is taken to be a regular periodic request. Otherwise, it MUST + have one of the three following values: + + 'started': The first HTTP GET request sent to the tracker MUST + have this value in the "event" parameter. + + 'stopped': This value SHOULD be sent to the tracker when the peer + is shutting down gracefully. + + 'completed': This value SHOULD be sent to the tracker when the + peer completes a download. The peer SHOULD NOT send this value + if it started up with the complete torrent. + + +5.2 Response + + Upon receiving the HTTP GET request, the tracker MUST respond with a + document having the "text/plain" MIME type. This document MUST + contain a bencoded dictionary with the following keys: + + 'failure reason': This key is OPTIONAL. If present, the dictionary + MUST NOT contain any other keys. The peer should interpret this + as if the attempt to join the torrent failed. The value is a + human readable string containing an error message with the failure + reason. + + 'interval': A peer must send regular HTTP GET requests to the tracker + to obtain an updated list of peers and update the tracker of its + status. The value of this key indicated the amount of time that a + peer should wait between to consecutive regular requests. This + key is REQUIRED. + + 'complete': This is an integer that indicates the number of seeders. + This key is OPTIONAL. + + 'incomplete': This is an integer that indicates the number of peers + downloading the torrent. This key is OPTIONAL. + + 'peers': This is a bencoded list of dictionaries containing a list of + peers that must be contacted in order to download a file. This + key is REQUIRED. It has the following structure: + + 'peer id': This is a REQUIRED string value containing the self- + designated ID of the peer. + + + + + + +Fonseca, et al. [Page 14] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + 'ip': This is a REQUIRED string value indicating the IP address of + the peer. This may be given as a dotted quad IPv4 format, + hexadecimal IPv6 format or DNS name. + + 'port': This is an integer value. It must contain the self- + designated port number of the peer. This key is REQUIRED. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 15] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +6. The Peer Wire Protocol + + The aim of the PWP, is to facilitate communication between + neighboring peers for the purpose of sharing file content. PWP + describes the steps taken by a peer after it has read in a metainfo + file and contacted a tracker to gather information about other peers + it may communicate with. PWP is layered on top of TCP and handles + all its communication using asynchronous messages. + +6.1 Peer Wire Guidelines + + PWP does not specify a standard algorithm for selecting elements from + a clients neighboring peers with whom to share pieces, although the + following guidelines are expected to be observed by any such + algorithm: + + The algorithm should not be constructed with the goal in mind to + reduce the amount of data uploaded compared to downloaded. At the + very least a peer should upload the same amount that it has + downloaded. + + The algorithm should not use a strict tit-for-tat schema when + dealing with remote peers that have just joined the swarm and thus + have no pieces to offer. + + The algorithm should make good use of both download and upload + bandwidth by putting a cap on the number of simultaneous + connection that actively send or receive data. By reducing the + number of active connections, TCP congestion can be avoided. + + The algorithm should pipeline data requests in order so saturate + active connections. + + The algorithm should be able to cooperate with peers that + implement a different algorithm. + + +6.2 Handshaking + + The local peer opens a port on which to listen for incoming + connections from remote peers. This port is then reported to the + tracker. As BTP/1.0 does not specify any standard port for listening + it is the sole responsibility of the implementation to select a port. + + Any remote peer wishing to communicate with the local peer must open + a TCP connection to this port and perform a handshake operation. The + handshake operation MUST be carried out before any other data is sent + from the remote peer. The local peer MUST NOT send any data back to + + + +Fonseca, et al. [Page 16] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + the remote peer before a well constructed handshake has been + recognized according to the rules below. If the handshake in any way + violates these rules the local peer MUST close the connection with + the remote peer. + + + + + + A handshake is a string of bytes with the following structure: + + ---------------------------------------------------------------- + | Name Length | Protocol Name | Reserved | Info Hash | Peer ID | + ---------------------------------------------------------------- + + Name Length: The unsigned value of the first byte indicates the + length of a character string containing the protocol name. In + BTP/1.0 this number is 19. The local peer knows its own protocol + name and hence also the length of it. If this length is different + than the value of this first byte, then the connection MUST be + dropped. + + Protocol Name: This is a character string which MUST contain the + exact name of the protocol in ASCII and have the same length as + given in the Name Length field. The protocol name is used to + identify to the local peer which version of BTP the remote peer + uses. In BTP/1.0 the name is 'BitTorrent protocol'. If this + string is different from the local peers own protocol name, then + the connection is to be dropped. + + Reserved: The next 8 bytes in the string are reserved for future + extensions and should be read without interpretation. + + Info Hash: The next 20 bytes in the string are to be interpreted as a + 20-byte SHA1 of the info key in the metainfo file. Presumably, + since both the local and the remote peer contacted the tracker as + a result of reading in the same .torrent file, the local peer will + recognize the info hash value and will be able to serve the remote + peer. If this is not the case, then the connection MUST be + dropped. This situation can arise if the local peer decides to no + longer serve the file in question for some reason. The info hash + may be used to enable the client to serve multiple torrents on the + same port. + + At this stage, if the connection has not been dropped, then the + local peer MUST send its own handshake back, which includes the + last step: + + + + +Fonseca, et al. [Page 17] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + Peer ID: The last 20 bytes of the handshake are to be interpreted as + the self-designated name of the peer. The local peer must use + this name to identify the connection hereafter. Thus, if this + name matches the local peers own ID name, the connection MUST be + dropped. Also, if any other peer has already identified itself to + the local peer using that same peer ID, the connection MUST be + dropped. + + In BTP/1.0 the handshake has a total of 68 bytes. + +6.3 Message Communication + + Following the PWP handshake both ends of the TCP channel may send + messages to each other in a completely asynchronous fashion. PWP + messages have the dual purpose of updating the state of neighboring + peers with regard to changes in the local peer, as well as + transferring data blocks between neighboring peers. + + PWP Messages fall into two different categories: + + State-oriented messages: These messages serve the sole purpose of + informing peers of changes in the state of neighboring peers. A + message of this type MUST be sent whenever a change occurs in a + peer's state, regardless of the state of other peers. The + following messages fall into this category: Interested, + Uninterested, Choked, Unchoked, Have and Bitfield. + + Data-oriented messages: These messages handle the requesting and + sending of data portions. The following messages fall into this + category: Request, Cancel and Piece. + + +6.3.1 Peer States + + For each end of a connection, a peer must maintain the following two + state flags: + + Choked: When true, this flag means that the choked peer is not + allowed to request data. + + Interested: When true, this flag means a peer is interested in + requesting data from another peer. This indicates that the peer + will start requesting blocks if it is unchoked. + + A choked peer MUST not send any data-oriented messages, but is free + to send any other message to the peer that has choked it. If a peer + chokes a remote peer, it MUST also discard any unanswered requests + for blocks previously received from the remote peer. + + + +Fonseca, et al. [Page 18] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + An unchoked peer is allowed to send data-oriented messages to the + remote peer. It is left to the implementation how many peers any + given peer may choose to choke or unchoke, and in what fashion. This + is done deliberately to allow peers to use different heuristics for + peer selection. + + An interested peer indicates to the remote peer that it must expect + to receive data-oriented messages as soon as it unchokes the + interested peer. It must be noted, that a peer must not assume a + remote peer is interested solely because it has pieces that the + remote peer is lacking. There may be valid reasons why a peer is not + interested in another peer other than data-based ones. + +6.3.2 Peer Wire Messages + + All integer members in PWP messages are encoded as a 4-byte big- + endian number. Furthermore, all index and offset members in PWP + messages are zero-based. + + A PWP message has the following structure: + + ----------------------------------------- + | Message Length | Message ID | Payload | + ----------------------------------------- + + Message Length: This is an integer which denotes the length of the + message, excluding the length part itself. If a message has no + payload, its size is 1. Messages of size 0 MAY be sent + periodically as keep-alive messages. Apart from the limit that + the four bytes impose on the message length, BTP does not specify + a maximum limit on this value. Thus an implementation MAY choose + to specify a different limit, and for instance disconnect a remote + peer that wishes to communicate using a message length that would + put too much strain on the local peer's resources. + + Message ID: This is a one byte value, indicating the type of the + message. BTP/1.0 specifies 9 different messages, as can be seen + further below. + + Payload: The payload is a variable length stream of bytes. + + If an incoming message in any way violates this structure then the + connection SHOULD be dropped. In particular the receiver SHOULD make + sure the message ID constitutes a valid message, and the payload + matches the the expected payload, as given below. + + For the purpose of compatibility with future protocol extensions the + client SHOULD ignore unknown messages. There may arise situations in + + + +Fonseca, et al. [Page 19] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + which a client may choose to drop a connection after receiving an + unknown message, either for security reasons, or because discarding + large unknown messages may be viewed as excessive waste. + + BTP/1.0 specifies the following messages: + +6.3.3 Choke + + This message has ID 0 and no payload. A peer sends this message to a + remote peer to inform the remote peer that it is being choked. + +6.3.4 Unchoke + + This message has ID 1 and no payload. A peer sends this message to a + remote peer to inform the remote peer that it is no longer being + choked. + +6.3.5 Interested + + This message has ID 2 and no payload. A peer sends this message to a + remote peer to inform the remote peer of its desire to request data. + +6.3.6 Uninterested + + This message has ID 3 and no payload. A peer sends this message to a + remote peer to inform it that it is not interested in any pieces from + the remote peer. + +6.3.7 Have + + This message has ID 4 and a payload of length 4. The payload is a + number denoting the index of a piece that the peer has successfully + downloaded and validated. A peer receiving this message must + validate the index and drop the connection if this index is not + within the expected bounds. Also, a peer receiving this message MUST + send an interested message to the sender if indeed it lacks the piece + announced. Further, it MAY also send a request for that piece. + +6.3.8 Bitfield + + This message has ID 5 and a variable payload length. The payload is + a bitfield representing the pieces that the sender has successfully + downloaded, with the high bit in the first byte corresponding to + piece index 0. If a bit is cleared it is to be interpreted as a + missing piece. A peer MUST send this message immediately after the + handshake operation, and MAY choose not to send it if it has no + pieces at all. This message MUST not be sent at any other time + during the communication. + + + +Fonseca, et al. [Page 20] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +6.3.9 Request + + This message has ID 6 and a payload of length 12. The payload is 3 + integer values indicating a block within a piece that the sender is + interested in downloading from the recipient. The recipient MUST + only send piece messages to a sender that has already requested it, + and only in accordance to the rules given above about the choke and + interested states. The payload has the following structure: + + --------------------------------------------- + | Piece Index | Block Offset | Block Length | + --------------------------------------------- + + +6.3.10 Piece + + This message has ID 7 and a variable length payload. The payload + holds 2 integers indicating from which piece and with what offset the + block data in the 3rd member is derived. Note, the data length is + implicit and can be calculated by subtracting 9 from the total + message length. The payload has the following structure: + + ------------------------------------------- + | Piece Index | Block Offset | Block Data | + ------------------------------------------- + + +6.3.11 Cancel + + This message has ID 8 and a payload of length 12. The payload is 3 + integer values indicating a block within a piece that the sender has + requested for, but is no longer interested in. The recipient MUST + erase the request information upon receiving this messages. The + payload has the following structure: + + --------------------------------------------- + | Piece Index | Block Offset | Block Length | + --------------------------------------------- + + +6.4 The End Game + + Towards the end of a download session, it may speed up the download + to send request messages for the remaining blocks to all the + neighboring peers. A client must issue cancel messages to all + pending requests sent to neighboring peers as soon as a piece is + downloaded successfully. This is referred to as the end game. + + + + +Fonseca, et al. [Page 21] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + A client usually sends requests for blocks in stages; sending + requests for newer blocks as replies for earlier requests are + received. The client enters the end game, when all remaining blocks + have been requested. + +6.5 Piece Selection Strategy + + BTP/1.0 does not force a particular order for selecting which pieces + to download. However, experience shows that downloading in rarest- + first order seems to lessen the wait time for pieces. To find the + rarest piece a client must calculate for each piece index the number + of times this index is true in the bitfield vectors of all the + neighboring peers. The piece with the lowest sum is then selected + for requesting. + +6.6 Peer Selection Strategy + + This section describes the choking algorithm recommended for + selecting neighboring peers with whom to exchange pieces. + Implementations are free to implement any strategy as long as the + guidelines in Section 6.1 are observed. + + After the initial handshake both ends of a connection set the Choked + flag to true and the Interested flag to false. + + All connections are periodically rated in terms of their ability to + provide the client with a better download rate. The rating may take + into account factors such as the remote peers willingness to maintain + an unchoked connection with the client over a certain period of time, + the remote peers upload rate to the client and other implementation + defined criteria. + + The peers are sorted according to their rating with regard to the + above mentioned scheme. Assume only 5 peers are allowed to download + at the same time. The peer selection algorithm will now unchoke as + many of the best rated peers as necessary so that exactly 5 of these + are interested. If one of the top rated peers at a later stage + becomes interested, then the peer selection algorithm will choke the + the worst unchoked peer. Notice that the worst unchoked peer is + always interested. + + The only lacking element from the above algorithm is the capability + to ensure that new peers can have a fair chance of downloading a + piece, even though they would evaluate poorly in the above schema. A + simple method is to make sure that a random peer is selected + periodically regardless of how it evaluates. Since this process is + repeated in a round robin manner, it ensures that ultimately even new + peers will have a chance of being unchoked. + + + +Fonseca, et al. [Page 22] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +7. Security Consideration + + This section examines security considerations for BTP/1.0.The + discussion does not include definitive solutions to the problems + revealed, though it does make some suggestions for reducing security + risks. + +7.1 Tracker HTTP Protocol Issues + + The use of the HTTP protocol for communication between the tracker + and the client makes BTP/1.0 vulnerable to the attacks mentioned in + the security consideration section of RFC 2616 [6]. + +7.2 Denial of Service Attacks on Trackers + + The nature of the tracker is to serve many clients. By mounting a + denial of service attack against the tracker the swarm attached to + the tracker can be starved. This type of attack is hard to defend + against, however, the metainfo file allows for multiple trackers to + be specified, making it possible to spread the load on a number of + trackers, and thus containing such an attack. + +7.3 Peer Identity Issues + + There is no strong authentication of clients when they contact the + tracker. The main option for trackers is to check peer ID and the IP + address of the client. The lack of authentication can be used to + mount an attack where a client can shut down another client if the + two clients are running on the same host and thus are sharing the + same IP address. In addition, a rogue peer may masquerade its + identity by using multiple peer IDs. Clients should there refrain + from taking the peer ID at face value. + +7.4 DNS Spoofing + + Clients using BTP/1.0 rely heavily on the Domain Name Service, which + can be used for both specifying the URI of the tracker and how to + contact a peer. Clients are thus generally prone to security attacks + based on the deliberate mis-association of IP addresses and DNS + names. Clients need to be cautious in assuming the continuing + validity of an IP address/DNS name association. + + In particular, BTP/1.0 clients SHOULD rely on their name resolver for + confirmation of an IP number/DNS name association, rather than + caching the result of previous host name lookups. If clients cache + the results of host name lookups in order to achieve a performance + improvement, they MUST observe the TTL information reported by DNS. + + + + +Fonseca, et al. [Page 23] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + If clients do not observe this rule, they could be spoofed when a + previously-accessed peers or trackers IP address changes. As network + renumbering is expected to become increasingly common according to + RFC 1900 [2], the possibility of this form of attack will grow. + Observing this requirement reduces this potential security + vulnerability. + +7.5 Issues with File and Directory Names + + The metainfo file provides a way to suggest a name of the downloaded + file for single-file torrents and the top-most directory for multi- + file torrents. This functionality is very like the Content- + Disposition header field documented in RFC 2183 [4] and the security + considerations mentioned in this RFC also apply to BitTorrent + clients. In short, BTP clients SHOULD verify that the suggested file + names in the metainfo file do not compromise services on the local + system. Furthermore, care must be taken for multi-file torrents to + validate that individual files are relative to the top-most directory + and that the paths do not contain path elements to the parent (that + is directory elevators such as ``..''), which can be used to place + files outside the top-most directory. + + Using UNIX as an example, some hazards would be: + + o Creating startup files (e.g., ".login"). + + o Creating or overwriting system files (e.g., "/etc/passwd"). + + o Overwriting any existing file. + + o Placing executable files into any command search path (e.g., + "~/bin/more"). + + o Sending the file to a pipe (e.g., "| sh"). + + It is very important to note that this is not an exhaustive list; it + is intended as a small set of examples only. Implementers must be + alert to the potential hazards on their target systems. In general, + the BTP client SHOULD NOT name or place files such that they will get + interpreted or executed without the user explicitly initiating the + action. + +7.6 Validating the Integrity of Data Exchanged Between Peers + + By default, all content served to the client from other peers should + be considered tainted and the client SHOULD validate the integrity of + the data before accepting it. The metainfo file contains information + for checking both individual pieces using SHA1, and optionally + + + +Fonseca, et al. [Page 24] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + + individual files using MD5. SHA1, being the strongest of the two, is + preferred. Furthermore, sole reliance on whole-file checking can + potentially render otherwise valid pieces invalid, and should only be + considered for small files, to limit the amount of data being + discarded. + + Trusting the validity of the resulting file or files ends up being a + matter of trusting the content of the metainfo file. Ensuring the + validity of the metainfo file is beyond the scope of this document. + +7.7 Transfer of Sensitive Information + + Some clients include information about themselves when generating the + peer ID string. Clients should be aware that this information can + potentially be used to determine whether a specific client has a + exploitable security hole. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Fonseca, et al. [Page 25] + +BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 + + +8. IANA Considerations + + This document makes no request of IANA. + +9. References + + [1] Postel, J. and J. Reynolds, "File Transfer Protocol", STD 9, + RFC 959, October 1985. + + [2] Carpenter, B. and Y. Rekhter, "Renumbering Needs Work", + RFC 1900, February 1996. + + [3] Bradner, S., "Key words for use in RFCs to Indicate Requirement + Levels", BCP 14, RFC 2119, March 1997. + + [4] Troost, R., Dorner, S., and K. Moore, "Communicating + Presentation Information in Internet Messages: The Content- + Disposition Header Field", RFC 2183, August 1997. + + [5] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax + Specifications: ABNF", RFC 2234, November 1997. + + [6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., + Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- + HTTP/1.1", RFC 2616, June 1999. + + +Authors' Addresses + + Jonas Fonseca + DIKU + + Email: fonseca@diku.dk + + + Basim Reza + DIKU + + Email: basim@diku.dk + + + Lilja Fjeldsted + DIKU + + Email: lilja@diku.dk + + + + + + +Fonseca, et al. [Page 26] + diff --git a/docs/links.txt b/docs/links.txt new file mode 100644 index 0000000..55902ef --- /dev/null +++ b/docs/links.txt @@ -0,0 +1,11 @@ +http://jonas.nitro.dk/bittorrent/bittorrent-rfc.html + +http://bittorrent.org/beps/bep_0003.html +http://bittorrent.org/beps/bep_0005.html + +https://wiki.theory.org/BitTorrentSpecification +http://wiki.vuze.com/w/Azureus_messaging_protocol + +Some random overview +http://www.kristenwidman.com/blog/33/how-to-write-a-bittorrent-client-part-1/ +http://www.kristenwidman.com/blog/71/how-to-write-a-bittorrent-client-part-2/ diff --git a/docs/out.txt b/docs/out.txt new file mode 100644 index 0000000..82bf6de --- /dev/null +++ b/docs/out.txt @@ -0,0 +1,65 @@ +metainfo file.: OpenBSD_songs_ogg-2016-03-25-0127.torrent +info hash.....: 1aa5af9f6f533961a65169bdeeb801e611724d32 + +directory name: OpenBSD_songs_ogg +files.........: 38 + OpenBSD_songs_ogg/song30.ogg (2636244) 2.51 MB + OpenBSD_songs_ogg/song31.ogg (2381018) 2.27 MB + OpenBSD_songs_ogg/song32.ogg (2253880) 2.15 MB + OpenBSD_songs_ogg/song33.ogg (3238559) 3.09 MB + OpenBSD_songs_ogg/song34.ogg (5062603) 4.83 MB + OpenBSD_songs_ogg/song35.ogg (7191264) 6.86 MB + OpenBSD_songs_ogg/song36.ogg (5197630) 4.96 MB + OpenBSD_songs_ogg/song37.ogg (13937255) 13.29 MB + OpenBSD_songs_ogg/song38.ogg (5860681) 5.59 MB + OpenBSD_songs_ogg/song38b.ogg (5759105) 5.49 MB + OpenBSD_songs_ogg/song39.ogg (5954997) 5.68 MB + OpenBSD_songs_ogg/song40.ogg (3610740) 3.44 MB + OpenBSD_songs_ogg/song41.ogg (8382781) 7.99 MB + OpenBSD_songs_ogg/song42.ogg (6713047) 6.40 MB + OpenBSD_songs_ogg/song43.ogg (6834505) 6.52 MB + OpenBSD_songs_ogg/song44.ogg (4628329) 4.41 MB + OpenBSD_songs_ogg/song45.ogg (4737475) 4.52 MB + OpenBSD_songs_ogg/song46.ogg (3766867) 3.59 MB + OpenBSD_songs_ogg/song47.ogg (6586251) 6.28 MB + OpenBSD_songs_ogg/song48.ogg (3143847) 3.00 MB + OpenBSD_songs_ogg/song49.ogg (6006834) 5.73 MB + OpenBSD_songs_ogg/song50.ogg (4184951) 3.99 MB + OpenBSD_songs_ogg/song51.ogg (4216946) 4.02 MB + OpenBSD_songs_ogg/song52.ogg (4283099) 4.08 MB + OpenBSD_songs_ogg/song53.ogg (4635216) 4.42 MB + OpenBSD_songs_ogg/song54.ogg (3195265) 3.05 MB + OpenBSD_songs_ogg/song55.ogg (6160151) 5.87 MB + OpenBSD_songs_ogg/song56.ogg (5471902) 5.22 MB + OpenBSD_songs_ogg/song57.ogg (4057801) 3.87 MB + OpenBSD_songs_ogg/song58a.ogg (3252520) 3.10 MB + OpenBSD_songs_ogg/song58b.ogg (4376163) 4.17 MB + OpenBSD_songs_ogg/song58c.ogg (3595317) 3.43 MB + OpenBSD_songs_ogg/song58d.ogg (7013091) 6.69 MB + OpenBSD_songs_ogg/song59a.ogg (5766764) 5.50 MB + OpenBSD_songs_ogg/song59b.ogg (5317415) 5.07 MB + OpenBSD_songs_ogg/songsh.ogg (4890381) 4.66 MB + OpenBSD_songs_ogg/songsi.ogg (5694670) 5.43 MB + OpenBSD_songs_ogg/songty.ogg (6045645) 5.77 MB +archive size..: 196041209 (747 * 262144 + 219641) 186.96 MB + +announce url..: http://OpenBSD.somedomain.net:6969/announce +creation date.: Fri Mar 25 02:27:56 2016 +created by....: mktorrent 1.0 +comment.......: ogg files from OpenBSD/songs +Created by andrew fresh (andrew@afresh1.com) +http://OpenBSD.somedomain.net/ + + +metainfo file.: OpenBSD_5.9_amd64_install59.iso-2016-03-29-0449.torrent +info hash.....: e840038dea1998c39614dcd28594501df02bd32d + +file name.....: OpenBSD_5.9_amd64_install59.iso +file size.....: 233662464 (891 * 262144 + 92160) 222.84 MB + +announce url..: http://OpenBSD.somedomain.net:6969/announce +creation date.: Tue Mar 29 06:49:18 2016 +created by....: mktorrent 1.0 +comment.......: OpenBSD/5.9/amd64/install59.iso +Created by andrew fresh (andrew@afresh1.com) +http://OpenBSD.somedomain.net/ -- cgit v1.2.3