From 38fc10d3858be4a4161775a52e9bbb5b97c23d7e Mon Sep 17 00:00:00 2001 From: Dimitri Sokolyuk Date: Sat, 15 Dec 2018 14:43:12 +0100 Subject: rename docs --- doc/bittorrent-rfc.txt | 1456 ------------------------------------------------ doc/links.txt | 11 - doc/out.txt | 65 --- 3 files changed, 1532 deletions(-) delete mode 100644 doc/bittorrent-rfc.txt delete mode 100644 doc/links.txt delete mode 100644 doc/out.txt (limited to 'doc') diff --git a/doc/bittorrent-rfc.txt b/doc/bittorrent-rfc.txt deleted file mode 100644 index 38fc8ac..0000000 --- a/doc/bittorrent-rfc.txt +++ /dev/null @@ -1,1456 +0,0 @@ - - - -BitTorrent Protocol version 1.0 J. Fonseca -$Revision: 1.33 $ B. Reza - L. Fjeldsted - DIKU - April 2005 - - - BitTorrent Protocol -- BTP/1.0 - - -Abstract - - This document describes the BitTorrent Protocol version 1.0 referred - to as "BTP/1.0". The BitTorrent Protocol (BTP) is a protocol for - collaborative file distribution across the Internet and has been in - place on the Internet since 2002. It is best classified as a peer- - to-peer (P2P) protocol, although it also contains highly centralized - elements. BTP has already been implemented many times for different - platforms, and could well be said to be a mature protocol, although a - formal, detailed and complete description has so far been lacking. - - BTP was devised and implemented by Bram Cohen as a P2P replacement to - standard File Transfer Protocol (FTP) to be used in places where the - usage of an FTP implementation poses too much strain on the server in - terms of request-processing and sheer bandwidth. Normally a client - does not use her upload capacity while downloading a file. The BTP - approach capitalizes on this fact by having clients upload bits of - the data to each other. In comparison to FTP this adds huge - scalability and cost-management advantages. - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 1] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -Table of Contents - - 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 4 - 1.1 Extensions . . . . . . . . . . . . . . . . . . . . . . . . 4 - 1.2 Audience . . . . . . . . . . . . . . . . . . . . . . . . . 4 - 1.3 Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 - 1.4 Overall Operation . . . . . . . . . . . . . . . . . . . . 5 - 2. Bencoding . . . . . . . . . . . . . . . . . . . . . . . . 7 - 2.1 Scalar Types . . . . . . . . . . . . . . . . . . . . . . . 7 - 2.2 Compound Types . . . . . . . . . . . . . . . . . . . . . . 7 - 3. Pieces and Blocks . . . . . . . . . . . . . . . . . . . . 8 - 3.1 Pieces . . . . . . . . . . . . . . . . . . . . . . . . . . 8 - 3.2 Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . 8 - 4. The Metainfo File . . . . . . . . . . . . . . . . . . . . 10 - 4.1 The Structure of the Metainfo File . . . . . . . . . . . . 10 - 4.1.1 Single File Torrents . . . . . . . . . . . . . . . . . . . 10 - 4.1.2 Multi File Torrents . . . . . . . . . . . . . . . . . . . 11 - 5. The Tracker HTTP Protocol . . . . . . . . . . . . . . . . 13 - 5.1 Request . . . . . . . . . . . . . . . . . . . . . . . . . 13 - 5.2 Response . . . . . . . . . . . . . . . . . . . . . . . . . 14 - 6. The Peer Wire Protocol . . . . . . . . . . . . . . . . . . 16 - 6.1 Peer Wire Guidelines . . . . . . . . . . . . . . . . . . . 16 - 6.2 Handshaking . . . . . . . . . . . . . . . . . . . . . . . 16 - 6.3 Message Communication . . . . . . . . . . . . . . . . . . 18 - 6.3.1 Peer States . . . . . . . . . . . . . . . . . . . . . . . 18 - 6.3.2 Peer Wire Messages . . . . . . . . . . . . . . . . . . . . 19 - 6.3.3 Choke . . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.4 Unchoke . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.5 Interested . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.6 Uninterested . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.7 Have . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.8 Bitfield . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 6.3.9 Request . . . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.3.10 Piece . . . . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.3.11 Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.4 The End Game . . . . . . . . . . . . . . . . . . . . . . . 21 - 6.5 Piece Selection Strategy . . . . . . . . . . . . . . . . . 22 - 6.6 Peer Selection Strategy . . . . . . . . . . . . . . . . . 22 - 7. Security Consideration . . . . . . . . . . . . . . . . . . 23 - 7.1 Tracker HTTP Protocol Issues . . . . . . . . . . . . . . . 23 - 7.2 Denial of Service Attacks on Trackers . . . . . . . . . . 23 - 7.3 Peer Identity Issues . . . . . . . . . . . . . . . . . . . 23 - 7.4 DNS Spoofing . . . . . . . . . . . . . . . . . . . . . . . 23 - 7.5 Issues with File and Directory Names . . . . . . . . . . . 24 - 7.6 Validating the Integrity of Data Exchanged Between Peers . 24 - 7.7 Transfer of Sensitive Information . . . . . . . . . . . . 25 - 8. IANA Considerations . . . . . . . . . . . . . . . . . . . 26 - 9. References . . . . . . . . . . . . . . . . . . . . . . . . 26 - - - -Fonseca, et al. [Page 2] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - Authors' Addresses . . . . . . . . . . . . . . . . . . . . 26 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 3] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -1. Introduction - - The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in RFC 2119 [3]. - - The File Transfer Protocol (FTP) [1] with the recent additions of - security extensions, still remains the standard for secure and - reliable transmission of large files over the Internet. However, its - highly centralized client-server approach also means, it is - inadequate for mass publication of files, where a single point may - expect to be requested by a critically large number of clients - simultaneously. To remedy this situation many organizations either - implement a cap on the number of simultaneous requests, or spread the - load on multiple mirror servers. Needless to say both approaches - have their drawbacks, and a solution that addresses these problems is - highly needed. - - The approach in BitTorrent Protocol (BTP) is to spread the load not - on mirror servers, but to the clients themselves by having them - upload bits of the file to each other while downloading it. Since - the clients usually do not utilize their upload capacity while - fetching a file, this approach does not put the clients in any - disadvantage. This has the added advantage that even small - organizations with limited resources can publish large files on the - Internet without having to invest in costly infrastructure. - -1.1 Extensions - - Since the introduction of BTP many modifications and extensions have - been proposed by individuals and community forums. To the extent - that these extensions have become part of what the BitTorrent - community considers best practice they have been included in this - document. However, many extensions have been omitted either because - they have been deemed to lack interoperability with existing - implementations, or because they are not regarded as being - sufficiently mature. - -1.2 Audience - - This document is aimed at developers who wish to implement BTP for a - particular platform. Also, system administrators and architects may - use this document to fully understand the implications of installing - an implementation of BTP. In particular, it is advised to study the - security implications in more detail, before installing an - implementation on a machine that also contains sensitive data. - Security implications are discussed in Section 7. - - - - -Fonseca, et al. [Page 4] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -1.3 Terminology - - Peer: A peer is a node in a network participating in file sharing. - It can simultaneously act both as a server and a client to other - nodes on the network. - - Neighboring peers: Peers to which a client has an active point to - point TCP connection. - - Client: A client is a user agent (UA) that acts as a peer on behalf - of a user. - - Torrent: A torrent is the term for the file (single-file torrent) or - group of files (multi-file torrent) the client is downloading. - - Swarm: A network of peers that actively operate on a given torrent. - - Seeder: A peer that has a complete copy of a torrent. - - Tracker: A tracker is a centralized server that holds information - about one or more torrents and associated swarms. It functions as - a gateway for peers into a swarm. - - Metainfo file: A text file that holds information about the torrent, - e.g. the URL of the tracker. It usually has the extension - .torrent. - - Peer ID: A 20-byte string that identifies the peer. How the peer ID - is obtained is outside the scope of this document, but a peer must - make sure that the peer ID it uses has a very high probability of - being unique in the swarm. - - Info hash: A SHA1 hash that uniquely identifies the torrent. It is - calculated from data in the metainfo file. - - -1.4 Overall Operation - - BTP consists of two logically distinct protocols, namely the Tracker - HTTP Protocol (THP), and the Peer Wire Protocol (PWP). THP defines a - method for contacting a tracker for the purposes of joining a swarm, - reporting progress etc. PWP defines a mechanism for communication - between peers, and is thus responsible for carrying out the actual - download and upload of the torrent. - - In order for a client to download a torrent the following steps must - be carried through: - - - - -Fonseca, et al. [Page 5] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 1. A metainfo file must be retrieved. - - 2. Instructions that will allow the client to contact other peers - must be periodically requested from the tracker using THP. - - 3. The torrent must be downloaded by connecting to peers in the - swarm and trading pieces using PWP. - - To publish a torrent the following steps must be taken: - - 1. A tracker must be set up. - - 2. A metainfo file pointing to the tracker and containing - information on the structure of the torrent must be produced and - published. - - 3. At least one seeder with access to the complete torrent must be - set up. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 6] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -2. Bencoding - - Bencoding encodes data in a platform independent way. In BTP/1.0 the - metainfo file and all responses from the tracker are encoded in the - bencoding format. The format specifies two scalar types (integers - and strings) and two compound types (lists and dictionaries). - - The Augmented BNF syntax [5] for bencoding format is: - - dictionary = "d" 1*(string anytype) "e" ; non-empty dictionary - list = "l" 1*anytype "e" ; non-empty list - integer = "i" signumber "e" - string = number ":" - anytype = dictionary / list / integer / string - signumber = "-" number / number - number = 1*DIGIT - CHAR = %x00-FF ; any 8-bit character - DIGIT = "0" / "1" / "2" / "3" / "4" / - "5" / "6" / "7" / "8" / "9" - -2.1 Scalar Types - - Integers are encoded by prefixing a string containing the base ten - representation of the integer with the letter "i" and postfixing it - with the letter "e". E.g. the integer 123 is encoded as i123e. - - Strings are encoded by prefixing the string content with the length - of the string followed by a colon. E.g. the string "announce" is - encoded as "8:announce". - -2.2 Compound Types - - The compound types provides a mean to structure elements of any - bencoding type. - - Lists are an arbitrary number of bencoded elements prefixed with the - letter "l" and postfixed with the letter "e". It follows that lists - can contain nested lists and dictionaries. For instance "li2e3:fooe" - defines a list containing the integer "2" and the string "foo". - - Dictionaries are an arbitrary number of key/value pairs delimited by - the letter "d" at the beginning and the letter "e" at the end. All - keys are bencoded strings while the associated value can be any - bencoded element. E.g. "d5:monthi4e4:name5:aprile" defines a - dictionary holding the associations: "month" => "4" and "name" => - "april". All dictionary keys MUST be sorted. - - - - - -Fonseca, et al. [Page 7] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -3. Pieces and Blocks - - This section describes how a torrent is organized in pieces and - blocks. The torrent is divided into one or more pieces. Each piece - represents a range of data which it is possible to verify using a - piece SHA1 hash. When distributing data over PWP pieces are divided - into one or more blocks, as shown in the following diagram: - - --------------------------------------- - | Piece #0 | Piece #1 | .. | Piece #N | - --------------------------------------- - _-' `-_ - _-' `-_ - ---------------------------- - | Block #0 | .. | Block #M | - ---------------------------- - -3.1 Pieces - - The number of pieces in the torrent is indicated in the metainfo - file. The size of each piece in the torrent remains fixed and can be - calculated using the following formula: - - fixed_piece_size = size_of_torrent / number_of_pieces - - where "/" is the integer division operator. Only the last piece of - the torrent is allowed to have fewer bytes than the fixed piece size. - - The size of a piece is determined by the publisher of the torrent. A - good recommendation is to use a piece size so that the metainfo file - does not exceed 70 kilobytes. - - For the sake of calculating the correct position of a piece within a - file, or files, the torrent is regarded as a single continuous byte - stream. In case the torrent consists of multiple files, it is to be - viewed as the concatenation of these files in the order of their - appearance in the metainfo file. Conceptually, the torrent is only - translated into files when all its pieces have been downloaded and - verified using their respective SHA1 values; although in practice an - implementation may choose a better approach in accordance with local - operating system and filesystem specific demands. - -3.2 Blocks - - The size of a block is an implementation defined value that is not - dependant on the fixed piece size. Once a fixed size is defined, the - number of blocks per piece can be calculated using the formula: - - - - -Fonseca, et al. [Page 8] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - number_of_blocks = (fixed_piece_size / fixed_block_size) - + !!(fixed_piece_size % fixed_block_size) - - where "%" denotes the modulus operator, and "!" the negation - operator. The negation operator is used to ensure that the last - factor only adds a value of 0 or 1 to the sum. Given the start - offset of the block its index within a piece can be calculated using - the formula: - - block_index = block_offset % fixed_block_size - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 9] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -4. The Metainfo File - - The metainfo file provides the client with information on the tracker - location as well as the torrent to be downloaded. Besides listing - which files will result from downloading the torrent, it also lists - how the client should split up and verify individual pieces making up - the complete torrent. - - In order for a client to recognize the metainfo file it SHOULD have - the extension .torrent and the associated the media type - "application/x-bittorrent". How the client retrieves the metainfo - file is beyond the scope of this document, however, the most user- - friendly approach is for a client to find the file on a web page, - click on it, and start the download immediately. This way, the - apparent complexity of BTP as opposed to FTP or HTTP transfer is - transparent to the user. - -4.1 The Structure of the Metainfo File - - The metainfo file contains a bencoded dictionary with the following - structure. A key below is REQUIRED unless otherwise noted. - - 'announce': This is a string value. It contains the announce URL of - the tracker. - - 'announce-list': This is an OPTIONAL list of string values. Each - value is a URL pointing to a backup tracker. This value is not - used in BTP/1.0. - - 'comment': This is an OPTIONAL string value that may contain any - comment by the author of the torrent. - - 'created by': This is an optional string value and may contain the - name and version of the program used to create the metainfo file. - - 'creation date': This is an OPTIONAL string value. It contains the - creation time of the torrent in standard Unix epoch format. - - 'info': This key points to a dictionary that contains information - about the files to download. The entries are explained in the - following sections. - - -4.1.1 Single File Torrents - - If the torrent only specifies one file, the info dictionary must have - the following keys: - - - - -Fonseca, et al. [Page 10] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'length': This is an integer value indicating the length of the file - in bytes. - - 'md5sum': This is an OPTIONAL value. If included it must be a string - of 32 characters corresponding to the MD5 sum of the file. This - value is not used in BTP/1.0. - - 'name': A string containing the name of the file. - - 'piece length': An integer indicating the number of bytes in each - piece. - - 'pieces': This is a string value containing the concatenation of the - 20-byte SHA1 hash value for all pieces in the torrent. For - example, the first 20 bytes of the string represent the SHA1 value - used to verify piece index 0. - - The complete file is derived by combining all the pieces into one - string of bytes. - -4.1.2 Multi File Torrents - - If the torrent specifies multiple files, the info dictionary must - have the following structure: - - 'files': This is a list of dictionaries. Each file in the torrent - has a dictionary associated to it having the following structure: - - 'length': This is an integer indicating the total length of the - file in bytes. - - 'md5sum': This is an OPTIONAL value. if included it must be a - string of 32 characters corresponding to the MD5 sum of the - file. This value is not used in BTP/1.0. - - 'path': This is a list of string elements that specify the path of - the file, relative to the topmost directory. The last element - in the list is the name of the file, and the elements preceding - it indicate the directory hierarchy in which this file is - situated. - - 'name': This is a string value. It contains the name of the top-most - directory in the file structure. - - 'piece length': This is an integer value. It contains the number of - bytes in each piece. - - - - - -Fonseca, et al. [Page 11] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'pieces': This is a string value. It must contain the concatenation - of all 20-byte SHA1 hash values that are used by BTP/1.0 to verify - each downloaded piece. The first 20 bytes of the string represent - the SHA1 value used to verify piece index 0. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 12] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -5. The Tracker HTTP Protocol - - The Tracker HTTP Protocol (THP) is a simple mechanism for introducing - peers to each other. A tracker is a HTTP service that must be - contacted by a peer in order to join a swarm. As such the tracker - constitutes the only centralized element in BTP/1.0. A tracker does - not by itself provide access to any downloadable data. A tracker - relies on peers sending regular requests. It may assume that a peer - is dead if it misses a request. - -5.1 Request - - To contact the tracker a peer MUST send a standard HTTP GET request - using the URL in the "announce" entry of the metainfo file. The GET - request must be parametrized as specified in the HTTP protocol. The - following parameters must be present in the request: - - 'info_hash': This is a REQUIRED 20-byte SHA1 hash value. In order to - obtain this value the peer must calculate the SHA1 of the value of - the "info" key in the metainfo file. - - 'peer_id': This is a REQUIRED string and must contain the 20-byte - self-designated ID of the peer. - - 'port': The port number that the peer is listening to for incoming - connections from other peers. BTP/1.0 does not specify a standard - port number, nor a port range to be used. This key is REQUIRED. - - 'uploaded': This is a base ten integer value. It denotes the total - amount of bytes that the peer has uploaded in the swarm since it - sent the "started" event to the tracker. This key is REQUIRED. - - 'downloaded': This is a base ten integer value. It denotes the total - amount of bytes that the peer has downloaded in the swarm since it - sent the "started" event to the tracker. This key is REQUIRED. - - 'left': This is a base ten integer value. It denotes the total - amount of bytes that the peer needs in this torrent in order to - complete its download. This key is REQUIRED. - - 'ip': This is an OPTIONAL value, and if present should indicate the - true, Internet-wide address of the peer, either in dotted quad - IPv4 format, hexadecimal IPv6 format, or a DNS name. - - 'numwant': This is an OPTIONAL value. If present, it should indicate - the number of peers that the local peer wants to receive from the - tracker. If not present, the tracker uses an implementation - defined value. - - - -Fonseca, et al. [Page 13] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'event': This parameter is OPTIONAL. If not specified, the request - is taken to be a regular periodic request. Otherwise, it MUST - have one of the three following values: - - 'started': The first HTTP GET request sent to the tracker MUST - have this value in the "event" parameter. - - 'stopped': This value SHOULD be sent to the tracker when the peer - is shutting down gracefully. - - 'completed': This value SHOULD be sent to the tracker when the - peer completes a download. The peer SHOULD NOT send this value - if it started up with the complete torrent. - - -5.2 Response - - Upon receiving the HTTP GET request, the tracker MUST respond with a - document having the "text/plain" MIME type. This document MUST - contain a bencoded dictionary with the following keys: - - 'failure reason': This key is OPTIONAL. If present, the dictionary - MUST NOT contain any other keys. The peer should interpret this - as if the attempt to join the torrent failed. The value is a - human readable string containing an error message with the failure - reason. - - 'interval': A peer must send regular HTTP GET requests to the tracker - to obtain an updated list of peers and update the tracker of its - status. The value of this key indicated the amount of time that a - peer should wait between to consecutive regular requests. This - key is REQUIRED. - - 'complete': This is an integer that indicates the number of seeders. - This key is OPTIONAL. - - 'incomplete': This is an integer that indicates the number of peers - downloading the torrent. This key is OPTIONAL. - - 'peers': This is a bencoded list of dictionaries containing a list of - peers that must be contacted in order to download a file. This - key is REQUIRED. It has the following structure: - - 'peer id': This is a REQUIRED string value containing the self- - designated ID of the peer. - - - - - - -Fonseca, et al. [Page 14] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - 'ip': This is a REQUIRED string value indicating the IP address of - the peer. This may be given as a dotted quad IPv4 format, - hexadecimal IPv6 format or DNS name. - - 'port': This is an integer value. It must contain the self- - designated port number of the peer. This key is REQUIRED. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 15] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -6. The Peer Wire Protocol - - The aim of the PWP, is to facilitate communication between - neighboring peers for the purpose of sharing file content. PWP - describes the steps taken by a peer after it has read in a metainfo - file and contacted a tracker to gather information about other peers - it may communicate with. PWP is layered on top of TCP and handles - all its communication using asynchronous messages. - -6.1 Peer Wire Guidelines - - PWP does not specify a standard algorithm for selecting elements from - a clients neighboring peers with whom to share pieces, although the - following guidelines are expected to be observed by any such - algorithm: - - The algorithm should not be constructed with the goal in mind to - reduce the amount of data uploaded compared to downloaded. At the - very least a peer should upload the same amount that it has - downloaded. - - The algorithm should not use a strict tit-for-tat schema when - dealing with remote peers that have just joined the swarm and thus - have no pieces to offer. - - The algorithm should make good use of both download and upload - bandwidth by putting a cap on the number of simultaneous - connection that actively send or receive data. By reducing the - number of active connections, TCP congestion can be avoided. - - The algorithm should pipeline data requests in order so saturate - active connections. - - The algorithm should be able to cooperate with peers that - implement a different algorithm. - - -6.2 Handshaking - - The local peer opens a port on which to listen for incoming - connections from remote peers. This port is then reported to the - tracker. As BTP/1.0 does not specify any standard port for listening - it is the sole responsibility of the implementation to select a port. - - Any remote peer wishing to communicate with the local peer must open - a TCP connection to this port and perform a handshake operation. The - handshake operation MUST be carried out before any other data is sent - from the remote peer. The local peer MUST NOT send any data back to - - - -Fonseca, et al. [Page 16] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - the remote peer before a well constructed handshake has been - recognized according to the rules below. If the handshake in any way - violates these rules the local peer MUST close the connection with - the remote peer. - - - - - - A handshake is a string of bytes with the following structure: - - ---------------------------------------------------------------- - | Name Length | Protocol Name | Reserved | Info Hash | Peer ID | - ---------------------------------------------------------------- - - Name Length: The unsigned value of the first byte indicates the - length of a character string containing the protocol name. In - BTP/1.0 this number is 19. The local peer knows its own protocol - name and hence also the length of it. If this length is different - than the value of this first byte, then the connection MUST be - dropped. - - Protocol Name: This is a character string which MUST contain the - exact name of the protocol in ASCII and have the same length as - given in the Name Length field. The protocol name is used to - identify to the local peer which version of BTP the remote peer - uses. In BTP/1.0 the name is 'BitTorrent protocol'. If this - string is different from the local peers own protocol name, then - the connection is to be dropped. - - Reserved: The next 8 bytes in the string are reserved for future - extensions and should be read without interpretation. - - Info Hash: The next 20 bytes in the string are to be interpreted as a - 20-byte SHA1 of the info key in the metainfo file. Presumably, - since both the local and the remote peer contacted the tracker as - a result of reading in the same .torrent file, the local peer will - recognize the info hash value and will be able to serve the remote - peer. If this is not the case, then the connection MUST be - dropped. This situation can arise if the local peer decides to no - longer serve the file in question for some reason. The info hash - may be used to enable the client to serve multiple torrents on the - same port. - - At this stage, if the connection has not been dropped, then the - local peer MUST send its own handshake back, which includes the - last step: - - - - -Fonseca, et al. [Page 17] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - Peer ID: The last 20 bytes of the handshake are to be interpreted as - the self-designated name of the peer. The local peer must use - this name to identify the connection hereafter. Thus, if this - name matches the local peers own ID name, the connection MUST be - dropped. Also, if any other peer has already identified itself to - the local peer using that same peer ID, the connection MUST be - dropped. - - In BTP/1.0 the handshake has a total of 68 bytes. - -6.3 Message Communication - - Following the PWP handshake both ends of the TCP channel may send - messages to each other in a completely asynchronous fashion. PWP - messages have the dual purpose of updating the state of neighboring - peers with regard to changes in the local peer, as well as - transferring data blocks between neighboring peers. - - PWP Messages fall into two different categories: - - State-oriented messages: These messages serve the sole purpose of - informing peers of changes in the state of neighboring peers. A - message of this type MUST be sent whenever a change occurs in a - peer's state, regardless of the state of other peers. The - following messages fall into this category: Interested, - Uninterested, Choked, Unchoked, Have and Bitfield. - - Data-oriented messages: These messages handle the requesting and - sending of data portions. The following messages fall into this - category: Request, Cancel and Piece. - - -6.3.1 Peer States - - For each end of a connection, a peer must maintain the following two - state flags: - - Choked: When true, this flag means that the choked peer is not - allowed to request data. - - Interested: When true, this flag means a peer is interested in - requesting data from another peer. This indicates that the peer - will start requesting blocks if it is unchoked. - - A choked peer MUST not send any data-oriented messages, but is free - to send any other message to the peer that has choked it. If a peer - chokes a remote peer, it MUST also discard any unanswered requests - for blocks previously received from the remote peer. - - - -Fonseca, et al. [Page 18] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - An unchoked peer is allowed to send data-oriented messages to the - remote peer. It is left to the implementation how many peers any - given peer may choose to choke or unchoke, and in what fashion. This - is done deliberately to allow peers to use different heuristics for - peer selection. - - An interested peer indicates to the remote peer that it must expect - to receive data-oriented messages as soon as it unchokes the - interested peer. It must be noted, that a peer must not assume a - remote peer is interested solely because it has pieces that the - remote peer is lacking. There may be valid reasons why a peer is not - interested in another peer other than data-based ones. - -6.3.2 Peer Wire Messages - - All integer members in PWP messages are encoded as a 4-byte big- - endian number. Furthermore, all index and offset members in PWP - messages are zero-based. - - A PWP message has the following structure: - - ----------------------------------------- - | Message Length | Message ID | Payload | - ----------------------------------------- - - Message Length: This is an integer which denotes the length of the - message, excluding the length part itself. If a message has no - payload, its size is 1. Messages of size 0 MAY be sent - periodically as keep-alive messages. Apart from the limit that - the four bytes impose on the message length, BTP does not specify - a maximum limit on this value. Thus an implementation MAY choose - to specify a different limit, and for instance disconnect a remote - peer that wishes to communicate using a message length that would - put too much strain on the local peer's resources. - - Message ID: This is a one byte value, indicating the type of the - message. BTP/1.0 specifies 9 different messages, as can be seen - further below. - - Payload: The payload is a variable length stream of bytes. - - If an incoming message in any way violates this structure then the - connection SHOULD be dropped. In particular the receiver SHOULD make - sure the message ID constitutes a valid message, and the payload - matches the the expected payload, as given below. - - For the purpose of compatibility with future protocol extensions the - client SHOULD ignore unknown messages. There may arise situations in - - - -Fonseca, et al. [Page 19] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - which a client may choose to drop a connection after receiving an - unknown message, either for security reasons, or because discarding - large unknown messages may be viewed as excessive waste. - - BTP/1.0 specifies the following messages: - -6.3.3 Choke - - This message has ID 0 and no payload. A peer sends this message to a - remote peer to inform the remote peer that it is being choked. - -6.3.4 Unchoke - - This message has ID 1 and no payload. A peer sends this message to a - remote peer to inform the remote peer that it is no longer being - choked. - -6.3.5 Interested - - This message has ID 2 and no payload. A peer sends this message to a - remote peer to inform the remote peer of its desire to request data. - -6.3.6 Uninterested - - This message has ID 3 and no payload. A peer sends this message to a - remote peer to inform it that it is not interested in any pieces from - the remote peer. - -6.3.7 Have - - This message has ID 4 and a payload of length 4. The payload is a - number denoting the index of a piece that the peer has successfully - downloaded and validated. A peer receiving this message must - validate the index and drop the connection if this index is not - within the expected bounds. Also, a peer receiving this message MUST - send an interested message to the sender if indeed it lacks the piece - announced. Further, it MAY also send a request for that piece. - -6.3.8 Bitfield - - This message has ID 5 and a variable payload length. The payload is - a bitfield representing the pieces that the sender has successfully - downloaded, with the high bit in the first byte corresponding to - piece index 0. If a bit is cleared it is to be interpreted as a - missing piece. A peer MUST send this message immediately after the - handshake operation, and MAY choose not to send it if it has no - pieces at all. This message MUST not be sent at any other time - during the communication. - - - -Fonseca, et al. [Page 20] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -6.3.9 Request - - This message has ID 6 and a payload of length 12. The payload is 3 - integer values indicating a block within a piece that the sender is - interested in downloading from the recipient. The recipient MUST - only send piece messages to a sender that has already requested it, - and only in accordance to the rules given above about the choke and - interested states. The payload has the following structure: - - --------------------------------------------- - | Piece Index | Block Offset | Block Length | - --------------------------------------------- - - -6.3.10 Piece - - This message has ID 7 and a variable length payload. The payload - holds 2 integers indicating from which piece and with what offset the - block data in the 3rd member is derived. Note, the data length is - implicit and can be calculated by subtracting 9 from the total - message length. The payload has the following structure: - - ------------------------------------------- - | Piece Index | Block Offset | Block Data | - ------------------------------------------- - - -6.3.11 Cancel - - This message has ID 8 and a payload of length 12. The payload is 3 - integer values indicating a block within a piece that the sender has - requested for, but is no longer interested in. The recipient MUST - erase the request information upon receiving this messages. The - payload has the following structure: - - --------------------------------------------- - | Piece Index | Block Offset | Block Length | - --------------------------------------------- - - -6.4 The End Game - - Towards the end of a download session, it may speed up the download - to send request messages for the remaining blocks to all the - neighboring peers. A client must issue cancel messages to all - pending requests sent to neighboring peers as soon as a piece is - downloaded successfully. This is referred to as the end game. - - - - -Fonseca, et al. [Page 21] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - A client usually sends requests for blocks in stages; sending - requests for newer blocks as replies for earlier requests are - received. The client enters the end game, when all remaining blocks - have been requested. - -6.5 Piece Selection Strategy - - BTP/1.0 does not force a particular order for selecting which pieces - to download. However, experience shows that downloading in rarest- - first order seems to lessen the wait time for pieces. To find the - rarest piece a client must calculate for each piece index the number - of times this index is true in the bitfield vectors of all the - neighboring peers. The piece with the lowest sum is then selected - for requesting. - -6.6 Peer Selection Strategy - - This section describes the choking algorithm recommended for - selecting neighboring peers with whom to exchange pieces. - Implementations are free to implement any strategy as long as the - guidelines in Section 6.1 are observed. - - After the initial handshake both ends of a connection set the Choked - flag to true and the Interested flag to false. - - All connections are periodically rated in terms of their ability to - provide the client with a better download rate. The rating may take - into account factors such as the remote peers willingness to maintain - an unchoked connection with the client over a certain period of time, - the remote peers upload rate to the client and other implementation - defined criteria. - - The peers are sorted according to their rating with regard to the - above mentioned scheme. Assume only 5 peers are allowed to download - at the same time. The peer selection algorithm will now unchoke as - many of the best rated peers as necessary so that exactly 5 of these - are interested. If one of the top rated peers at a later stage - becomes interested, then the peer selection algorithm will choke the - the worst unchoked peer. Notice that the worst unchoked peer is - always interested. - - The only lacking element from the above algorithm is the capability - to ensure that new peers can have a fair chance of downloading a - piece, even though they would evaluate poorly in the above schema. A - simple method is to make sure that a random peer is selected - periodically regardless of how it evaluates. Since this process is - repeated in a round robin manner, it ensures that ultimately even new - peers will have a chance of being unchoked. - - - -Fonseca, et al. [Page 22] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -7. Security Consideration - - This section examines security considerations for BTP/1.0.The - discussion does not include definitive solutions to the problems - revealed, though it does make some suggestions for reducing security - risks. - -7.1 Tracker HTTP Protocol Issues - - The use of the HTTP protocol for communication between the tracker - and the client makes BTP/1.0 vulnerable to the attacks mentioned in - the security consideration section of RFC 2616 [6]. - -7.2 Denial of Service Attacks on Trackers - - The nature of the tracker is to serve many clients. By mounting a - denial of service attack against the tracker the swarm attached to - the tracker can be starved. This type of attack is hard to defend - against, however, the metainfo file allows for multiple trackers to - be specified, making it possible to spread the load on a number of - trackers, and thus containing such an attack. - -7.3 Peer Identity Issues - - There is no strong authentication of clients when they contact the - tracker. The main option for trackers is to check peer ID and the IP - address of the client. The lack of authentication can be used to - mount an attack where a client can shut down another client if the - two clients are running on the same host and thus are sharing the - same IP address. In addition, a rogue peer may masquerade its - identity by using multiple peer IDs. Clients should there refrain - from taking the peer ID at face value. - -7.4 DNS Spoofing - - Clients using BTP/1.0 rely heavily on the Domain Name Service, which - can be used for both specifying the URI of the tracker and how to - contact a peer. Clients are thus generally prone to security attacks - based on the deliberate mis-association of IP addresses and DNS - names. Clients need to be cautious in assuming the continuing - validity of an IP address/DNS name association. - - In particular, BTP/1.0 clients SHOULD rely on their name resolver for - confirmation of an IP number/DNS name association, rather than - caching the result of previous host name lookups. If clients cache - the results of host name lookups in order to achieve a performance - improvement, they MUST observe the TTL information reported by DNS. - - - - -Fonseca, et al. [Page 23] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - If clients do not observe this rule, they could be spoofed when a - previously-accessed peers or trackers IP address changes. As network - renumbering is expected to become increasingly common according to - RFC 1900 [2], the possibility of this form of attack will grow. - Observing this requirement reduces this potential security - vulnerability. - -7.5 Issues with File and Directory Names - - The metainfo file provides a way to suggest a name of the downloaded - file for single-file torrents and the top-most directory for multi- - file torrents. This functionality is very like the Content- - Disposition header field documented in RFC 2183 [4] and the security - considerations mentioned in this RFC also apply to BitTorrent - clients. In short, BTP clients SHOULD verify that the suggested file - names in the metainfo file do not compromise services on the local - system. Furthermore, care must be taken for multi-file torrents to - validate that individual files are relative to the top-most directory - and that the paths do not contain path elements to the parent (that - is directory elevators such as ``..''), which can be used to place - files outside the top-most directory. - - Using UNIX as an example, some hazards would be: - - o Creating startup files (e.g., ".login"). - - o Creating or overwriting system files (e.g., "/etc/passwd"). - - o Overwriting any existing file. - - o Placing executable files into any command search path (e.g., - "~/bin/more"). - - o Sending the file to a pipe (e.g., "| sh"). - - It is very important to note that this is not an exhaustive list; it - is intended as a small set of examples only. Implementers must be - alert to the potential hazards on their target systems. In general, - the BTP client SHOULD NOT name or place files such that they will get - interpreted or executed without the user explicitly initiating the - action. - -7.6 Validating the Integrity of Data Exchanged Between Peers - - By default, all content served to the client from other peers should - be considered tainted and the client SHOULD validate the integrity of - the data before accepting it. The metainfo file contains information - for checking both individual pieces using SHA1, and optionally - - - -Fonseca, et al. [Page 24] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - - individual files using MD5. SHA1, being the strongest of the two, is - preferred. Furthermore, sole reliance on whole-file checking can - potentially render otherwise valid pieces invalid, and should only be - considered for small files, to limit the amount of data being - discarded. - - Trusting the validity of the resulting file or files ends up being a - matter of trusting the content of the metainfo file. Ensuring the - validity of the metainfo file is beyond the scope of this document. - -7.7 Transfer of Sensitive Information - - Some clients include information about themselves when generating the - peer ID string. Clients should be aware that this information can - potentially be used to determine whether a specific client has a - exploitable security hole. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Fonseca, et al. [Page 25] - -BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005 - - -8. IANA Considerations - - This document makes no request of IANA. - -9. References - - [1] Postel, J. and J. Reynolds, "File Transfer Protocol", STD 9, - RFC 959, October 1985. - - [2] Carpenter, B. and Y. Rekhter, "Renumbering Needs Work", - RFC 1900, February 1996. - - [3] Bradner, S., "Key words for use in RFCs to Indicate Requirement - Levels", BCP 14, RFC 2119, March 1997. - - [4] Troost, R., Dorner, S., and K. Moore, "Communicating - Presentation Information in Internet Messages: The Content- - Disposition Header Field", RFC 2183, August 1997. - - [5] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax - Specifications: ABNF", RFC 2234, November 1997. - - [6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., - Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- - HTTP/1.1", RFC 2616, June 1999. - - -Authors' Addresses - - Jonas Fonseca - DIKU - - Email: fonseca@diku.dk - - - Basim Reza - DIKU - - Email: basim@diku.dk - - - Lilja Fjeldsted - DIKU - - Email: lilja@diku.dk - - - - - - -Fonseca, et al. [Page 26] - diff --git a/doc/links.txt b/doc/links.txt deleted file mode 100644 index 55902ef..0000000 --- a/doc/links.txt +++ /dev/null @@ -1,11 +0,0 @@ -http://jonas.nitro.dk/bittorrent/bittorrent-rfc.html - -http://bittorrent.org/beps/bep_0003.html -http://bittorrent.org/beps/bep_0005.html - -https://wiki.theory.org/BitTorrentSpecification -http://wiki.vuze.com/w/Azureus_messaging_protocol - -Some random overview -http://www.kristenwidman.com/blog/33/how-to-write-a-bittorrent-client-part-1/ -http://www.kristenwidman.com/blog/71/how-to-write-a-bittorrent-client-part-2/ diff --git a/doc/out.txt b/doc/out.txt deleted file mode 100644 index 82bf6de..0000000 --- a/doc/out.txt +++ /dev/null @@ -1,65 +0,0 @@ -metainfo file.: OpenBSD_songs_ogg-2016-03-25-0127.torrent -info hash.....: 1aa5af9f6f533961a65169bdeeb801e611724d32 - -directory name: OpenBSD_songs_ogg -files.........: 38 - OpenBSD_songs_ogg/song30.ogg (2636244) 2.51 MB - OpenBSD_songs_ogg/song31.ogg (2381018) 2.27 MB - OpenBSD_songs_ogg/song32.ogg (2253880) 2.15 MB - OpenBSD_songs_ogg/song33.ogg (3238559) 3.09 MB - OpenBSD_songs_ogg/song34.ogg (5062603) 4.83 MB - OpenBSD_songs_ogg/song35.ogg (7191264) 6.86 MB - OpenBSD_songs_ogg/song36.ogg (5197630) 4.96 MB - OpenBSD_songs_ogg/song37.ogg (13937255) 13.29 MB - OpenBSD_songs_ogg/song38.ogg (5860681) 5.59 MB - OpenBSD_songs_ogg/song38b.ogg (5759105) 5.49 MB - OpenBSD_songs_ogg/song39.ogg (5954997) 5.68 MB - OpenBSD_songs_ogg/song40.ogg (3610740) 3.44 MB - OpenBSD_songs_ogg/song41.ogg (8382781) 7.99 MB - OpenBSD_songs_ogg/song42.ogg (6713047) 6.40 MB - OpenBSD_songs_ogg/song43.ogg (6834505) 6.52 MB - OpenBSD_songs_ogg/song44.ogg (4628329) 4.41 MB - OpenBSD_songs_ogg/song45.ogg (4737475) 4.52 MB - OpenBSD_songs_ogg/song46.ogg (3766867) 3.59 MB - OpenBSD_songs_ogg/song47.ogg (6586251) 6.28 MB - OpenBSD_songs_ogg/song48.ogg (3143847) 3.00 MB - OpenBSD_songs_ogg/song49.ogg (6006834) 5.73 MB - OpenBSD_songs_ogg/song50.ogg (4184951) 3.99 MB - OpenBSD_songs_ogg/song51.ogg (4216946) 4.02 MB - OpenBSD_songs_ogg/song52.ogg (4283099) 4.08 MB - OpenBSD_songs_ogg/song53.ogg (4635216) 4.42 MB - OpenBSD_songs_ogg/song54.ogg (3195265) 3.05 MB - OpenBSD_songs_ogg/song55.ogg (6160151) 5.87 MB - OpenBSD_songs_ogg/song56.ogg (5471902) 5.22 MB - OpenBSD_songs_ogg/song57.ogg (4057801) 3.87 MB - OpenBSD_songs_ogg/song58a.ogg (3252520) 3.10 MB - OpenBSD_songs_ogg/song58b.ogg (4376163) 4.17 MB - OpenBSD_songs_ogg/song58c.ogg (3595317) 3.43 MB - OpenBSD_songs_ogg/song58d.ogg (7013091) 6.69 MB - OpenBSD_songs_ogg/song59a.ogg (5766764) 5.50 MB - OpenBSD_songs_ogg/song59b.ogg (5317415) 5.07 MB - OpenBSD_songs_ogg/songsh.ogg (4890381) 4.66 MB - OpenBSD_songs_ogg/songsi.ogg (5694670) 5.43 MB - OpenBSD_songs_ogg/songty.ogg (6045645) 5.77 MB -archive size..: 196041209 (747 * 262144 + 219641) 186.96 MB - -announce url..: http://OpenBSD.somedomain.net:6969/announce -creation date.: Fri Mar 25 02:27:56 2016 -created by....: mktorrent 1.0 -comment.......: ogg files from OpenBSD/songs -Created by andrew fresh (andrew@afresh1.com) -http://OpenBSD.somedomain.net/ - - -metainfo file.: OpenBSD_5.9_amd64_install59.iso-2016-03-29-0449.torrent -info hash.....: e840038dea1998c39614dcd28594501df02bd32d - -file name.....: OpenBSD_5.9_amd64_install59.iso -file size.....: 233662464 (891 * 262144 + 92160) 222.84 MB - -announce url..: http://OpenBSD.somedomain.net:6969/announce -creation date.: Tue Mar 29 06:49:18 2016 -created by....: mktorrent 1.0 -comment.......: OpenBSD/5.9/amd64/install59.iso -Created by andrew fresh (andrew@afresh1.com) -http://OpenBSD.somedomain.net/ -- cgit v1.2.3