aboutsummaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
authorDimitri Sokolyuk <demon@dim13.org>2018-12-15 14:43:12 +0100
committerDimitri Sokolyuk <demon@dim13.org>2018-12-15 14:43:12 +0100
commit38fc10d3858be4a4161775a52e9bbb5b97c23d7e (patch)
tree3d708cad162df89a105bfc08a91241db9b30575b /doc
parent74f1a9e6a121b4699425bfad4b9f6ec0ba7bec61 (diff)
rename docs
Diffstat (limited to 'doc')
-rw-r--r--doc/bittorrent-rfc.txt1456
-rw-r--r--doc/links.txt11
-rw-r--r--doc/out.txt65
3 files changed, 0 insertions, 1532 deletions
diff --git a/doc/bittorrent-rfc.txt b/doc/bittorrent-rfc.txt
deleted file mode 100644
index 38fc8ac..0000000
--- a/doc/bittorrent-rfc.txt
+++ /dev/null
@@ -1,1456 +0,0 @@
-
-
-
-BitTorrent Protocol version 1.0 J. Fonseca
-$Revision: 1.33 $ B. Reza
- L. Fjeldsted
- DIKU
- April 2005
-
-
- BitTorrent Protocol -- BTP/1.0
-
-
-Abstract
-
- This document describes the BitTorrent Protocol version 1.0 referred
- to as "BTP/1.0". The BitTorrent Protocol (BTP) is a protocol for
- collaborative file distribution across the Internet and has been in
- place on the Internet since 2002. It is best classified as a peer-
- to-peer (P2P) protocol, although it also contains highly centralized
- elements. BTP has already been implemented many times for different
- platforms, and could well be said to be a mature protocol, although a
- formal, detailed and complete description has so far been lacking.
-
- BTP was devised and implemented by Bram Cohen as a P2P replacement to
- standard File Transfer Protocol (FTP) to be used in places where the
- usage of an FTP implementation poses too much strain on the server in
- terms of request-processing and sheer bandwidth. Normally a client
- does not use her upload capacity while downloading a file. The BTP
- approach capitalizes on this fact by having clients upload bits of
- the data to each other. In comparison to FTP this adds huge
- scalability and cost-management advantages.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 1]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-Table of Contents
-
- 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 4
- 1.1 Extensions . . . . . . . . . . . . . . . . . . . . . . . . 4
- 1.2 Audience . . . . . . . . . . . . . . . . . . . . . . . . . 4
- 1.3 Terminology . . . . . . . . . . . . . . . . . . . . . . . 5
- 1.4 Overall Operation . . . . . . . . . . . . . . . . . . . . 5
- 2. Bencoding . . . . . . . . . . . . . . . . . . . . . . . . 7
- 2.1 Scalar Types . . . . . . . . . . . . . . . . . . . . . . . 7
- 2.2 Compound Types . . . . . . . . . . . . . . . . . . . . . . 7
- 3. Pieces and Blocks . . . . . . . . . . . . . . . . . . . . 8
- 3.1 Pieces . . . . . . . . . . . . . . . . . . . . . . . . . . 8
- 3.2 Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . 8
- 4. The Metainfo File . . . . . . . . . . . . . . . . . . . . 10
- 4.1 The Structure of the Metainfo File . . . . . . . . . . . . 10
- 4.1.1 Single File Torrents . . . . . . . . . . . . . . . . . . . 10
- 4.1.2 Multi File Torrents . . . . . . . . . . . . . . . . . . . 11
- 5. The Tracker HTTP Protocol . . . . . . . . . . . . . . . . 13
- 5.1 Request . . . . . . . . . . . . . . . . . . . . . . . . . 13
- 5.2 Response . . . . . . . . . . . . . . . . . . . . . . . . . 14
- 6. The Peer Wire Protocol . . . . . . . . . . . . . . . . . . 16
- 6.1 Peer Wire Guidelines . . . . . . . . . . . . . . . . . . . 16
- 6.2 Handshaking . . . . . . . . . . . . . . . . . . . . . . . 16
- 6.3 Message Communication . . . . . . . . . . . . . . . . . . 18
- 6.3.1 Peer States . . . . . . . . . . . . . . . . . . . . . . . 18
- 6.3.2 Peer Wire Messages . . . . . . . . . . . . . . . . . . . . 19
- 6.3.3 Choke . . . . . . . . . . . . . . . . . . . . . . . . . . 20
- 6.3.4 Unchoke . . . . . . . . . . . . . . . . . . . . . . . . . 20
- 6.3.5 Interested . . . . . . . . . . . . . . . . . . . . . . . . 20
- 6.3.6 Uninterested . . . . . . . . . . . . . . . . . . . . . . . 20
- 6.3.7 Have . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
- 6.3.8 Bitfield . . . . . . . . . . . . . . . . . . . . . . . . . 20
- 6.3.9 Request . . . . . . . . . . . . . . . . . . . . . . . . . 21
- 6.3.10 Piece . . . . . . . . . . . . . . . . . . . . . . . . . . 21
- 6.3.11 Cancel . . . . . . . . . . . . . . . . . . . . . . . . . . 21
- 6.4 The End Game . . . . . . . . . . . . . . . . . . . . . . . 21
- 6.5 Piece Selection Strategy . . . . . . . . . . . . . . . . . 22
- 6.6 Peer Selection Strategy . . . . . . . . . . . . . . . . . 22
- 7. Security Consideration . . . . . . . . . . . . . . . . . . 23
- 7.1 Tracker HTTP Protocol Issues . . . . . . . . . . . . . . . 23
- 7.2 Denial of Service Attacks on Trackers . . . . . . . . . . 23
- 7.3 Peer Identity Issues . . . . . . . . . . . . . . . . . . . 23
- 7.4 DNS Spoofing . . . . . . . . . . . . . . . . . . . . . . . 23
- 7.5 Issues with File and Directory Names . . . . . . . . . . . 24
- 7.6 Validating the Integrity of Data Exchanged Between Peers . 24
- 7.7 Transfer of Sensitive Information . . . . . . . . . . . . 25
- 8. IANA Considerations . . . . . . . . . . . . . . . . . . . 26
- 9. References . . . . . . . . . . . . . . . . . . . . . . . . 26
-
-
-
-Fonseca, et al. [Page 2]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- Authors' Addresses . . . . . . . . . . . . . . . . . . . . 26
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 3]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-1. Introduction
-
- The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in RFC 2119 [3].
-
- The File Transfer Protocol (FTP) [1] with the recent additions of
- security extensions, still remains the standard for secure and
- reliable transmission of large files over the Internet. However, its
- highly centralized client-server approach also means, it is
- inadequate for mass publication of files, where a single point may
- expect to be requested by a critically large number of clients
- simultaneously. To remedy this situation many organizations either
- implement a cap on the number of simultaneous requests, or spread the
- load on multiple mirror servers. Needless to say both approaches
- have their drawbacks, and a solution that addresses these problems is
- highly needed.
-
- The approach in BitTorrent Protocol (BTP) is to spread the load not
- on mirror servers, but to the clients themselves by having them
- upload bits of the file to each other while downloading it. Since
- the clients usually do not utilize their upload capacity while
- fetching a file, this approach does not put the clients in any
- disadvantage. This has the added advantage that even small
- organizations with limited resources can publish large files on the
- Internet without having to invest in costly infrastructure.
-
-1.1 Extensions
-
- Since the introduction of BTP many modifications and extensions have
- been proposed by individuals and community forums. To the extent
- that these extensions have become part of what the BitTorrent
- community considers best practice they have been included in this
- document. However, many extensions have been omitted either because
- they have been deemed to lack interoperability with existing
- implementations, or because they are not regarded as being
- sufficiently mature.
-
-1.2 Audience
-
- This document is aimed at developers who wish to implement BTP for a
- particular platform. Also, system administrators and architects may
- use this document to fully understand the implications of installing
- an implementation of BTP. In particular, it is advised to study the
- security implications in more detail, before installing an
- implementation on a machine that also contains sensitive data.
- Security implications are discussed in Section 7.
-
-
-
-
-Fonseca, et al. [Page 4]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-1.3 Terminology
-
- Peer: A peer is a node in a network participating in file sharing.
- It can simultaneously act both as a server and a client to other
- nodes on the network.
-
- Neighboring peers: Peers to which a client has an active point to
- point TCP connection.
-
- Client: A client is a user agent (UA) that acts as a peer on behalf
- of a user.
-
- Torrent: A torrent is the term for the file (single-file torrent) or
- group of files (multi-file torrent) the client is downloading.
-
- Swarm: A network of peers that actively operate on a given torrent.
-
- Seeder: A peer that has a complete copy of a torrent.
-
- Tracker: A tracker is a centralized server that holds information
- about one or more torrents and associated swarms. It functions as
- a gateway for peers into a swarm.
-
- Metainfo file: A text file that holds information about the torrent,
- e.g. the URL of the tracker. It usually has the extension
- .torrent.
-
- Peer ID: A 20-byte string that identifies the peer. How the peer ID
- is obtained is outside the scope of this document, but a peer must
- make sure that the peer ID it uses has a very high probability of
- being unique in the swarm.
-
- Info hash: A SHA1 hash that uniquely identifies the torrent. It is
- calculated from data in the metainfo file.
-
-
-1.4 Overall Operation
-
- BTP consists of two logically distinct protocols, namely the Tracker
- HTTP Protocol (THP), and the Peer Wire Protocol (PWP). THP defines a
- method for contacting a tracker for the purposes of joining a swarm,
- reporting progress etc. PWP defines a mechanism for communication
- between peers, and is thus responsible for carrying out the actual
- download and upload of the torrent.
-
- In order for a client to download a torrent the following steps must
- be carried through:
-
-
-
-
-Fonseca, et al. [Page 5]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- 1. A metainfo file must be retrieved.
-
- 2. Instructions that will allow the client to contact other peers
- must be periodically requested from the tracker using THP.
-
- 3. The torrent must be downloaded by connecting to peers in the
- swarm and trading pieces using PWP.
-
- To publish a torrent the following steps must be taken:
-
- 1. A tracker must be set up.
-
- 2. A metainfo file pointing to the tracker and containing
- information on the structure of the torrent must be produced and
- published.
-
- 3. At least one seeder with access to the complete torrent must be
- set up.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 6]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-2. Bencoding
-
- Bencoding encodes data in a platform independent way. In BTP/1.0 the
- metainfo file and all responses from the tracker are encoded in the
- bencoding format. The format specifies two scalar types (integers
- and strings) and two compound types (lists and dictionaries).
-
- The Augmented BNF syntax [5] for bencoding format is:
-
- dictionary = "d" 1*(string anytype) "e" ; non-empty dictionary
- list = "l" 1*anytype "e" ; non-empty list
- integer = "i" signumber "e"
- string = number ":" <number long sequence of any CHAR>
- anytype = dictionary / list / integer / string
- signumber = "-" number / number
- number = 1*DIGIT
- CHAR = %x00-FF ; any 8-bit character
- DIGIT = "0" / "1" / "2" / "3" / "4" /
- "5" / "6" / "7" / "8" / "9"
-
-2.1 Scalar Types
-
- Integers are encoded by prefixing a string containing the base ten
- representation of the integer with the letter "i" and postfixing it
- with the letter "e". E.g. the integer 123 is encoded as i123e.
-
- Strings are encoded by prefixing the string content with the length
- of the string followed by a colon. E.g. the string "announce" is
- encoded as "8:announce".
-
-2.2 Compound Types
-
- The compound types provides a mean to structure elements of any
- bencoding type.
-
- Lists are an arbitrary number of bencoded elements prefixed with the
- letter "l" and postfixed with the letter "e". It follows that lists
- can contain nested lists and dictionaries. For instance "li2e3:fooe"
- defines a list containing the integer "2" and the string "foo".
-
- Dictionaries are an arbitrary number of key/value pairs delimited by
- the letter "d" at the beginning and the letter "e" at the end. All
- keys are bencoded strings while the associated value can be any
- bencoded element. E.g. "d5:monthi4e4:name5:aprile" defines a
- dictionary holding the associations: "month" => "4" and "name" =>
- "april". All dictionary keys MUST be sorted.
-
-
-
-
-
-Fonseca, et al. [Page 7]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-3. Pieces and Blocks
-
- This section describes how a torrent is organized in pieces and
- blocks. The torrent is divided into one or more pieces. Each piece
- represents a range of data which it is possible to verify using a
- piece SHA1 hash. When distributing data over PWP pieces are divided
- into one or more blocks, as shown in the following diagram:
-
- ---------------------------------------
- | Piece #0 | Piece #1 | .. | Piece #N |
- ---------------------------------------
- _-' `-_
- _-' `-_
- ----------------------------
- | Block #0 | .. | Block #M |
- ----------------------------
-
-3.1 Pieces
-
- The number of pieces in the torrent is indicated in the metainfo
- file. The size of each piece in the torrent remains fixed and can be
- calculated using the following formula:
-
- fixed_piece_size = size_of_torrent / number_of_pieces
-
- where "/" is the integer division operator. Only the last piece of
- the torrent is allowed to have fewer bytes than the fixed piece size.
-
- The size of a piece is determined by the publisher of the torrent. A
- good recommendation is to use a piece size so that the metainfo file
- does not exceed 70 kilobytes.
-
- For the sake of calculating the correct position of a piece within a
- file, or files, the torrent is regarded as a single continuous byte
- stream. In case the torrent consists of multiple files, it is to be
- viewed as the concatenation of these files in the order of their
- appearance in the metainfo file. Conceptually, the torrent is only
- translated into files when all its pieces have been downloaded and
- verified using their respective SHA1 values; although in practice an
- implementation may choose a better approach in accordance with local
- operating system and filesystem specific demands.
-
-3.2 Blocks
-
- The size of a block is an implementation defined value that is not
- dependant on the fixed piece size. Once a fixed size is defined, the
- number of blocks per piece can be calculated using the formula:
-
-
-
-
-Fonseca, et al. [Page 8]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- number_of_blocks = (fixed_piece_size / fixed_block_size)
- + !!(fixed_piece_size % fixed_block_size)
-
- where "%" denotes the modulus operator, and "!" the negation
- operator. The negation operator is used to ensure that the last
- factor only adds a value of 0 or 1 to the sum. Given the start
- offset of the block its index within a piece can be calculated using
- the formula:
-
- block_index = block_offset % fixed_block_size
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 9]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-4. The Metainfo File
-
- The metainfo file provides the client with information on the tracker
- location as well as the torrent to be downloaded. Besides listing
- which files will result from downloading the torrent, it also lists
- how the client should split up and verify individual pieces making up
- the complete torrent.
-
- In order for a client to recognize the metainfo file it SHOULD have
- the extension .torrent and the associated the media type
- "application/x-bittorrent". How the client retrieves the metainfo
- file is beyond the scope of this document, however, the most user-
- friendly approach is for a client to find the file on a web page,
- click on it, and start the download immediately. This way, the
- apparent complexity of BTP as opposed to FTP or HTTP transfer is
- transparent to the user.
-
-4.1 The Structure of the Metainfo File
-
- The metainfo file contains a bencoded dictionary with the following
- structure. A key below is REQUIRED unless otherwise noted.
-
- 'announce': This is a string value. It contains the announce URL of
- the tracker.
-
- 'announce-list': This is an OPTIONAL list of string values. Each
- value is a URL pointing to a backup tracker. This value is not
- used in BTP/1.0.
-
- 'comment': This is an OPTIONAL string value that may contain any
- comment by the author of the torrent.
-
- 'created by': This is an optional string value and may contain the
- name and version of the program used to create the metainfo file.
-
- 'creation date': This is an OPTIONAL string value. It contains the
- creation time of the torrent in standard Unix epoch format.
-
- 'info': This key points to a dictionary that contains information
- about the files to download. The entries are explained in the
- following sections.
-
-
-4.1.1 Single File Torrents
-
- If the torrent only specifies one file, the info dictionary must have
- the following keys:
-
-
-
-
-Fonseca, et al. [Page 10]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- 'length': This is an integer value indicating the length of the file
- in bytes.
-
- 'md5sum': This is an OPTIONAL value. If included it must be a string
- of 32 characters corresponding to the MD5 sum of the file. This
- value is not used in BTP/1.0.
-
- 'name': A string containing the name of the file.
-
- 'piece length': An integer indicating the number of bytes in each
- piece.
-
- 'pieces': This is a string value containing the concatenation of the
- 20-byte SHA1 hash value for all pieces in the torrent. For
- example, the first 20 bytes of the string represent the SHA1 value
- used to verify piece index 0.
-
- The complete file is derived by combining all the pieces into one
- string of bytes.
-
-4.1.2 Multi File Torrents
-
- If the torrent specifies multiple files, the info dictionary must
- have the following structure:
-
- 'files': This is a list of dictionaries. Each file in the torrent
- has a dictionary associated to it having the following structure:
-
- 'length': This is an integer indicating the total length of the
- file in bytes.
-
- 'md5sum': This is an OPTIONAL value. if included it must be a
- string of 32 characters corresponding to the MD5 sum of the
- file. This value is not used in BTP/1.0.
-
- 'path': This is a list of string elements that specify the path of
- the file, relative to the topmost directory. The last element
- in the list is the name of the file, and the elements preceding
- it indicate the directory hierarchy in which this file is
- situated.
-
- 'name': This is a string value. It contains the name of the top-most
- directory in the file structure.
-
- 'piece length': This is an integer value. It contains the number of
- bytes in each piece.
-
-
-
-
-
-Fonseca, et al. [Page 11]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- 'pieces': This is a string value. It must contain the concatenation
- of all 20-byte SHA1 hash values that are used by BTP/1.0 to verify
- each downloaded piece. The first 20 bytes of the string represent
- the SHA1 value used to verify piece index 0.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 12]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-5. The Tracker HTTP Protocol
-
- The Tracker HTTP Protocol (THP) is a simple mechanism for introducing
- peers to each other. A tracker is a HTTP service that must be
- contacted by a peer in order to join a swarm. As such the tracker
- constitutes the only centralized element in BTP/1.0. A tracker does
- not by itself provide access to any downloadable data. A tracker
- relies on peers sending regular requests. It may assume that a peer
- is dead if it misses a request.
-
-5.1 Request
-
- To contact the tracker a peer MUST send a standard HTTP GET request
- using the URL in the "announce" entry of the metainfo file. The GET
- request must be parametrized as specified in the HTTP protocol. The
- following parameters must be present in the request:
-
- 'info_hash': This is a REQUIRED 20-byte SHA1 hash value. In order to
- obtain this value the peer must calculate the SHA1 of the value of
- the "info" key in the metainfo file.
-
- 'peer_id': This is a REQUIRED string and must contain the 20-byte
- self-designated ID of the peer.
-
- 'port': The port number that the peer is listening to for incoming
- connections from other peers. BTP/1.0 does not specify a standard
- port number, nor a port range to be used. This key is REQUIRED.
-
- 'uploaded': This is a base ten integer value. It denotes the total
- amount of bytes that the peer has uploaded in the swarm since it
- sent the "started" event to the tracker. This key is REQUIRED.
-
- 'downloaded': This is a base ten integer value. It denotes the total
- amount of bytes that the peer has downloaded in the swarm since it
- sent the "started" event to the tracker. This key is REQUIRED.
-
- 'left': This is a base ten integer value. It denotes the total
- amount of bytes that the peer needs in this torrent in order to
- complete its download. This key is REQUIRED.
-
- 'ip': This is an OPTIONAL value, and if present should indicate the
- true, Internet-wide address of the peer, either in dotted quad
- IPv4 format, hexadecimal IPv6 format, or a DNS name.
-
- 'numwant': This is an OPTIONAL value. If present, it should indicate
- the number of peers that the local peer wants to receive from the
- tracker. If not present, the tracker uses an implementation
- defined value.
-
-
-
-Fonseca, et al. [Page 13]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- 'event': This parameter is OPTIONAL. If not specified, the request
- is taken to be a regular periodic request. Otherwise, it MUST
- have one of the three following values:
-
- 'started': The first HTTP GET request sent to the tracker MUST
- have this value in the "event" parameter.
-
- 'stopped': This value SHOULD be sent to the tracker when the peer
- is shutting down gracefully.
-
- 'completed': This value SHOULD be sent to the tracker when the
- peer completes a download. The peer SHOULD NOT send this value
- if it started up with the complete torrent.
-
-
-5.2 Response
-
- Upon receiving the HTTP GET request, the tracker MUST respond with a
- document having the "text/plain" MIME type. This document MUST
- contain a bencoded dictionary with the following keys:
-
- 'failure reason': This key is OPTIONAL. If present, the dictionary
- MUST NOT contain any other keys. The peer should interpret this
- as if the attempt to join the torrent failed. The value is a
- human readable string containing an error message with the failure
- reason.
-
- 'interval': A peer must send regular HTTP GET requests to the tracker
- to obtain an updated list of peers and update the tracker of its
- status. The value of this key indicated the amount of time that a
- peer should wait between to consecutive regular requests. This
- key is REQUIRED.
-
- 'complete': This is an integer that indicates the number of seeders.
- This key is OPTIONAL.
-
- 'incomplete': This is an integer that indicates the number of peers
- downloading the torrent. This key is OPTIONAL.
-
- 'peers': This is a bencoded list of dictionaries containing a list of
- peers that must be contacted in order to download a file. This
- key is REQUIRED. It has the following structure:
-
- 'peer id': This is a REQUIRED string value containing the self-
- designated ID of the peer.
-
-
-
-
-
-
-Fonseca, et al. [Page 14]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- 'ip': This is a REQUIRED string value indicating the IP address of
- the peer. This may be given as a dotted quad IPv4 format,
- hexadecimal IPv6 format or DNS name.
-
- 'port': This is an integer value. It must contain the self-
- designated port number of the peer. This key is REQUIRED.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 15]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-6. The Peer Wire Protocol
-
- The aim of the PWP, is to facilitate communication between
- neighboring peers for the purpose of sharing file content. PWP
- describes the steps taken by a peer after it has read in a metainfo
- file and contacted a tracker to gather information about other peers
- it may communicate with. PWP is layered on top of TCP and handles
- all its communication using asynchronous messages.
-
-6.1 Peer Wire Guidelines
-
- PWP does not specify a standard algorithm for selecting elements from
- a clients neighboring peers with whom to share pieces, although the
- following guidelines are expected to be observed by any such
- algorithm:
-
- The algorithm should not be constructed with the goal in mind to
- reduce the amount of data uploaded compared to downloaded. At the
- very least a peer should upload the same amount that it has
- downloaded.
-
- The algorithm should not use a strict tit-for-tat schema when
- dealing with remote peers that have just joined the swarm and thus
- have no pieces to offer.
-
- The algorithm should make good use of both download and upload
- bandwidth by putting a cap on the number of simultaneous
- connection that actively send or receive data. By reducing the
- number of active connections, TCP congestion can be avoided.
-
- The algorithm should pipeline data requests in order so saturate
- active connections.
-
- The algorithm should be able to cooperate with peers that
- implement a different algorithm.
-
-
-6.2 Handshaking
-
- The local peer opens a port on which to listen for incoming
- connections from remote peers. This port is then reported to the
- tracker. As BTP/1.0 does not specify any standard port for listening
- it is the sole responsibility of the implementation to select a port.
-
- Any remote peer wishing to communicate with the local peer must open
- a TCP connection to this port and perform a handshake operation. The
- handshake operation MUST be carried out before any other data is sent
- from the remote peer. The local peer MUST NOT send any data back to
-
-
-
-Fonseca, et al. [Page 16]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- the remote peer before a well constructed handshake has been
- recognized according to the rules below. If the handshake in any way
- violates these rules the local peer MUST close the connection with
- the remote peer.
-
-
-
-
-
- A handshake is a string of bytes with the following structure:
-
- ----------------------------------------------------------------
- | Name Length | Protocol Name | Reserved | Info Hash | Peer ID |
- ----------------------------------------------------------------
-
- Name Length: The unsigned value of the first byte indicates the
- length of a character string containing the protocol name. In
- BTP/1.0 this number is 19. The local peer knows its own protocol
- name and hence also the length of it. If this length is different
- than the value of this first byte, then the connection MUST be
- dropped.
-
- Protocol Name: This is a character string which MUST contain the
- exact name of the protocol in ASCII and have the same length as
- given in the Name Length field. The protocol name is used to
- identify to the local peer which version of BTP the remote peer
- uses. In BTP/1.0 the name is 'BitTorrent protocol'. If this
- string is different from the local peers own protocol name, then
- the connection is to be dropped.
-
- Reserved: The next 8 bytes in the string are reserved for future
- extensions and should be read without interpretation.
-
- Info Hash: The next 20 bytes in the string are to be interpreted as a
- 20-byte SHA1 of the info key in the metainfo file. Presumably,
- since both the local and the remote peer contacted the tracker as
- a result of reading in the same .torrent file, the local peer will
- recognize the info hash value and will be able to serve the remote
- peer. If this is not the case, then the connection MUST be
- dropped. This situation can arise if the local peer decides to no
- longer serve the file in question for some reason. The info hash
- may be used to enable the client to serve multiple torrents on the
- same port.
-
- At this stage, if the connection has not been dropped, then the
- local peer MUST send its own handshake back, which includes the
- last step:
-
-
-
-
-Fonseca, et al. [Page 17]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- Peer ID: The last 20 bytes of the handshake are to be interpreted as
- the self-designated name of the peer. The local peer must use
- this name to identify the connection hereafter. Thus, if this
- name matches the local peers own ID name, the connection MUST be
- dropped. Also, if any other peer has already identified itself to
- the local peer using that same peer ID, the connection MUST be
- dropped.
-
- In BTP/1.0 the handshake has a total of 68 bytes.
-
-6.3 Message Communication
-
- Following the PWP handshake both ends of the TCP channel may send
- messages to each other in a completely asynchronous fashion. PWP
- messages have the dual purpose of updating the state of neighboring
- peers with regard to changes in the local peer, as well as
- transferring data blocks between neighboring peers.
-
- PWP Messages fall into two different categories:
-
- State-oriented messages: These messages serve the sole purpose of
- informing peers of changes in the state of neighboring peers. A
- message of this type MUST be sent whenever a change occurs in a
- peer's state, regardless of the state of other peers. The
- following messages fall into this category: Interested,
- Uninterested, Choked, Unchoked, Have and Bitfield.
-
- Data-oriented messages: These messages handle the requesting and
- sending of data portions. The following messages fall into this
- category: Request, Cancel and Piece.
-
-
-6.3.1 Peer States
-
- For each end of a connection, a peer must maintain the following two
- state flags:
-
- Choked: When true, this flag means that the choked peer is not
- allowed to request data.
-
- Interested: When true, this flag means a peer is interested in
- requesting data from another peer. This indicates that the peer
- will start requesting blocks if it is unchoked.
-
- A choked peer MUST not send any data-oriented messages, but is free
- to send any other message to the peer that has choked it. If a peer
- chokes a remote peer, it MUST also discard any unanswered requests
- for blocks previously received from the remote peer.
-
-
-
-Fonseca, et al. [Page 18]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- An unchoked peer is allowed to send data-oriented messages to the
- remote peer. It is left to the implementation how many peers any
- given peer may choose to choke or unchoke, and in what fashion. This
- is done deliberately to allow peers to use different heuristics for
- peer selection.
-
- An interested peer indicates to the remote peer that it must expect
- to receive data-oriented messages as soon as it unchokes the
- interested peer. It must be noted, that a peer must not assume a
- remote peer is interested solely because it has pieces that the
- remote peer is lacking. There may be valid reasons why a peer is not
- interested in another peer other than data-based ones.
-
-6.3.2 Peer Wire Messages
-
- All integer members in PWP messages are encoded as a 4-byte big-
- endian number. Furthermore, all index and offset members in PWP
- messages are zero-based.
-
- A PWP message has the following structure:
-
- -----------------------------------------
- | Message Length | Message ID | Payload |
- -----------------------------------------
-
- Message Length: This is an integer which denotes the length of the
- message, excluding the length part itself. If a message has no
- payload, its size is 1. Messages of size 0 MAY be sent
- periodically as keep-alive messages. Apart from the limit that
- the four bytes impose on the message length, BTP does not specify
- a maximum limit on this value. Thus an implementation MAY choose
- to specify a different limit, and for instance disconnect a remote
- peer that wishes to communicate using a message length that would
- put too much strain on the local peer's resources.
-
- Message ID: This is a one byte value, indicating the type of the
- message. BTP/1.0 specifies 9 different messages, as can be seen
- further below.
-
- Payload: The payload is a variable length stream of bytes.
-
- If an incoming message in any way violates this structure then the
- connection SHOULD be dropped. In particular the receiver SHOULD make
- sure the message ID constitutes a valid message, and the payload
- matches the the expected payload, as given below.
-
- For the purpose of compatibility with future protocol extensions the
- client SHOULD ignore unknown messages. There may arise situations in
-
-
-
-Fonseca, et al. [Page 19]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- which a client may choose to drop a connection after receiving an
- unknown message, either for security reasons, or because discarding
- large unknown messages may be viewed as excessive waste.
-
- BTP/1.0 specifies the following messages:
-
-6.3.3 Choke
-
- This message has ID 0 and no payload. A peer sends this message to a
- remote peer to inform the remote peer that it is being choked.
-
-6.3.4 Unchoke
-
- This message has ID 1 and no payload. A peer sends this message to a
- remote peer to inform the remote peer that it is no longer being
- choked.
-
-6.3.5 Interested
-
- This message has ID 2 and no payload. A peer sends this message to a
- remote peer to inform the remote peer of its desire to request data.
-
-6.3.6 Uninterested
-
- This message has ID 3 and no payload. A peer sends this message to a
- remote peer to inform it that it is not interested in any pieces from
- the remote peer.
-
-6.3.7 Have
-
- This message has ID 4 and a payload of length 4. The payload is a
- number denoting the index of a piece that the peer has successfully
- downloaded and validated. A peer receiving this message must
- validate the index and drop the connection if this index is not
- within the expected bounds. Also, a peer receiving this message MUST
- send an interested message to the sender if indeed it lacks the piece
- announced. Further, it MAY also send a request for that piece.
-
-6.3.8 Bitfield
-
- This message has ID 5 and a variable payload length. The payload is
- a bitfield representing the pieces that the sender has successfully
- downloaded, with the high bit in the first byte corresponding to
- piece index 0. If a bit is cleared it is to be interpreted as a
- missing piece. A peer MUST send this message immediately after the
- handshake operation, and MAY choose not to send it if it has no
- pieces at all. This message MUST not be sent at any other time
- during the communication.
-
-
-
-Fonseca, et al. [Page 20]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-6.3.9 Request
-
- This message has ID 6 and a payload of length 12. The payload is 3
- integer values indicating a block within a piece that the sender is
- interested in downloading from the recipient. The recipient MUST
- only send piece messages to a sender that has already requested it,
- and only in accordance to the rules given above about the choke and
- interested states. The payload has the following structure:
-
- ---------------------------------------------
- | Piece Index | Block Offset | Block Length |
- ---------------------------------------------
-
-
-6.3.10 Piece
-
- This message has ID 7 and a variable length payload. The payload
- holds 2 integers indicating from which piece and with what offset the
- block data in the 3rd member is derived. Note, the data length is
- implicit and can be calculated by subtracting 9 from the total
- message length. The payload has the following structure:
-
- -------------------------------------------
- | Piece Index | Block Offset | Block Data |
- -------------------------------------------
-
-
-6.3.11 Cancel
-
- This message has ID 8 and a payload of length 12. The payload is 3
- integer values indicating a block within a piece that the sender has
- requested for, but is no longer interested in. The recipient MUST
- erase the request information upon receiving this messages. The
- payload has the following structure:
-
- ---------------------------------------------
- | Piece Index | Block Offset | Block Length |
- ---------------------------------------------
-
-
-6.4 The End Game
-
- Towards the end of a download session, it may speed up the download
- to send request messages for the remaining blocks to all the
- neighboring peers. A client must issue cancel messages to all
- pending requests sent to neighboring peers as soon as a piece is
- downloaded successfully. This is referred to as the end game.
-
-
-
-
-Fonseca, et al. [Page 21]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- A client usually sends requests for blocks in stages; sending
- requests for newer blocks as replies for earlier requests are
- received. The client enters the end game, when all remaining blocks
- have been requested.
-
-6.5 Piece Selection Strategy
-
- BTP/1.0 does not force a particular order for selecting which pieces
- to download. However, experience shows that downloading in rarest-
- first order seems to lessen the wait time for pieces. To find the
- rarest piece a client must calculate for each piece index the number
- of times this index is true in the bitfield vectors of all the
- neighboring peers. The piece with the lowest sum is then selected
- for requesting.
-
-6.6 Peer Selection Strategy
-
- This section describes the choking algorithm recommended for
- selecting neighboring peers with whom to exchange pieces.
- Implementations are free to implement any strategy as long as the
- guidelines in Section 6.1 are observed.
-
- After the initial handshake both ends of a connection set the Choked
- flag to true and the Interested flag to false.
-
- All connections are periodically rated in terms of their ability to
- provide the client with a better download rate. The rating may take
- into account factors such as the remote peers willingness to maintain
- an unchoked connection with the client over a certain period of time,
- the remote peers upload rate to the client and other implementation
- defined criteria.
-
- The peers are sorted according to their rating with regard to the
- above mentioned scheme. Assume only 5 peers are allowed to download
- at the same time. The peer selection algorithm will now unchoke as
- many of the best rated peers as necessary so that exactly 5 of these
- are interested. If one of the top rated peers at a later stage
- becomes interested, then the peer selection algorithm will choke the
- the worst unchoked peer. Notice that the worst unchoked peer is
- always interested.
-
- The only lacking element from the above algorithm is the capability
- to ensure that new peers can have a fair chance of downloading a
- piece, even though they would evaluate poorly in the above schema. A
- simple method is to make sure that a random peer is selected
- periodically regardless of how it evaluates. Since this process is
- repeated in a round robin manner, it ensures that ultimately even new
- peers will have a chance of being unchoked.
-
-
-
-Fonseca, et al. [Page 22]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-7. Security Consideration
-
- This section examines security considerations for BTP/1.0.The
- discussion does not include definitive solutions to the problems
- revealed, though it does make some suggestions for reducing security
- risks.
-
-7.1 Tracker HTTP Protocol Issues
-
- The use of the HTTP protocol for communication between the tracker
- and the client makes BTP/1.0 vulnerable to the attacks mentioned in
- the security consideration section of RFC 2616 [6].
-
-7.2 Denial of Service Attacks on Trackers
-
- The nature of the tracker is to serve many clients. By mounting a
- denial of service attack against the tracker the swarm attached to
- the tracker can be starved. This type of attack is hard to defend
- against, however, the metainfo file allows for multiple trackers to
- be specified, making it possible to spread the load on a number of
- trackers, and thus containing such an attack.
-
-7.3 Peer Identity Issues
-
- There is no strong authentication of clients when they contact the
- tracker. The main option for trackers is to check peer ID and the IP
- address of the client. The lack of authentication can be used to
- mount an attack where a client can shut down another client if the
- two clients are running on the same host and thus are sharing the
- same IP address. In addition, a rogue peer may masquerade its
- identity by using multiple peer IDs. Clients should there refrain
- from taking the peer ID at face value.
-
-7.4 DNS Spoofing
-
- Clients using BTP/1.0 rely heavily on the Domain Name Service, which
- can be used for both specifying the URI of the tracker and how to
- contact a peer. Clients are thus generally prone to security attacks
- based on the deliberate mis-association of IP addresses and DNS
- names. Clients need to be cautious in assuming the continuing
- validity of an IP address/DNS name association.
-
- In particular, BTP/1.0 clients SHOULD rely on their name resolver for
- confirmation of an IP number/DNS name association, rather than
- caching the result of previous host name lookups. If clients cache
- the results of host name lookups in order to achieve a performance
- improvement, they MUST observe the TTL information reported by DNS.
-
-
-
-
-Fonseca, et al. [Page 23]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- If clients do not observe this rule, they could be spoofed when a
- previously-accessed peers or trackers IP address changes. As network
- renumbering is expected to become increasingly common according to
- RFC 1900 [2], the possibility of this form of attack will grow.
- Observing this requirement reduces this potential security
- vulnerability.
-
-7.5 Issues with File and Directory Names
-
- The metainfo file provides a way to suggest a name of the downloaded
- file for single-file torrents and the top-most directory for multi-
- file torrents. This functionality is very like the Content-
- Disposition header field documented in RFC 2183 [4] and the security
- considerations mentioned in this RFC also apply to BitTorrent
- clients. In short, BTP clients SHOULD verify that the suggested file
- names in the metainfo file do not compromise services on the local
- system. Furthermore, care must be taken for multi-file torrents to
- validate that individual files are relative to the top-most directory
- and that the paths do not contain path elements to the parent (that
- is directory elevators such as ``..''), which can be used to place
- files outside the top-most directory.
-
- Using UNIX as an example, some hazards would be:
-
- o Creating startup files (e.g., ".login").
-
- o Creating or overwriting system files (e.g., "/etc/passwd").
-
- o Overwriting any existing file.
-
- o Placing executable files into any command search path (e.g.,
- "~/bin/more").
-
- o Sending the file to a pipe (e.g., "| sh").
-
- It is very important to note that this is not an exhaustive list; it
- is intended as a small set of examples only. Implementers must be
- alert to the potential hazards on their target systems. In general,
- the BTP client SHOULD NOT name or place files such that they will get
- interpreted or executed without the user explicitly initiating the
- action.
-
-7.6 Validating the Integrity of Data Exchanged Between Peers
-
- By default, all content served to the client from other peers should
- be considered tainted and the client SHOULD validate the integrity of
- the data before accepting it. The metainfo file contains information
- for checking both individual pieces using SHA1, and optionally
-
-
-
-Fonseca, et al. [Page 24]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
- individual files using MD5. SHA1, being the strongest of the two, is
- preferred. Furthermore, sole reliance on whole-file checking can
- potentially render otherwise valid pieces invalid, and should only be
- considered for small files, to limit the amount of data being
- discarded.
-
- Trusting the validity of the resulting file or files ends up being a
- matter of trusting the content of the metainfo file. Ensuring the
- validity of the metainfo file is beyond the scope of this document.
-
-7.7 Transfer of Sensitive Information
-
- Some clients include information about themselves when generating the
- peer ID string. Clients should be aware that this information can
- potentially be used to determine whether a specific client has a
- exploitable security hole.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Fonseca, et al. [Page 25]
-
-BTP/1.0 BitTorrent Protocol -- BTP/1.0 April 2005
-
-
-8. IANA Considerations
-
- This document makes no request of IANA.
-
-9. References
-
- [1] Postel, J. and J. Reynolds, "File Transfer Protocol", STD 9,
- RFC 959, October 1985.
-
- [2] Carpenter, B. and Y. Rekhter, "Renumbering Needs Work",
- RFC 1900, February 1996.
-
- [3] Bradner, S., "Key words for use in RFCs to Indicate Requirement
- Levels", BCP 14, RFC 2119, March 1997.
-
- [4] Troost, R., Dorner, S., and K. Moore, "Communicating
- Presentation Information in Internet Messages: The Content-
- Disposition Header Field", RFC 2183, August 1997.
-
- [5] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
- Specifications: ABNF", RFC 2234, November 1997.
-
- [6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L.,
- Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol --
- HTTP/1.1", RFC 2616, June 1999.
-
-
-Authors' Addresses
-
- Jonas Fonseca
- DIKU
-
- Email: fonseca@diku.dk
-
-
- Basim Reza
- DIKU
-
- Email: basim@diku.dk
-
-
- Lilja Fjeldsted
- DIKU
-
- Email: lilja@diku.dk
-
-
-
-
-
-
-Fonseca, et al. [Page 26]
-
diff --git a/doc/links.txt b/doc/links.txt
deleted file mode 100644
index 55902ef..0000000
--- a/doc/links.txt
+++ /dev/null
@@ -1,11 +0,0 @@
-http://jonas.nitro.dk/bittorrent/bittorrent-rfc.html
-
-http://bittorrent.org/beps/bep_0003.html
-http://bittorrent.org/beps/bep_0005.html
-
-https://wiki.theory.org/BitTorrentSpecification
-http://wiki.vuze.com/w/Azureus_messaging_protocol
-
-Some random overview
-http://www.kristenwidman.com/blog/33/how-to-write-a-bittorrent-client-part-1/
-http://www.kristenwidman.com/blog/71/how-to-write-a-bittorrent-client-part-2/
diff --git a/doc/out.txt b/doc/out.txt
deleted file mode 100644
index 82bf6de..0000000
--- a/doc/out.txt
+++ /dev/null
@@ -1,65 +0,0 @@
-metainfo file.: OpenBSD_songs_ogg-2016-03-25-0127.torrent
-info hash.....: 1aa5af9f6f533961a65169bdeeb801e611724d32
-
-directory name: OpenBSD_songs_ogg
-files.........: 38
- OpenBSD_songs_ogg/song30.ogg (2636244) 2.51 MB
- OpenBSD_songs_ogg/song31.ogg (2381018) 2.27 MB
- OpenBSD_songs_ogg/song32.ogg (2253880) 2.15 MB
- OpenBSD_songs_ogg/song33.ogg (3238559) 3.09 MB
- OpenBSD_songs_ogg/song34.ogg (5062603) 4.83 MB
- OpenBSD_songs_ogg/song35.ogg (7191264) 6.86 MB
- OpenBSD_songs_ogg/song36.ogg (5197630) 4.96 MB
- OpenBSD_songs_ogg/song37.ogg (13937255) 13.29 MB
- OpenBSD_songs_ogg/song38.ogg (5860681) 5.59 MB
- OpenBSD_songs_ogg/song38b.ogg (5759105) 5.49 MB
- OpenBSD_songs_ogg/song39.ogg (5954997) 5.68 MB
- OpenBSD_songs_ogg/song40.ogg (3610740) 3.44 MB
- OpenBSD_songs_ogg/song41.ogg (8382781) 7.99 MB
- OpenBSD_songs_ogg/song42.ogg (6713047) 6.40 MB
- OpenBSD_songs_ogg/song43.ogg (6834505) 6.52 MB
- OpenBSD_songs_ogg/song44.ogg (4628329) 4.41 MB
- OpenBSD_songs_ogg/song45.ogg (4737475) 4.52 MB
- OpenBSD_songs_ogg/song46.ogg (3766867) 3.59 MB
- OpenBSD_songs_ogg/song47.ogg (6586251) 6.28 MB
- OpenBSD_songs_ogg/song48.ogg (3143847) 3.00 MB
- OpenBSD_songs_ogg/song49.ogg (6006834) 5.73 MB
- OpenBSD_songs_ogg/song50.ogg (4184951) 3.99 MB
- OpenBSD_songs_ogg/song51.ogg (4216946) 4.02 MB
- OpenBSD_songs_ogg/song52.ogg (4283099) 4.08 MB
- OpenBSD_songs_ogg/song53.ogg (4635216) 4.42 MB
- OpenBSD_songs_ogg/song54.ogg (3195265) 3.05 MB
- OpenBSD_songs_ogg/song55.ogg (6160151) 5.87 MB
- OpenBSD_songs_ogg/song56.ogg (5471902) 5.22 MB
- OpenBSD_songs_ogg/song57.ogg (4057801) 3.87 MB
- OpenBSD_songs_ogg/song58a.ogg (3252520) 3.10 MB
- OpenBSD_songs_ogg/song58b.ogg (4376163) 4.17 MB
- OpenBSD_songs_ogg/song58c.ogg (3595317) 3.43 MB
- OpenBSD_songs_ogg/song58d.ogg (7013091) 6.69 MB
- OpenBSD_songs_ogg/song59a.ogg (5766764) 5.50 MB
- OpenBSD_songs_ogg/song59b.ogg (5317415) 5.07 MB
- OpenBSD_songs_ogg/songsh.ogg (4890381) 4.66 MB
- OpenBSD_songs_ogg/songsi.ogg (5694670) 5.43 MB
- OpenBSD_songs_ogg/songty.ogg (6045645) 5.77 MB
-archive size..: 196041209 (747 * 262144 + 219641) 186.96 MB
-
-announce url..: http://OpenBSD.somedomain.net:6969/announce
-creation date.: Fri Mar 25 02:27:56 2016
-created by....: mktorrent 1.0
-comment.......: ogg files from OpenBSD/songs
-Created by andrew fresh (andrew@afresh1.com)
-http://OpenBSD.somedomain.net/
-
-
-metainfo file.: OpenBSD_5.9_amd64_install59.iso-2016-03-29-0449.torrent
-info hash.....: e840038dea1998c39614dcd28594501df02bd32d
-
-file name.....: OpenBSD_5.9_amd64_install59.iso
-file size.....: 233662464 (891 * 262144 + 92160) 222.84 MB
-
-announce url..: http://OpenBSD.somedomain.net:6969/announce
-creation date.: Tue Mar 29 06:49:18 2016
-created by....: mktorrent 1.0
-comment.......: OpenBSD/5.9/amd64/install59.iso
-Created by andrew fresh (andrew@afresh1.com)
-http://OpenBSD.somedomain.net/