Web services and peer-to-peer technology |
|
conundrum
尊榮會員 發表:893 回覆:1272 積分:643 註冊:2004-01-06 發送簡訊給我 |
IFS - an Internet File System implementation based on Web services and peer-to-peer technology By Stoyan Damov Internet File System from scratch - making web services and peer-to-peer technology work together to build a virtual file system http://www.codeproject.com/webservices/ifs.asp " Download source and this article - 205 KB " Latest source code and updated documentation is (will be) available here soon. On this page 1. Introduction 2. What is Internet File System? a. The IFS architecture b. Why this IFS is (or will be) cool? c. How it works? 3. Using the library a. Peer registration & login b. Working with folders c. Working with files 4. The IFS Browser 5. Compiling and configuring IFS a. Compilation b. Configuration c. Setting up the IFS database 6. About MC , some tricks 'n' traps 7. A thought about warnings I wanted to share... 8. TODO(s): Oh, my! 9. Conclusion 10. Reporting bugs 11. Latest changes 12. Disclaimer Introduction You've seen Napster, Gnucleus, Morpheus and the like file-sharing applications. If you haven't, I'm sure you've heard about the exciting peer-to-peer (P2P) technology. I don't doubt you've seen the big hype about web services, but you probably haven't seen anything smarter than an HTML screen-scrapping web service, or one, exposing some proprietary technology or software to the web-aware public. This article demonstrates a virtual file system, based on P2P and web services, which in all modesty I call "The Internet File System" or IFS. Unfortunately, big articles look bad, so I'll show you how to use the library I've written, and if you are smart, (and I know you are) you'll get the rest from the source code. Have a pleasant reading... What is Internet File System? The Internet File System is an imaginary file system. A real Internet file system could hardly exist, because of the variety of hardware and software platforms (OSes) scattered around the world. The IFS simulates a file system by having a central repository of folder and file links (shortcuts) published by a multitude of computers, known as peers. Peers share files and folders in the repository, but do not use it to actually store the files contents. They communicate to each other instead, to download (copy) the files between themselves. The IFS Architecture In this simple implementation, the repository is managed by the peers via a web service (IFSWS), and the peer-to-peer communication is handled by a P2P framework library (P2PLIB). Because of the complexity to use directly the web service and the P2P framework, I've built a stand-alone P2P server (P2PSRV), which runs on the peer computers and handle all the P2P communication automatically. Furthermore, to hide the details of the IFS implementation, I've built a Internet File System library (IFSLIB) on top of IFSWS and P2PLIB, that exposes an easy to use object model for manipulation of the central repository, as well as easy copy (download) of files between peers. As you can see on the diagram above, there are two big parties here: the IFS Web Service, and the peers: The IFS Web Service It is only responsible for storing and retrieving information - it just serves as a repository. Actually, it registers/unregisters/logs in peers, publishes folders and files and retrieves vital information about the peers, etc., but the coolest part of IFS is done by the library. The Peers I'm not a native English speaker, I'm Bulgarian. That's why I cant explain exactly what a peer is. In my dictionary, the word "peer" has just one meaning as a noun, and it is not "a PC connected to the Internet, that can communicated directly whith other PCs, using some communication protocol". However, that's the meaning of peer I've put in this article. There are 3 things that make the dumb PCs peers - these are the P2P library, the P2P server, and the IFS library. The IFSLIB works on top of the web service, and the P2P server on top of the library. One can use the IFS library to build any kind of P2P application - GUI-based Windows Explorer-like IFS browser, console tools for automatic/scheduled download of files, etc. I started working on an IFS browser, but due to the limited time (I started this article to enter the web contest) could not finish it. However, I included the crippled demo in the source code, so anyone who likes the idea (and can code in C#) could finish (or rewrite) it. (Now, as I am updating the article, two weeks later, the status of the IFSBrowser is unchanged. That's because I thought that fixing bugs and adding features to the library has a bigger priority at least for me.) Why this IFS is (or could be) cool? Well, you have 3 scenarios when you have to copy a file from one peer to another. To avoid tautology, lets name the interested peer (which wants to download a file) P1, and the owner peer -- P2. " Both peers are not behind firewall No problem. P1 pulls the file from P2 " P2 is behind firewall Ooops! P1 can't connect to P2 just that easy. So what can it do? It can execute a web service method that will assign a task to P2 to push the file to P1. The P2P server on P2's PC is scheduled to retrieve its tasks, using the web service. When it gets a push task, it connects to P1 and pushes the file. Is this easy? Yes, and I have almost implemented this feature. " Both peers are behind firewall Yeaah! What shall we do now? Well, if P1 could assign an upload task to P2, the latter could upload the file, using the web service to the web service's server PC. When P2 is done, it could assign a download task to P1 and the latter will download the file! Voila! This feature is not implemented at all but is very easy to implement. However, because I'm pressed by a DEADline sooooo badly, I don't promise that I'll implement it for the next week... How it works? How works what?:) OK, I'll try to focus and tell you how the library works. I won't discuss how the basic functionality works, i.e. peer registration, log in, folder and file publishing, etc. These are just entries in the database via the IFS web service. I'll show you how a typical file download works in both of the easier scenarios, i.e. the "no firewalls" and "owner is behind firewall" situations. And because I know you're SMART guys (and I'm so lazy:) I haven't drawn any pictures, so read along... No firewalls I'll use P1 and P2 again for the interested and the owner peer. P1 has a IfsFile instance (see library objects below) and calls the Download method. The IfsFile object has a OwnerPeer property, which returns an IfsPeer instance. The latter has an IpEndPoint property, containing the IP address and port of the remote peer. At this moment, the IFS library sends a "library pull" command to the P2P server, running on P1's PC. This command means the the library wants the server to pull a file from a remote peer. The P2P server gets the remote peer's end point from the command, as well as the file name P1 wants to pull and the folder in which to download the file and then sends a "peer pull" command to the remote peer (P2). P2's P2P server accepts the command, gets the file size and CRC (CRC is not implemented) and sends them back the P1. P1 now knows exactly how many bytes to accept and starts to receive the bits P2 sends and write them in the destination folder. After P2 has sent all the bytes and P1 has received them, P1 sends an "OK" response (or "ERROR" if something goes wrong) to P2 and closes the connection. That's pretty much what a download is. Owner behind firewall P1 assigns a "push task" to P2 via the IFS web service and "thinks" that the file has been downloaded :) At some time, P2's P2P server reads its task via the IFSWS and sees it has a "push task". P2P issues to itself (the task reader is in another thread) a "library push" command. As the peer command listener thread receives the command, it gets the file size and CRC (CRC again is not implemented) and forms a "peer push" command. P2 connects to P1 (the task contains P1's end point) and issues the "peer push" command and starts to stream the file. P1 gets the command, receives the file and stores it in the destination folder, indicated by P2. (P2 knows the destination folder, because P1 has sent it in the "push task".) That's all. Using the library There's a long way to go until you actually start using the library, but we'll get to that stuff (Compiling and configuring IFS) soon. Now I just want to show you how easy it is to use IFSLIB. The library consists of only 4 classes, that hide everything about peer-to-peer and web services stuff: IfsSystem, IfsPeer, IfsFolder and the IfsFile. In order to use IFS (after it is already set up), you should register in the repository as a peer. There's nothing easier than that, like you'll see in a moment, but I'll warn you something first: unless you have registered or logged in as a peer with IFS, you won't be able to use the IFS's most important object - the IfsSystem object. It is implemented as singleton to avoid having multiple instances of IFS peers on the same peer computer. If you actually try to use even the simpliest property of the IfsSystem class, you'll get a runtime exception, stating that you haven't logged in/registered with IFS. First, you have to register as a new peer: // first, you'll have to get a "handle" of the IFS singleton object IfsSystem __gc* ifs = IfsSystem::Instance; // now you can register IfsPeer __gc* peer = ifs->RegisterPeer ( S"Stoyan Damov", // alias S"Stoyan", // login name S"Secret", // password S"BG", // country code (unused in this version) false); // behind firewall or NAT? Of course, you may not register more than one time in IFS (unless you're prepared for exceptions), so once you have registered, the next time you should sign in: IfsPeer __gc* peer = ifs->LoginPeer (S"Stoyan", S"Secret"); // in fact you can throw the peer away, you won't use it for // anything, except for examining its properties A peer (IfsPeer instance) has the following properties: " ID - the unique peer ID in IFS " Alias - peer's human-readable name " Login - visible only to the peer (other peers see empty string) " Password - ditto " IPEndPoint - peer's IP address and port " BehindFirewall - indicates whether the peer is behind firewall or NAT, i.e. whether the peer's PC is "visible" or externally "connectable":) Once you've registered or logged in, you can start using the IfsSystem object's properties, the most important (and usable) of which is the RootFolder property, which returns an IfsFolder object, representing the virtual root folder. Working with folders You can get the root folder just that easy: IfsFolder __gc* root = ifs->RootFolder; Each folder (including the root one has several properties): " ID - the unique folder ID in IFS " Name " Description " OwnerPeer - the IfsPeer object, owner of the folder " ParentFolder - the parent IfsFolder (evaluated on request and cached) " PeerFolderPath - the physical folder on the owner peer's PC " VirtualPath - the IFS logical path (evaluated on request and cached) " Published - a boolean flag, indicating whether the folder has been published or is brand new IfsFolder instance After you get to the root folder, there are many things you can do with it: " publish a sub-folder in it " get all of its sub-folders " find sub-folders having a specified name (recursively) " publish a file in it " get all of its files " find files having a specified name (recursively) Actually you can perform these actions with all folder objects you get a pointer to, and you can very easily get an arbitrary folder like this: IfsFolder __gc* folder = IfsFolder::GetFolderByPath (S"./Docs/PDF"); Below, I am giving some examples of the fore-mentioned operations, and once you've seen them, you can move to some useful static methods of the IfsFolder class. You can publish a folder into an existing one: IfsFolder __gc* subFolder = root->PublishFolderHere ( S"VirtualFolderName", S"Description", // may be omitted in an overload S"c:\\physical\\folder\\path"); // and for more fun: IfsFolder __gc* subSub = (root->PublishFolderHere ( S"Folder", S"Description", // may be omitted in an overload S"c:\\physical\\folder"))->PublishFolderHere ( S"SubFolder", S"c:\\physical\\folder\\subFolder"); or publish a brand new folder: // when you know the destination path IfsFolder __gc* folder = new IfsFolder ( S"VirtualName", S"Description", // may be omitted in an overload S"c:\\physical\\folder\\path"); folder->Publish ("./target/virtual/path"); // when you have the destination folder object folder->Publish (targetFolder); The lazy guys (this includes me) can publish folders using the static methods: IfsFolder __gc* folder = IfsFolder::PublishFolder ( S"VirtualName", S"Description", // may be omitted in an overload S"c:\\physical\\folder\\path", S"./target/virtual/path"); Oooh, I forgot to tell you how do you rename a folder // I assume you got one already folder->RenameTo (S"NewFolderName"); // wow! how difficult :) There may (or may not) exist other methods (either static or instance ones) for publishing a folder, but I think these were enough to show you how easy it is done. Now, its time to see what you can do with the published folders: You can find sub-folders: // the statement below will return all folders, arbitrary level // below the "folder" one, which name is "docs" (recursively) ArrayList __gc* folders = folder->FindSubFolders (S"docs"); or get all folders: // this statement will return all folders below the "folder" object ArrayList __gc* folders = folder->GetFolders (); or even find folders in the entire IFS: ArrayList __gc* folders = IfsFolder::FindFolders (S"docs"); // the above is equivalent to: ArrayList __gc* folders = root->FindSubFolders (S"docs"); Working with files In the previous version of the article, I forgot to write a lot of things about the files. I forgot to tell you that a file (IfsFile) has some useful properties: " ID - the unique file ID in IFS " Name - the file's name (w/o path) " Description - guess what:) " OwnerPeer - the IfsPeer object, that owns the file:) " PeerFilePath - the full path (including the file name) in the owner peer's PC " Folder - the IfsFolder object, where the file resides " Published - a boolean flag, indicating whether the file has been published or is brand new IfsFile instance Again, in the previous article, I mentioned that the IfsFolder has several instance and static methods to publish a folder and a file. Do you know why the folder should publish a file, and not a file publish itself? Because I was stupid. I was not able to use the IfsFolder class in the IfsFile one, because I would create a cyclic header include. Every C programmer knows s/he should not include the header, but rather just declare the class in the header like class __gc* IfsFolder;. That's what I did then, but it didn't work and I thought that either I suck, or Visual C does. Well, I suck, but let me tell you why. I forgot that all classes in the IFS library were wrapped in two namespaces. That's why either I should have written class __gc* IfsFolder; inside the namespaces of the "IfsFile.h" header, or wrap the declaration in the namespaces, like this: namespace InternetFileSystem { namespace Library { public __gc class IfsFolder; }}. So, that's what I did, and now the IfsFile class has six instance or static methods for publishing. You can create a brand new file like this: IfsFile __gc* file = new IfsFile ( S"fileName", S"file description", // may be omitted in an overload S"x:\\full\\path\\to\\fileName"); And publish it like that: // a) calling the static Publish method (laziest) IfsFile __gc* file = IfsFile::Publish ( S"fileName", S"file description", // may be omitted in an overload S"x:\\full\\path\\to\\fileName", S"./ifs/target/path/"); // b) calling another static Publish method (you should have an // IfsFolder before that) assuming you have the targetFolder, // which is an instance of the IfsFolder class IfsFile __gc* file = IfsFile::Publish ( S"fileName", S"file description", // may be omitted in an overload S"x:\\full\\path\\to\\fileName", targetFolder); // c) You have a brand new file and you want to publish it file->Publish (S"./ifs/target/path/"); // d) You have a brand new file and an IfsFolder instance // (targetFolder); file->Publish (targetFolder); You can get an IfsFile object in several ways: // get the file (assuming you have the folder already) IfsFile __gc* file = folder->GetFile (S"readme.txt"); // get a folder's files ArrayList __gc* files = folder->GetFiles (); // or search in the whole IFS for a given file ArrayList __gc* files = IfsFolder::FindFiles (S"readme.txt"); // now you get a file like this: IfsFile __gc* file = static_cast |
本站聲明 |
1. 本論壇為無營利行為之開放平台,所有文章都是由網友自行張貼,如牽涉到法律糾紛一切與本站無關。 2. 假如網友發表之內容涉及侵權,而損及您的利益,請立即通知版主刪除。 3. 請勿批評中華民國元首及政府或批評各政黨,是藍是綠本站無權干涉,但這裡不是政治性論壇! |