Rorur RoadmapAccording to our current vision, there must be several stages in the development and deployment of the Distributed Search Engine (DSE). The functioning of DSE is based on splitting the workload between the participating agents, in particular, on allocation of pages to nodes. ^^ Therefore, it is necessary to first sample the web and collect a sample of urls to seed the crawl for each of the nodes. Furthermore, we must ensure that there are enough nodes to bootstrap the distribution algorithm that assigns the urls to nodes. Therefore, the first stage in the deployment will consist in autonomous operation of nodes that independently sample the web. The nodes only collect links and construct the web graph at this stage. The condition to transit to the next stage will be decided centrally by a ”monitor” node, and amounts to sampling around 10% of urls (about 109 urls).We have already written the code that pertains Stage 1 and most of the code that pertains Stage 2. Now it is the issue of having enough hardware to run this project and harvest the results. Most of the software development now pertains researching and implementing ranking algorithms. In essence, this is the most interesting part of the project, when most of the dirty work is already done.RorurRRR_296409e4-62ab-11ed-a5a1-28b91083ea00Stanislav Srednyak, Ph.D.A distributed search engine_2964122c-62ab-11ed-a5a1-28b91083ea00To specify application programming interfaces to support distributed query service nodes_296412f4-62ab-11ed-a5a1-28b91083ea00AutonomyPeerismAPICommunicate between the monitor and the nodes_2964138a-62ab-11ed-a5a1-28b91083ea00Stage 1At stage 1 communication goes only between the monitor and the nodes, there is no communication between nodes. Since there are about 1010 urls currently in use, the storage of the database of these urls amount to about 100 ∗ 1010 = 1T B ( assuming about 100B per url). Note that we do not construct the web graph at this stage. Construction and maintainance of the web graph will require about 100 times more space ( estimating that there are about 100 outgoing links per page). ^^ It is very important that enough nodes join at this stage. Although some crawl can be done with small number of nodes, for example, by restricting to the most popular pages, this kind of machine would be of limited utility. ^^ We should note that with 104 nodes that operate with standard bandwidth it is possible to do a crawl of the whole web in less than a day.URLsMaintain local data base of the visited urls_29641420-62ab-11ed-a5a1-28b91083ea00api1.1fetch urls, parse them, and maintain local data base of the visited urls. No url is visited twice._12ba2c9a-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_1[To be determined]DataCollect the url data from all nodes_296414ac-62ab-11ed-a5a1-28b91083ea00api1.2the collected links are packed and sent to the monitor. Monitor collects the url data from all nodes._12ba2fd8-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_2[To be determined]HistoryMaintain the history of the crawl for each of the nodes_29641542-62ab-11ed-a5a1-28b91083ea00api1.3Monitor keeps history of the crawl for each of the nodes. Once the necessary percentage is crawled, the monitor issues a signal to transit to the next stage. Node code is written so that it auto-renews itself on the command form the monitor, by fetching a new copy of the code from a pre-defined repository._12ba321c-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_3[To be determined]TransferSupport secure data transfer between the monitor and the nodes, and back_296415d8-62ab-11ed-a5a1-28b91083ea00api1.4there is corresponding protocol that allows for secure data transfer between the monitor and the nodes, and back. At stage 1 the set of commands is limited to the transfer of url lists._12ba3320-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_4[To be determined]AllocationAllocate pages among the nodes for analysis and ranking_2964166e-62ab-11ed-a5a1-28b91083ea00Stage 2This stage constitutes the core of the web analysis and ranking. At this stage the nodes receive the allocation of pages that they will be responsible for. The pages are distributed in pseudo-random fashion. No single host will be assigned to individual node. The url assignment must respect the disk space limits of individual nodes, i.e. , the url list that each node will be serving must have total size less than the disk limit specified by the user on the installation of node code. ^^ At this stage, nodes need to communicate to each other in order to ^^ 1. efficiently crawl the web ^ 2. compute the rank. ^^ At this stage, nodes will not crawl the web randomly, as at stage 1, rather, they will only crawl the assigned pages. When they encounter a link that does not belong to their name space, they will send it out to the peers that are responsible for this url. This dictates the following functionality:MappingMaintain the globally available table that maps urls to nodes._29641768-62ab-11ed-a5a1-28b91083ea00api2.1On the surface of it, this looks like a large data base that must be maintained in distributed fashion. However, there is a workaround based on hashing the urls and assigning the segments to the nodes. This results in a much smaller database, which is independent of web size, and linear in the number of nodes. We can, for example, maintain it on a central server, or even let each node keep its own copy. This last option may be the most efficient ,since it will not require fetching the node address when sending the url to the corresponding node._12ba34ec-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_5[To be determined]ListeningOpen ports to listen for incoming data_29641812-62ab-11ed-a5a1-28b91083ea00api2.2Nodes must open ports to listen for incoming data that contains the urls harvested by the peers that belongs to their name segment._12ba35be-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_6[To be determined]Service, Ranking & AnalysesServe, rank and analyze data in html pages_296418bc-62ab-11ed-a5a1-28b91083ea00api2.3Nodes must serve html pages that contain the statistics of their crawl, ranking and data analysis, as well as history of the receive and send transactions ( for debugging purposes)._12ba3690-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_7[To be determined]IndicesConstruct local indices of pages_29641966-62ab-11ed-a5a1-28b91083ea00api2.4Nodes construct the local index of pages in their name segment. In contrast to Stage 1, at this stage nodes save the html of the pages._12ba3762-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_8[To be determined]Latency & LoadingCollect and share latency information and load information_29641a10-62ab-11ed-a5a1-28b91083ea00api2.5Nodes collect the latency information and load information for each pair, and send this information to pre-assigned monitor nodes. The monitor nodes collect this information, and based on this, calculate the roles the nodes will play in collecting information when user queries will be served. The monitor nodes are maintained by the network operators and are trusted nodes at this stage. The monitors are the trusted nodes that perform network optimization and role allocation. It is their responsibility to design the distributed index tree and return tree ( as described in the white paper) and to assign corresponding privileges to the nodes. The privilege table will be globally available. The nodes will be looking it up when verifying the url and rank information that they receive from peers._12ba383e-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_9[To be determined]RankingCompute the ranking structure for pages_29641aba-62ab-11ed-a5a1-28b91083ea00api2.6Nodes compute the ranking structure for the pages in their name segment. At this stage, we may limit this computation to the most primitive pagerank ( which is just a number), but the code must be written in such a way as to easily incorporate new ranking algorithms , on the request from the maintainer nodes. The defining feature of the network ranks of the pages lies in its dependence on the web graph. ^^ Therefore, this computation must be performed globally by the network, in several iterative stages, each of them consisting in evaluation of the rank based on the data received from the peers according to the incoming links for a page. It must be ensured that the rank structure is designed in such a way that this iterative computation converges in few steps ( pagerank for example has this property). We may add the feature that nodes check the existence of the incoming links , to reduce the probability of fraud. ^^ Consistency of the rank must be verified at this stage. It seems sufficient to cross check the rank computed by different nodes that cover the same name segment. However, we may add special set of pseudo-privileges for nodes to query the peers for the rank, and then check consistency at randomly chosen url. Therefore, we may add a set of functions to the node communication protocol that will allow such requests._12ba3924-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_10[To be determined]TransmissionTransmit information serving queries_29641b64-62ab-11ed-a5a1-28b91083ea00api2.7Nodes receive part of the code that is responsible for collection and transmission of information that serves particular query. They are assigned roles in this process and receive corresponding privileges. The tree information is globally available. In particular, nodes know the addresses for the parent nodes to which to send the merged rank lists , and the addresses and ids of the child nodes, from whom they can receive rank lists (more details see in the white paper). ^^ At this stage we will have fully functioning search engine._12ba3a0a-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_11[To be determined]AdvertisingImplement advertising_29641d3a-62ab-11ed-a5a1-28b91083ea00Stage 3At this stage, we will implement advertising system. We will detail the api later on.DetailsTo be determined_29641e0c-62ab-11ed-a5a1-28b91083ea00api2.8_12ba3bb8-62ac-11ed-90e5-55e61083ea00PLACEHOLDER_12[To be determined]_12ba3cc6-62ac-11ed-90e5-55e61083ea002021-07-152022-11-12https://rorur.com/roadmap3.pdf_12ba3e4c-62ac-11ed-90e5-55e61083ea00OwenAmburOwen.Ambur@verizon.net