Maximizing processor utilization and minimizing network bandwidth requirements in throughput compute clusters

{
  "publication_number": "US2005060608A1",
  "country": "US",
  "kind": "A1",
  "title": "Maximizing processor utilization and minimizing network bandwidth requirements in throughput compute clusters",
  "abstract": "Exemplary methods and apparatus for improving speed, scalability, robustness and dynamism of data transfers and workload distribution to remote computers are provided. Computing applications, such as Genomics, Proteomics, Seismic, Risk Management require a priori or on-demand transfer of sets of files or other data to remote computers prior to processing taking place. The fully distributed data transfer and data replication protocol of the present invention permits transfers which minimize processing requirements on master transfer nodes by spreading work across the network and automatically synchronizing the enabling and disabling of job dispatch functions with workload distribution mechanisms to enable/disable job dispatch activities resulting in higher scalability than current methods, more dynamism and allowing fault-tolerance by distribution of functionality. Data transfers occur asynchronously to job distribution allowing full utilization of remote system resources to receive data for job queues while processing jobs for previously transferred data. Processor utilization is further increased as file accesses are local to systems and bear no additional network latencies that reduce processing efficiency.",
  "claims": [
    "1. A method comprising: transferring data with a workload distribution mechanism between at least two computing devices using a transfer protocol; and synchronizing workload distribution mechanisms with a synchronizer wherein job dispatch functions of at least two computing devices are enabled or disabled.",
    "2. The method of claim 1 wherein the transfer protocol comprises a multicast protocol.",
    "3. The method of claim 1 wherein the transfer protocol comprises a broadcast protocol.",
    "4. The method of claim 1 wherein transferring data is used for transferring already transferred data from one of the at least two computing devices to a newly connected computing device.",
    "5. The method of claim 1 wherein transferring data is used for completing interrupted data transfers.",
    "6. The method of claim 1 wherein the transferred data comprises segments of a file.",
    "7. The method of claim 1, further comprising recording received data and received jobs in a log at each computing device of said at least two computing devices.",
    "8. The method of claim 1, further comprising performing a security check on a job description file to validate a request.",
    "9. The method of claim 8 wherein validation comprises file access permissions.",
    "10. The method of claim 8 wherein validation comprises execution permissions.",
    "11. A computing device for transferring data and synchronizing workload distributions comprising: a data transfer module configured for transferring data to a second computing device using a transfer protocol; and a synchronization module configured for synchronizing work load distribution mechanisms and enabling or disabling a job dispatch function.",
    "12. The computing device of claim 11 wherein the protocol comprises a broadcast protocol.",
    "13. The computing device of claim 11 wherein the protocol comprises a multicast protocol.",
    "14. The computing device of claim 11 further comprising a security module for performing a security check on a job description file to validate a request.",
    "15. The computing device of claim 14 wherein the security module validates file access permissions.",
    "16. The computing of claim 14 wherein the security module validates execution permissions.",
    "17. A computer readable medium having embodied thereon a program, the program being executable by a machine to perform a method of transferring data and synchronizing workload distributions, the method comprising: transferring data based on a data transfer phase between at least two computing devices using a transfer protocol; and synchronizing workload distribution mechanisms based on a synchronization phase wherein job dispatch functions of at least two computing devices are enabled or disabled.",
    "18. The computer readable medium of claim 17 wherein the computer readable medium is executed by an electronic appliance.",
    "19. The computer readable medium of claim 18 wherein the electronic appliance is a personal computer.",
    "20. The computer readable medium of claim 18 wherein the electronic appliance is a cellular phone.",
    "21. The computer readable medium of claim 18 wherein the electronic appliance is a PDA."
  ],
  "description_excerpt": "1. Field of the Invention\n\nThe present invention relates to transferring and replicating data among geographically separated computing devices and synchronizing data transfers with workload distribution management job processing. The invention also relates to asynchronously maintaining replicated data files, synchronizing job processing notwithstanding computer failures and introducing new computers into a network without user intervention.\n\n2. Description of the Related Art\n\nGrid computers, computer farms and similar computer clusters are currently used to deploy applications by splitting jobs among a set of physically independent computers. Disadvantageously, job processing using on-demand file transfer systems reduces processing efficiency and eventually limits scalability. Alternatively, data files can first be replicated to remote nodes prior to a computation taking place, but synchronization with workload distribution systems must then be handled manually; that is, a task administrator reboots a failed node or introduces a new node to the system.\n\nThe existing art as it pertains to address data file transfer and workload distribution synchronization generally falls into four categories: on-demand file transfer, manual file transfer through a point-to-point protocol, manual transfer through a multicast protocol and specialized point-to-point schemes.\n\nTasks can make use of on-demand file transfer apparatus, better known as file servers, Network Attached Storage (NAS) and Storage Area Network (SAN).",
  "cpc": [
    "H04L 67/06",
    "H04L 12/1877",
    "H04L 67/1095",
    "H04L 69/329",
    "H04L 9/40"
  ],
  "ipc": [
    "H04L 12/18",
    "H04L 29/06",
    "H04L 29/08"
  ],
  "assignees": [
    "EXLUDUS TECHNOLOGIES Inc"
  ],
  "inventors": [
    "Benoit Marchand"
  ],
  "filing_date": "2004-07-16",
  "publication_date": "2005-03-17",
  "priority_date": "2002-05-23",
  "application_number": "US-89375204-A",
  "family_id": "34279326",
  "cited_by_count": 83,
  "citations": [
    "US3905023A",
    "US4130865A",
    "US4228496A",
    "US4412281A",
    "US4569015A",
    "US4644542A",
    "US4718002A",
    "US6279029B1",
    "US5459725A",
    "US5845077A",
    "US6327617B1",
    "US20020016956A1",
    "US6073214A",
    "US5764875A",
    "US5944779A",
    "US5905871A",
    "US6031818A",
    "US6247059B1",
    "US6351467B1",
    "US6278716B1",
    "US6112323A",
    "US6505253B1",
    "US6418554B1",
    "US20030145317A1",
    "US6256673B1",
    "US6415312B1",
    "US6370565B1",
    "US6801949B1",
    "US6753857B1",
    "US6601763B1",
    "US6957186B1",
    "US6446086B1",
    "US6952741B1",
    "US6567929B1",
    "US6640244B1",
    "US7181539B1",
    "US7062556B1",
    "US6557111B1",
    "US7058601B1",
    "US7340532B2",
    "US6704842B1",
    "US6987741B2",
    "US6990513B2",
    "US6522650B1",
    "US6965938B1",
    "US20040030787A1",
    "US7418522B2",
    "US7421505B2",
    "US6816897B2",
    "US20030182358A1",
    "US20070168478A1",
    "US20080201414A1"
  ]
}