Scheduling

Scheduling is meant to automate the processes of data loading and transformation. The term usually describes a batch processing in which a single job is triggered at a specified time or event. A job may comprise one or many individual sub-steps which are linked by dependencies. The jobs and the sub-steps may be run sequentially or in parallel depending on the application needs. An optimal scheduling solution will allow to efficiently utilize the shared resources, and simultaneously to meet the service level agreements.

There are numerous commercial as well as open source tools available in the market. These tools allow create and submit jobs as well as to manage and control resources. Majority of them provide a user friendly GUI. Cron is one of the examples of a simple scheduler, implemented in Unix systems. Cron allows to periodically run jobs in the background at a specified time. Other, more flexible solutions, are listed below:

• CONTROL-M (commercial BMC Software)
• COSbatch (commercial, OSM)
• Cronacle (commercial, Redwood Software)
• Tivoli Workload Scheduler (commercial, IBM)
• Automation Center (commercial, OpsWise)
• JobSheduler (Open Source, SOS-Berlin)
• TORQUE Resource Manager (Open Source)

One has to take into account a number of factors when selecting the best scheduler for the job. These include:

• User interface (configuration files, www services, GUI)
• Supported operating systems and DBMS's
• Job queuing, resources sharing
• Jobs management (monitoring, logging restarting capabilities, manual job triggering, incident notification)
• Handling distributed applications which are working on many servers and different operating systems
• Metadata (backup and recovery, easy migration)