# TamerLib TamerLib is a library that allows PHP programs to compute with parallel processing using sub-processes or remote processes as workers where a client can push tasks to the workers and optionally get the results back, all the tasks are executed in parallel in the background. TamerLib works similarly to asynchronous programming but instead of using callbacks, it uses a task queue and a result queue which is more convenient and doesn't result in callback hell. You can either run Tamer locally to simply super-charge your application ``` ┌────────┐ ┌──────────────┐ │ │ │ │ │ Client ├──────►│ Redis Server │ │ │ │ │ └───┬────┘ └──────────────┘ │ │ ┌─────┬─────┬─┴───┬─────┬─────┬─────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌─▼─┐ ┌─▼─┐ ┌─▼─┐ ┌─▼─┐ ┌─▼─┐ ┌─▼─┐ ┌─▼─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ Workers ``` Or you could create Tamer Nodes which are basically Tamer Clients only acting as a supervisor for its workers while all the workers & clients communicate with a central Redis Server, allowing your software to scale horizontally. ``` ┌───────────────┐ ┌───────────────┐ │ │ │ │ │ Tamer Client │ │ Tamer Node │ │ │ │ │ ├───────────────┼─────┐ ┌─────►───────────────┤ │ │ │ │ │ │ │ Server Node │ │ ┌─┼─────┤ Server Node │ │ ◄──┐ │ │ │ │ │ └───────────────┘ │ │ │ │ └───────────────┘ │ │ │ │ ┌───────────────┐ │ │ ┌───────────────┐ │ │ ┌───────────────┐ │ │ │ │ │ │ │ │ │ │ │ Tamer Client │ │ │ │ Redis Server │ │ │ │ Tamer Node │ │ │ │ │ │ │ │ │ │ │ ├───────────────┼──┼──┼────► ├───┼─┼─────►───────────────┤ │ │ │ │ │ │ │ │ │ │ │ Server Node │ │ │ │ Server Node │ │ │ │ Server Node │ │ ◄──┼──┼────┤ ◄─┬─┴─┼─────┤ │ └───────────────┘ │ │ └───────────────┘ │ │ └───────────────┘ │ │ │ │ ┌───────────────┐ │ │ │ │ ┌───────────────┐ │ │ │ │ │ │ │ │ │ Tamer Client │ │ │ │ │ │ Tamer Node │ │ │ │ │ │ │ │ │ ├───────────────┼──┼──┘ │ └─────►───────────────┤ │ │ │ │ │ │ │ Server Node │ │ │ │ Server Node │ │ ◄──┘ └─────────┤ │ └───────────────┘ └───────────────┘ ``` This README will contain documentation on how to effectively use TamerLib in your PHP application ## Table of Contents * [TamerLib](#tamerlib) * [Table of Contents](#table-of-contents) * [Requirements](#requirements) * [Installation](#installation) * [Building TamerLib](#building-tamerlib) * [Introduction](#introduction) * [Implementing a client](#implementing-a-client) * [Implementing a worker](#implementing-a-worker) * [Implementing a node](#implementing-a-node) * [Job and Error Handling](#job-and-error-handling) * [Methods](#methods) * [Global Methods](#global-methods) * [initialize](#initialize) * [shutdown](#shutdown) * [getMode](#getmode) * [monitor](#monitor) * [Client Methods](#client-methods) * [createWorker](#createworker) * [do](#do) * [dof](#dof) * [wait](#wait) * [waitFor](#waitfor) * [doWait](#dowait) * [clear](#clear) * [Worker Methods](#worker-methods) * [addFunction](#addfunction) * [removeFunction](#removefunction) * [getFunctions](#getfunctions) * [run](#run) ## Requirements * [ncc](https://git.n64.cc/nosial/ncc) (For building & installing TamerLib) * PHP 8.0+ * redis-server (Optional for stand-alone clients) * php-redis extension ## Installation To install TamerLib using ncc, run the following command: ``` ncc package install -p "nosial/libs.tamer=latest@n64" ``` if you don't have the N64 package repository added to ncc, you can add it by running: ``` ncc source add --name n64 --type gitlab --host git.n64.cc ``` To add the package as a dependency to your project, add this in `project.json` ```json { "name": "net.nosial.tamerlib", "version": "latest", "source_type": "remote", "source": "nosial/libs.tamer=latest@n64" } ``` ## Building TamerLib To build the TamerLib package, run the following command: ```shell ncc build --config="release" ``` This will create a ncc binary package under `build/release`, to install the package, run: ```shell ncc package install --package="build/release/net.nosial.tamerlib.ncc" ``` or optionally you can use the [Makefile](Makefile) to build & install the package: ```shell make clean build install ``` # Introduction Once TamerLib is installed and available on your system, you can start using it in your PHP application, below is a simple demonstration on how TamerLib could be used in a real-world application. This is an implementation or usage of TamerLib as a client, to demonstrate how TamerLib can be implemented into your application's code structure. ```php email_client = new EmailClient(); try { // Try to initialize as a worker, if the process isn't in worker mode, this will throw an exception // so we can try to initialize as a client \TamerLib\tm::initialize(\TamerLib\Enums\TamerMode::WORKER); // Register functions that can be called by the client \TamerLib\tm::addFunction('sendEmail', [$this->email_client, 'sendEmail']); } catch(TamerException $e) { // Initialize TamerLib as a client (This will also spawn a redis-server process) \TamerLib\tm::initialize(\TamerLib\Enums\TamerMode::CLIENT); // Spawn workers \TamerLib\tm::createWorker(8, __DIR__ . DIRECTORY_SEPARATOR . 'ExampleApplication.php'); } } /** * @param string $to * @param string $subject * @param string $message * @return void */ public function sendEmail(string $to, string $subject, string $message): void { // If we are in client mode, we run this job in the background if(!\TamerLib\tm::getMode() === \TamerLib\Enums\TamerMode::CLIENT) { // Send an email using a worker, do and forget. \TamerLib\tm::dof('sendEmail', $to, $subject, $message); return; } // This can throw an exception but Tamer will catch this and return it to the client // and re-throw the exception on the client side. Just as you were calling this // function directly, if you are expected to handle exceptions from this function. $email_client->send($to, $subject, $message); } } ?> ``` The example above demonstrates a very simple stand-alone example that shouldn't be really used in production, but it demonstrates how TamerLib can be used in your application. In the example above, we have a class called `ExampleApplication` that has a method called `sendEmail` that sends an email using an email client. The interesting part is the constructor, where we initialize TamerLib as a worker, if the process is not in worker mode, we initialize TamerLib as a client and spawn 8 workers using the current file as the worker file. Because when TamerLib tries to run the current file as a worker, the constructor will be called again, but this time the first call to initialize as a worker wouldn't throw an exception because the process is already in worker mode. So we could use this class like this in the front-end ```php // Initialize the application $app = new ExampleApplication(); if(TamerLib\tm::getMode() === \TamerLib\Enums\TamerMode::WORKER) { // We are in worker mode, so we listen to jobs and execute them \TamerLib\tm::run(); } // Otherwise we are in client mode, so we can call the sendEmail function several times $app->sendEmail('johndoe@example.com', 'Hello John', 'How are you?'); $app->sendEmail('johndoe@example.com', 'Hello John', 'How are you?'); $app->sendEmail('johndoe@example.com', 'Hello John', 'How are you?'); $app->sendEmail('johndoe@example.com', 'Hello John', 'How are you?'); // Then we wait for all jobs to finish \TamerLib\tm::wait(); ``` The execution time of a script like this would be reduced by a lot, because the `sendEmail` function is executed in the background by a worker, so the script doesn't have to wait for each email to be sent before it can continue to the next one. This is one approach if you want to implement a client & worker into one class. Though it is recommended to read the documentation below and implement TamerLib in the way that suits your application's design approach best, for example - TamerLib can be implemented as a stand-alone application that both is responsible for monitoring its workers & the redis-server process, this allows you to implement parallel processing into one application without needing to setup a dedicated redis-server process. - You can also implement it as a distributed node system, where one or more machines are running your application in a node-like structure, where each node is responsible for its own workers but all clients and workers connect to a central redis-server that is hosted on a dedicated machine. ## Implementing a client Finally, we have a worker but we need something to push jobs to it, this is where the client comes in. The client is responsible for pushing jobs to the worker. Optionally it can also be used to receive the result of the job. Basically this is your application. Let's begin with initializing the client: ```php require 'ncc'; import('net.nosial.tamerlib'); $server_configuration = new ServerConfiguration('127.0.0.1', 6379, 'optional_password', 0); \TamerLib\tm::initialize(\TamerLib\Enums\TamerMode::CLIENT, $server_configuration); \TamerLib\tm::createWorker(8, __DIR__ . DIRECTORY_SEPARATOR . 'worker.php'); ``` `tm::initialize` takes two arguments, the first one is the mode, which has to be `TamerMode::CLIENT` for the client, the second argument is the server configuration, which is optional. If you don't specify a server configuration, the client will generate its own configuration and spawn its own redis server instance that it controls privately. If you already have a central redis server running that you want multiple servers to connect to, you can specify the server configuration as shown above. `tm::createWorker`, spawns a worker process, the first argument is the amount of workers to spawn, the second argument is the optional path to the worker script. If you don't specify a path, the client will use a generic subproc worker that doesn't do anything at the moment. > **Note:** The use of `tm::createWorker` is optional for clients, in some cases you may only have "nodes" running that is only responsible for supervising workers and nothing more, see [Implementing a Node](#implementing-a-node) for more details on how Nodes work and when to use them. After initializing TamerLib as a client, clients are able to push jobs to a worker and listen to it's return channel for notifications about jobs being completed, allowing you to implement TamerLib calls into your application's execution flow similarly to how you would approach asynchronous programming with some altered concepts. First, let's call the `sleep` function on the worker we created earlier several times to demonstrate a quick example ```php $job1 = \TamerLib\tm::do('sleep', 5); $job2 = \TamerLib\tm::do('sleep', 5); $job3 = \TamerLib\tm::sleep(5); \TamerLib\tm::wait(function($job_id, $result){ echo "Job $job_id completed with result $result\n"; }); ``` [`do()`](#do) is used to push a function call to a worker, internally the job ID is watched so that when the [`wait()`](#wait) method is called, TamerLib knows which job ID to wait for. Alternatively this can be called as `\TamerLib\tm::function_name(...$arguments)` which is a shorthand for `do($function_name, ...$arguments)`. [`wait()`](#wait) is used to wait for all jobs to complete, the callback function it optionally takes is executed whenever a job is completed and gets pushed back into to the client's return channel, this means you will be able to process completed jobs as they come in, instead of waiting for all jobs to complete before processing them. In this example, we call the `sleep` function 3 times, each call returns a job id that we can use to identify the job later. After calling the `sleep` function 3 times, we call [`wait()`](#wait) which takes a callback function as an argument, this callback function will be called when a job is completed, the callback function takes two arguments, the first one is the job id, the second one is the result of the job. In this example, we simply print the job id and the result. > **Note:** Functions can return objects, arrays, strings, integers, floats, booleans, null except resources and > closures. This is because internally TamerLib will serialize the function call parameters and the result of the > function call using PHP's `serialize()` function. ## Implementing a worker First, we implement the worker. A worker is responsible for executing functions and returning the result back to the client. In this example, we will implement a worker that implements the following functions: * `sleep(int $seconds)` - sleeps for a given amount of seconds * `pi(int $iterations)` - calculates pi using the Leibniz formula for a given amount of iterations ```php getMessage() . "\n"; } ``` Including specific exceptions ```php $job = \TamerLib\tm::do('throw_exception'); try { \TamerLib\tm::wait(); } catch(\SmtpException $e) { echo "Caught exception: " . $e->getMessage() . "\n"; } ``` > **Note:** Exceptions are serialized using PHP's `serialize()` function, this means that the exception must be > available in the global namespace of the client application or the same packages must be imported in the client > and worker application. If the exception cannot be unserialized a generic `\Exception` will be thrown instead. Most of TamerLib's static methods are capable of throwing exceptions but usually it will be a `\RuntimeException` or `\InvalidArgumentException` if the method is called with invalid parameters. Crucial methods such as `initialize()` and `run()` will throw a `\TamerLib\Exceptions\TamerException` if the method if there is an error with the TamerLib system itself. # Methods TamerLib provides static methods that can be used to interact with the TamerLib system, these methods are often only applicable depending on the current mode of the process. ## Global Methods These methods can be used in any mode ### initialize > `\TamerLib\tm::initialize(?string $mode, ?ServerConfiguration $server_config=null): void` The initialize method is used to initialize TamerLib in the specified mode - `$mode` - The mode to initialize TamerLib in, use `\TamerLib\Enums\TamerMode` to specify the mode such as `TamerMode::CLIENT` or `TamerMode::WORKER` - `$server_config` - *(Optional)* The server configuration to use, `\TamerLib\Objects\ServerConfiguration` can be constructed and passed on to this method to specify a server configuration, this is only applicable to clients. If no server configuration is specified, the client will generate its own server configuration and spawn its own redis server instance that it controls privately. To initialize TamerLib as a client with this method, you can use the following code: ```php $server_config = new \TamerLib\Objects\ServerConfiguration('127.0.0.1', 6379, 'optional_password', 0); \TamerLib\tm::initialize(\TamerLib\Enums\TamerMode::CLIENT, $server_config); ``` if you just initialize TamerLib without specifying a server configuration, the client will spawn it's own redis server/ ```php \TamerLib\tm::initialize(\TamerLib\Enums\TamerMode::CLIENT); ``` Workers do not need to pass on any sort of configuration as the configuration is obtained by the information passed on by the parent process when the sub-process is created, however initializing TamerLib this way and executing the process directly will result in an exception being thrown. ```php \TamerLib\tm::initialize(\TamerLib\Enums\TamerMode::WORKER); ``` *** ### shutdown > `\TamerLib\tm::shutdown(): void` The shutdown method is used to shutdown TamerLib and release all it's resources and connections ```php \TamerLib\tm::shutdown(); ``` *** ### getMode > `getMode(): string` The getMode method is used to get the current mode of TamerLib, if TamerLib is not initialized, this method will return `TamerMode::NONE` ```php $mode = \TamerLib\tm::getMode(); ``` *** ### monitor > `\TamerLib\tm::monitor(int $timeout=0): void` The monitor method is used to monitor the processes that TamerLib is responsible for, this method is often called internally by TamerLib in methods such as `wait()` and `do()`, however it can be called manually to monitor the processes and obtain their latest updates. Note that this method will print out the latest updates to the console - `$timeout` - *(Optional)* The timeout in seconds to monitor the processes for, if this is set to 0, the method will block indefinitely, or if the value is set to -1 the method will not block at all and will return immediately after it's first iteration. Otherwise the method will block for the specified amount of seconds. ```php \TamerLib\tm::monitor(); ``` *** ## Client Methods These methods are only available if TamerLib is initialized as a client, otherwise an exception will be thrown when attempting to call these methods. ### createWorker > `\TamerLib\tm::createWorker(int $count=8, ?string $path=null, int $channel=0): void` Creates a new worker process with the specified amount & what file to execute - `$count` - *(Optional)* The amount of workers to spawn, defaults to 8 - `$path` - *(Optional)* The path to the file to execute, if this is not specified, the client will use a generic subproc worker that doesn't do anything at the moment. - `$channel` - *(Optional)* The channel for the worker to listen to if it's a subproc worker, this doesn't do anything if `$path` is not specified. To spawn 5 subproc workers that listen to channel 0, you can use the following code: ```php \TamerLib\tm::createWorker(5, null, 0); ``` To spawn 5 workers of a specific file, you can use the following code: ```php \TamerLib\tm::createWorker(5, '/path/to/file.php'); ``` To learn how to create a worker, see the [Implementing a worker](#implementing-a-worker) section. *** ### do > `\TamerLib\tm::do(string $function, array $arguments, int $channel=0): int` Executes a function on a worker and returns the Job ID, this ID is used to identify the job at a later state once it's completed. this method can also be called statically using the function name as a method name. - `$function` - The function to execute on the worker - `$arguments` - The arguments to pass on to the function (can contain classes but they must be serializable, eg. no resources or closures) - `$channel` - *(Optional)* The channel to execute the function on, this is only applicable to some workers that may listen to multiple channels, if this is not specified, the function will be executed on channel 0. This is useful for separating different types of jobs on different channels for different workers. ```php $job_id = \TamerLib\tm::do('sleep', [5]); ``` or ```php $job_id = \TamerLib\tm::sleep(5); ``` *** ### dof > `\TamerLib\tm::dof(string $function, array $arguments, int $channel=0): void` The same as [`do`](#do) but in a "Do and forget" method. This method will send the job packet to the server without a return channel, this essentially means that once that worker has finished executing the job and if fails or succeeds, the job will be dropped regardless and the client will not be notified of the result. This method is useful for executing jobs that do not need to return a result, such as logging or sending emails. ```php \TamerLib\tm::dof('send_email', ['to' => 'johndoe@example.com', 'subject' => 'Hello World', 'body' => 'Hello World']); ``` *** ### wait > `\TamerLib\tm::wait(callable $callback, int $timeout=0): void` This method is responsible for waiting for all dispatched jobs to finish executing, this method will block until all jobs have finished executing or until the timeout has been reached. This is usually called after dispatching one or more jobs and your application needs to wait for the results. **Note:** This method throw an exception if a Job returns an exception - `$callback` - The callback to call when a job has finished executing, this callback will receive the job ID and the result of the job as arguments. - `$timeout` - *(Optional)* The timeout in seconds to wait for, if this is set to 0, the method will block indefinitely, or if the value is set to -1 the method will not block at all and will return immediately after it's first iteration. Otherwise the method will block for the specified amount of seconds. ```php \TamerLib\tm::do('sleep', [5]); \TamerLib\tm::do('sleep', [10]); \TamerLib\tm::do('sleep', [15]); \TamerLib\tm::wait(); ``` or ```php print(\TamerLib\tm::do('sleep', [5]) . PHP_EOL); print(\TamerLib\tm::do('sleep', [10]) . PHP_EOL); print(\TamerLib\tm::do('sleep', [15]) . PHP_EOL); \TamerLib\tm::wait(function($job_id, $result) { echo "Job $job_id finished with result $result" . PHP_EOL; }); ``` *** ### waitFor > `\TamerLib\tm::waitFor(int $job_id, int $timeout=0): mixed` Similar to [`wait`](#wait) but only waits for a specific job to finish executing, this method will block until the job has finished executing or until the timeout has been reached. The return value of this method is the result of the job. **Note:** This method throw an exception if a Job returns an exception - `$job_id` - The ID of the job to wait for - `$timeout` - *(Optional)* The timeout in seconds to wait for, if this is set to 0, the method will block indefinitely, or if the value is set to -1 the method will not block at all and will return immediately after it's first iteration. Otherwise the method will block for the specified amount of seconds. ```php $job_id = \TamerLib\tm::do('sleep', [5]); $result = \TamerLib\tm::waitFor($job_id); // or more simply, exact same thing as running sleep(5) directly $result = \TamerLib\tm::waitFor(\TamerLib\tm::do('sleep', [5])); ``` *** ### doWait > `doWait(string $function, array $arguments, int $channel=0, int $timeout=0): mixed` This method is a combination of [`do`](#do) and [`waitFor`](#waitfor), this method will execute a function on a worker and wait for the result of the job to be returned, this method will block until the job has finished executing or until the timeout has been reached. The return value of this method is the result of the job. **Note:** This method throw an exception if a Job returns an exception - `$function` - The function to execute on the worker - `$arguments` - The arguments to pass on to the function (can contain classes but they must be serializable, eg. no resources or closures) - `$channel` - *(Optional)* The channel to execute the function on, this is only applicable to some workers that may listen to multiple channels, if this is not specified, the function will be executed on channel 0. This is useful for separating different types of jobs on different channels for different workers. - `$timeout` - *(Optional)* The timeout in seconds to wait for, if this is set to 0, the method will block indefinitely, or if the value is set to -1 the method will not block at all and will return immediately after it's first iteration. Otherwise the method will block for the specified amount of seconds. ```php $result = \TamerLib\tm::doWait('sleep', [5]); ``` *** ### clear > `\TamerLib\tm::clear(): void` Clears the internal watch list of jobs, this watch list is used by the [`wait`](#wait) method to determine which jobs to wait for. You normally don't need to call this method as TamerLib will automatically remove jobs from the watch list once it doesn't need it anymore, but if you want to clear the watch list manually, you can call this method. ```php \TamerLib\tm::clear(); ``` *** ## Worker Methods A worker is a process that executes jobs, worker-specific methods are used to configure the worker before running it. ### addFunction > `\TamerLib\tm::addFunction(string $function, callable $callback): void` Registers a callable function to the worker, the function name must be a valid function name and must not be a reserved name such as `do`, `dof`, `wait`, `waitFor`, `doWait` or any other method name of the `tm` class. - `$function` - The name of the function to register - `$callback` - The callback to call when the function is executed, this callback will receive the arguments passed to the function as arguments. Internally this is called using call_user_func_array, so the callback can be a closure or an array containing a class and a method name. ```php \TamerLib\tm::addFunction('sleep', function($seconds) { sleep($seconds); }); ``` or ```php // Example taken from the PHP documentation function foobar($arg, $arg2) { echo __FUNCTION__, " got $arg and $arg2\n"; } class foo { function bar($arg, $arg2) { echo __METHOD__, " got $arg and $arg2\n"; } } \TamerLib\tm::addFunction('foobar', 'foobar'); \TamerLib\tm::addFunction('foobar_the_second', ['foo', 'bar']); ``` *** ### removeFunction > `\TamerLib\tm::removeFunction(string $function): void` Removes a registered function from the worker, this method will throw an exception if the function is not registered. - `$function` - The name of the function to remove ```php \TamerLib\tm::removeFunction('sleep'); ``` *** ### getFunctions > `\TamerLib\tm::getFunctions(): array` Returns an array of all the registered function names. ```php $functions = \TamerLib\tm::getFunctions(); ``` or ```php foreach(\TamerLib\tm::getFunctions() as $function) { \TamerLib\tm::removeFunction($function); } ``` *** ### run > `\TamerLib\tm::run(int|array $channel=0, int $timeout=0): void` Executes the worker in a running state, this method runs in an infinite loop until the worker is killed or the timeout has been reached, this will listen to the specified channel(s) and execute any jobs that are received on the channel(s), if any exception is raised during this process the worker will log the exception and continue running, including Job exceptions which are logged as warnings. - `$channel` - *(Optional)* The channel or an array of channels to listen to, if this is not specified, the worker will listen to channel 0. This is useful for separating different types of jobs on different channels for different workers. - `$timeout` - *(Optional)* The timeout in seconds to wait for, if this is set to 0, the method will block indefinitely, or if the value is set to -1 the method will not block at all and will return immediately after it's first iteration. Otherwise, the method will block for the specified amount of seconds. ```php \TamerLib\tm::run(); ``` or ```php \TamerLib\tm::run([0, 1, 2]); ``` # License TamerLib is licensed under the MIT license, see the [LICENSE](LICENSE) file for more information.