# ncc Documentation This document serves the purpose of presenting the documentation for using/developing ncc, from basic installation, basic usage, standards, and much more. ## Table of contents * [ncc Documentation](#ncc-documentation) * [Table of contents](#table-of-contents) * [Introduction](#introduction) * [What is ncc?](#what-is-ncc) * [How does ncc work?](#how-does-ncc-work) * [What do I need to use ncc?](#what-do-i-need-to-use-ncc) * [How do I get started?](#how-do-i-get-started) * [Building & installing ncc](#building--installing-ncc) * [Building from source](#building-from-source) * [Requirements to build](#requirements-to-build) * [Installing phpab](#installing-phpab) * [Building ncc](#building-ncc) * [Redist](#redist) * [Tar](#tar) * [Debian](#debian) * [Building ncc for docker](#building-ncc-for-docker) * [Installing ncc](#installing-ncc) * [Command line arguments](#command-line-arguments) * [Uninstalling ncc](#uninstalling-ncc) * [Helpful Information](#helpful-information) * [Package Naming](#package-naming) * [Remote Package Syntax (RPS)](#remote-package-syntax-rps) * [ncc Binary Package](#ncc-binary-package) * [Package Header](#package-header) * [Package Data](#package-data) * [How to build binary executable files](#how-to-build-binary-executable-files) * [How are they compiled?](#how-are-they-compiled) * [Can I adjust the compiler process?](#can-i-adjust-the-compiler-process) * [What does an executable build configuration look like?](#what-does-an-executable-build-configuration-look-like) * [ncc cli](#ncc-cli) * [Project Management (project)](#project-management-project) * [Creating a new project (create)](#creating-a-new-project-create) * [Applying Templates (template)](#applying-templates-template) * [phpcli template](#phpcli-template) * [phplib template](#phplib-template) * [Install Dependencies (install)](#install-dependencies-install) * [Package Management (package or pkg)](#package-management-package-or-pkg) * [Listing Installed Packages (list)](#listing-installed-packages-list) * [Installing Packages (install)](#installing-packages-install) * [Uninstalling Packages (uninstall)](#uninstalling-packages-uninstall) * [Uninstalling All Packages (uninstall-all)](#uninstalling-all-packages-uninstall-all) * [Fix Broken Packages (fix-broken)](#fix-broken-packages-fix-broken) * [Credentials Management (cred)](#credentials-management-cred) * [Adding a credential (add)](#adding-a-credential-add) * [Removing a credential (remove)](#removing-a-credential-remove) * [Listing credential entries (list)](#listing-credential-entries-list) * [Managing Repositories (repository or repo)](#managing-repositories-repository-or-repo) * [Adding a repository (add)](#adding-a-repository-add) * [Removing a repository (remove)](#removing-a-repository-remove) * [Listing repositories (list)](#listing-repositories-list) * [Inspecting a package (ins)](#inspecting-a-package-ins) * [Package Information (info)](#package-information-info) * [Package Headers (headers)](#package-headers-headers) * [Package Metadata (metadata)](#package-metadata-metadata) * [Assembly Information (assembly)](#assembly-information-assembly) * [Package Dependencies (dependencies)](#package-dependencies-dependencies) * [Execution Units (execution_units)](#execution-units-executionunits) * [Building Projects (build)](#building-projects-build) * [Execute (exec)](#execute-exec) * [Project Configuration (package.json)](#project-configuration-packagejson) * [Root Section](#root-section) * [Project (object)](#project-object) * [Compiler (object)](#compiler-object) * [UpdateSource (object)](#updatesource-object) * [Repository (object)](#repository-object) * [Assembly (object)](#assembly-object) * [Build (object)](#build-object) * [Dependency (Object)](#dependency-object) * [BuildConfiguration (Object)](#buildconfiguration-object) * [ExecutionPolicy (object)](#executionpolicy-object) * [Execute (object)](#execute-object) ------------------------------------------------------------------------------------------------------------------------ ## Introduction This section serves the basic introduction of ncc, what it's used for and how you can use it in your own projects or use it to run and build other projects that are designed to be used with ncc. ### What is ncc? ncc (*Acronym for **N**osial **C**ode **C**ompiler*) is a multipurpose compiler, package manager and toolkit. Allowing projects to be managed and built more easily without having to mess with all the traditional tools that comes with your language of choice. Right now ncc only supports PHP as it's written in PHP but extensions for other languages/frameworks can be built into the software in the future when the need comes for it. ncc can make the process of building your code into a redistributable package much more efficient by treating each building block of your project as a component that is interconnected in your environment instead of the more popular route taken by package/dependency managers such as [composer](https://getcomposer.org/) which attempts to copy what [npm](https://www.npmjs.com/) does but for PHP, which is not a bad thing, but it's not the best approach for PHP and its ecosystem. ### How does ncc work? ncc's command-line interface serves as the central hub for managing your projects and packages. This CLI empowers you to perform tasks like project creation, package installation, and project building. When you compile your project, it generates a "ncc package" file with a ".ncc" extension. This unique file consolidates all essential project files, including the utilized components. Optionally, you have the flexibility to create a static version of your project that contains all dependencies and components, simplifying deployment and execution across different machines through a single file. ncc harnesses these packages to install its dependencies and components globally, akin to how Composer installs packages into a project's vendor directory. However, in ncc's case, it installs them into a global directory accessible to other projects. Moreover, ncc can retrieve packages from various remote sources, including GitHub, GitLab, Gitea, and even Packagist. If ncc cannot locate a specific ncc package for a dependency or package, it will attempt to build it from source using the package's source code. Additionally, ncc features a compatibility layer for Composer packages, enabling you to install them without needing to install or use Composer itself. You can imagine ncc as apt-get for PHP, but with a few extra features that make it more powerful and flexible. ### What do I need to use ncc? ncc is a command-line tool, so you will need to be familiar with using the command-line interface. You will also need to have PHP 8.0+ installed on your machine, along with the following PHP extensions: - php-mbstring - php-ctype - php-common (covers tokenizer & posix among others) - php-zip These extensions are required for ncc to function properly, if you don't have these extensions installed, ncc may not work correctly or may suffer from performance issues. For instance, `php-ctype` & `php-mbstring` are required for ncc however, they will still work without them thanks to Symfony's polyfill library, but it's recommended to install the extensions for better performance. ### How do I get started? [RTFM](https://en.wikipedia.org/wiki/RTFM), this documentation is a good place to start, it covers everything you need to know about ncc and how to use it. ------------------------------------------------------------------------------------------------------------------------ # Building & installing ncc ncc must be built from source before it can be installed, this is because ncc is a PHP application and PHP applications are not compiled into machine code, instead, they are compiled into a redistributable source that can be installed on the machine. This includes the auto-loader files that ncc needs to locate its components and dependencies. ## Building from source Building ncc from source is easy with very few requirements to start building. At the moment, ncc can only be debugged or tested by building a redistributable source and installing it. ### Requirements to build - php8.0+ - php-mbstring - php-ctype - php-common (covers tokenizer & posix among others) - make - phpab - tar *(optional)* For building different variants, such as building a debian package, you will need to install the required tools for that specific variant. For more information, check the [Makefile](Makefile) for the required tools for a specific variant. ### Installing phpab phpab is also known as [PHP Autoload Builder](https://github.com/theseer/Autoload), phpab is an open-source tool used for creating autoload files, ncc needs this tool in order to generate its autoload files whenever there are any changes to its source code. This tool is only required for building and or creating a redistributable package of ncc. This component is not required to be installed to use ncc. for some components that require static loading, ncc will automatically load it using its own [autoloader](src/autoload/autoload.php) The recommended way to install phpab is by using [phive](https://phar.io/), if you don't have phive installed, you can install it by running these commands in your terminal (from the official documentation) ```shell wget -O phive.phar https://phar.io/releases/phive.phar wget -O phive.phar.asc https://phar.io/releases/phive.phar.asc gpg --keyserver hkps://keys.openpgp.org --recv-keys 0x9D8A98B29B2D5D79 gpg --verify phive.phar.asc phive.phar chmod +x phive.phar sudo mv phive.phar /usr/local/bin/phive ``` Once phive is installed, you can run the final command to install phpab ```shell sudo phive install phpab --global ``` or you can run this command to install it locally ```shell phive install phpab ``` **Note:** Optionally, you may want to have `phab` available in your `$PATH`, this can be done with this command. *(Replace `x.xx.x` with your version number)* this is if you installed it locally ```shell ln -s /home/user/.phive/phars/phpab-x.xx.x.phar /usr/local/bin/phpab ``` ### Building ncc First, navigate to the main directory of ncc's source code where the [Makefile](Makefile) is present. If you already attempted to or had built ncc before, it's recommended to use `make clean` before building. #### Redist Running `redist` from the Makefile will generate all the required autoloader for ncc and move all the required files into one redistributable source folder under a directory called `build/src` ```shell make redist ``` #### Tar Running `tar` will run redist before packaging the redistributable source into a tar.gz file that can be distributed to other machines, this process is not a requirement. ```shell make tar ``` #### Debian Running `deb` will run `redist` before packaging the redistributable source into a debian package that can be installed on debian based machines, this process is not a requirement. ```shell make deb ``` Once you have a populated `build/ncc_x.x.x` folder, you can simply run execute the `installer` file to install your build of ncc onto the running machine. > Note: you may need to run `sudo` before executing the installer file. ------------------------------------------------------------------------------------------------------------------------ ## Building ncc for docker ncc is also available for docker, you can build ncc for docker by running the `docker-debian` or `docker-alpine` make tasks, this will build ncc for docker and create a docker image that you can use to run ncc. ```shell make docker-debian docker-alpine ``` The respective docker files are located in - [Dockerfile Debian](Dockerfile.debian) - [Dockerfile Alpine](Dockerfile) You may also run and test these docker builds with the `docker-debian-run` and `docker-alpine-run` make tasks. ```shell make docker-debian-run docker-alpine-run ``` > Contributor Note: contributions are welcomed here to expand ncc's docker support to other distros and to improve the > existing docker files. ------------------------------------------------------------------------------------------------------------------------ ## Installing ncc Installing ncc is easy, you can either download the redistributable source from the [releases](https://git.n64.cc/nosial/ncc/-/releases) page or you can build it from source using the instructions above. Once you have the redistributable source, you can simply run execute the `INSTALL` file to install ncc onto the running machine. usually this installation process will require root privileges, so it's recommended to run the installer with `sudo` or as root. ### Command line arguments The installer accepts a few command line arguments that can be used to alter the installation process. `--help` Displays the help message `--bypass-cli-check` Bypasses the check in the installer that checks if the installer is being run from the command line, this is useful if you want to install ncc from a script. `--bypass-checksum` Bypasses the checksum check in the installer, this is useful if you made modifications to the installation files and want to install a modified version of ncc. But this isn't recommended, and the proper way to do this is to modify the source code and build ncc from source, the Makefile task will automatically rebuild the checksum file for you. ## Uninstalling ncc Uninstalling ncc is easy, simply delete the directory where ncc was installed to, by default this is `/etc/ncc`. It's recommended to run `ncc package --uninstall-all` before uninstalling ncc, this will uninstall all the packages that were installed using ncc and remove any artifacts that is installed on the system such as symlink registrations and so on. **Note:** - To delete all the data that ncc has created, you can also delete the `/usr/share/ncc` directory. - Finally, remove the symlink that was created in `/usr/bin`to the `ncc` entry point file. ------------------------------------------------------------------------------------------------------------------------ # Helpful Information This section covers helpful information about ncc, such as conventions, standards, and so on. ## Package Naming ncc follows a package naming convention that is inspired by Java's package naming convention. This convention is designed to ensure clarity, avoid naming conflicts, and make it easier to identify the origin of packages. Below are the key rules for naming packages in ncc: - The package name must be in all lowercases - The package name must be in reverse domain notation - The package name must be separated by a dot `.` - The package name must not contain any special characters other than '-' or '_' - The package name must not contain any spaces In PHP, similar to Java, package naming conventions follow the reversed domain notation. This means that a package name, such as "symfony/process" in Composer, is transformed into "com.symfony.process" in PHP. The components of the package name are separated by dots (".") to create a hierarchical structure. It's essential to adhere to specific guidelines when naming packages in PHP: - **Character Limitation**: Package names should consist of only hyphens ("-") or underscores ("_") as special characters. This restriction ensures compatibility with both file systems and package management tools. - **Avoiding Spaces**: Spaces should be avoided in package names. The inclusion of spaces can lead to confusion and compatibility issues. Instead, use hyphens or underscores if spacing is necessary within a package name. ## Remote Package Syntax (RPS) Remote packages are packages that are hosted on a remote source, such as GitHub, GitLab, Gitea, and so on. ncc uses a special syntax for specifying remote packages, this syntax is called Remote Package Syntax or RPS for short. This syntax is simply a query to tell ncc what package to install and where to install it from. This syntax is used when installing packages from the command-line or defining dependencies in a project's package.json file. The syntax for RPS is as follows: ``` /=@ ``` | Component | Description | |----------------|---------------------------------------------------------------------------------------------------------------| | `` | The vendor name of the package, this is usually the username or organization name on the source eg; symfony | | `` | The package name, this is the name of the package eg; console | | `` | The version of the package to install, this can be a version number or simply "latest" for the latest version | | `` | The repository to install the package from, this has to be a name of a repository that's configured in ncc | **Note:** The version number can be omitted, in which case ncc will install the latest version of the package. Here are some examples of RPS: ```text symfony/console=latest@packagist # installs the latest version of symfony/console from packagist johndoe/hello_world=latest@github # installs the latest version of hello_world from github ``` For instances like Gitlab where organizations may have subgroups, you can specify the subgroup by using a dot (".") to separate the group name from the subgroup name. ```text nosial/libs.config@n64 # installs the latest version of ConfigLib from n64 nosial/libs.config=1.0.0@n64 # installs version 1.0.0 of ConfigLib from n64 ``` ## ncc Binary Package ncc binary packages are packages that are compiled by ncc, these packages are used to install packages on the system and can also be executed directly by ncc, the file format is a flexible memory safe archive type similiar to `PHAR` or the `ZIP` file format. This section will attempt to explain the file sturcture of a binary package and how they work. ![ncc binary package file structure](assets/package_structure.png) The ncc Binary Package File consists of three main sections: the package header, the package data, and the end of package marker. ### Package Header The package header is the data that ncc will load into memory to be able to find data on the package without needing to load the entire package into memory. The package begins with the magic bytes `ncc_pkg` followed by the header data, this header data ends with the sequence of bytes `1F 1F 1F 1F` to indicate the end of the header data. ![ncc binary package file structure](assets/header_structure.png) Once ncc is able to locate and decode the header data, it's able to identify where all the files and components are located in the package, this is because the header data provides an offset and length for each component. It's up to ncc's PackageReader to determine the correct offset to be able to read the data correctly, as ncc binary packages may also be embedded into other files such as an executable file. The header data is encoded using `msgpack` to ensure that the data is compact and can be decoded quickly, the header cannot be compressed as it's required to be loaded into memory as-is. ### Package Data After the end of the header marker, the package data begins, this is where all the files and components are located, each component and file and it's data contents are not seperated or easily identifiable, this is because the package data is located using the header data. The data section ends with the sequence of bytes `FF AA 55 F0` to indicate that this is the end of the ncc binary package data, any data beyond that is not part of the package file. ![ncc binary package file structure](assets/data_structure.png) The data sections can be any sort of data. Additionally, if the headers contain a flag for compression, ncc will attempt to decompress the data as you read it, this is useful for reducing the size of the package file and to reduce the amount of memory required to load the package into memory. Though this is optional and is not required for ncc to be able to read the package file. ## How to build binary executable files ncc can build binary executable files for your project, this is useful for CLI applications or for creating a single file that can be directly executed on any machine or even used as a dependency for other projects. This section will attempt to explain how to build binary executable files and more importantly, how they work. > **Note:** This process requires `gcc` to be installed on the system. ### How are they compiled? ncc utilizes a bootstrap approach to transform your project into a binary executable file. However, it's important to note that your project won't be able to function independently without ncc. This dependency exists because ncc needs to load your project's components and dependencies into memory before execution, this also requires php to actually execute the code. The key component for this process is the bootstrap file, which is located at [src/ncc/Classes/PhpExtension/bootstrap_main.c](src/ncc/Classes/PhpExtension/bootstrap_main.c). ![ncc compiler process](assets/execution_process.png) Here's how the compiler process unfolds: initially, it generates an ncc binary package file for your project. To achieve this, your executable build configuration must include an option called `ncc_configuration`. This option anticipates a value where the key corresponds to the name of a build configuration that produces an ncc binary package file. Once the ncc binary package file is created, ncc proceeds to convert it into a hexadecimal representation. Following the creation of the hex representation of the binary package file, ncc invokes the gcc compiler. This, in combination with the hex representation of the binary package file, results in the creation of an executable file. If your package contains a designated main execution point, this executable file can be run directly. If your package lacks a main execution point, the executable file will function as a library and can be employed as a dependency in other projects or imported during runtime. ![ncc compiler process](assets/compiler_process.png) > **Note:** If the program requires dependencies that are not available on the system, direct execution will fail. In > such cases, you should add the `static` option to the build configuration to create a static executable file. This > file will contain all the dependencies and components required to run the program. ### Can I adjust the compiler process? You have the flexibility to customize the compiler process by providing additional parameters to gcc. This customization can be achieved by modifying the `options` property within your executable build configuration, with each option name prefixed by gcc-, for example: If you need to pass a key-value option, include it like this: `{ "gcc-