Caveat: This document is really two different documents in one. Much of the expository text (rationale, extended descriptions) belongs in an overview, background, or introductory document. The more concise “man pages” belong in an API reference manual. And, of course, a tutorial is needed. Lacking a tutorial, we suspect the best thing for new users to do is to scan this introductory chapter and then look over and play with some of the small example programs, outlined below in section 1.6.1. Feedback appreciated, and bear with us.
The OSKit is a framework and set of modularized components and library code, together with extensive documentation, for the construction of operating system kernels, servers, and other OS-level functionality. Its purpose is to provide, as a set of easily reusable modules, much of the infrastructure “grunge” that usually takes up a large percentage of development time in any operating system or OS-related project, and allow developers to concentrate their efforts on the unique and interesting aspects of the new OS in question. The goal is for someone to be able to take the OSKit and immediately have a base on which they can start concentrating on “real” OS issues such as scheduling, VM, IPC, file systems, security, or whatever. Alternately they can concentrate on the real language issues raised by using advanced languages inside operating systems, such as Java, Lisp, Scheme, or ML--instead of spending six months first writing boot loader code, startup code, device drivers, kernel printf and malloc code, a kernel debugger, etc. With the recent addition of extensive multithreading and sophisticated scheduling support, the OSKit also provides a modular platform for embedded applications.
Although it can provide a complete OS environment for many domains, the primary intention of this toolkit is not to “write the OS for you”; we certainly want to leave the OS writing to the OS writer. The dividing line between the “OS” and the “OS toolkit,” as we see it, is basically the line between what OS writers want to write and what they would otherwise have to write but don’t really want to. Naturally this will vary between different OS groups and developers. If you really want to write your own x86 protected-mode startup code, or have found a clever way to do it “better,” you’re perfectly free to do so and simply not use the corresponding code in our toolkit. However, our goal is that the toolkit be modular enough that you can still easily use other parts of it to fill in other functional areas you don’t want to have to deal with yourself (or areas that you just don’t have time to do “yet”).
As such, the toolkit is designed to be usable either as a whole or in arbitrary subsets, as requirements dictate. It can be used either as a set of support libraries to be linked into an operating system kernel and/or its support programs, or it can be used merely as a collection of “spare parts”: example source code to be ripped apart and cannibalized for whatever purpose. (Naturally, we prefer that the toolkit be used in library fashion, since this keeps a cleaner interface between the toolkit and the OS and makes them both easier to maintain; however, we recognize that in some situations this will not be practical for technical or political reasons.)
The toolkit is also intended to be useful for things that aren’t kernels but are OS-related, such as boot loaders or OS-level servers running on top of a microkernel.
The facilities provided by the OSKit are currently organized into three main categories, corresponding to the three main sections of this manual: interfaces, function libraries, and component libraries. Note that the distinction between the two types of libraries has lessened since the majority of this document was written, and in some cases is rather arbitrary. There is also a small Utilities section (V) describing useful OSKit programs; this currently describes only the “NetBoot” utility.
The OSKit’s interfaces are a set of clean, object-oriented interfaces specified in the framework of the Component Object Model (COM), described in Chapter 4. These interfaces are made available through thoroughly commented public C header files, with corresponding documentation in Part II of this manual. For example, the OSKit provides a “block I/O” interface for communication between file systems and disk device drivers, a “network I/O” interface for communication between network device drivers and protocol stacks, and a file system interface similar to the “VFS” interface in BSD. These interfaces are used and shared by the various OSKit components in order to provide consistency and allow them to be used together easily. The OS developer may additionally use or build on these interfaces in defining the fundamental structure of the client OS, but is not required to do so; alternatively, the developer may simply use them when writing the “glue” code necessary to incorporate a particular OSKit component into a new or existing OS environment.
The OSKit’s function libraries provide basic low-level services in a traditional C-language function-oriented style. For example, the OSKit provides libraries exporting kernel bootstrap support, standard C library functions, and executable program loading. These libraries are designed to be usable and controllable in a very fine-grained manner, generally on a function-by-function basis, allowing the client OS to easily use particular library functions while leaving out or individually overriding other functions. The dependencies between library functions are minimized, as are dependencies of library functions on other libraries; where these dependencies inevitably exist, they are well-defined and explicitly exposed to the OS developer. In general, the implementation details of these libraries are intentionally exposed rather than hidden. The function libraries make only minimal use of the OSKit’s object-oriented COM interfaces, instead generally defining their own function-oriented interfaces in ordinary C header files. This design strategy provides maximum power and flexibility to the OS developer at the cost of increasing the dependence of the client OS on the implementation details of the libraries; we have found this to be a good design tradeoff for the low-level OSKit facilities which usually must be customized extensively in order to fit into any particular OS environment.
Following is a summary of the function libraries currently provided by the OSKit along with the chapter numbers in which each is described:
The following four libraries are provided mainly for convenience; they are not as general or as well documented as the other libraries but are still useful. See the source directories and READMEs for details.
Finally, the OSKit’s component libraries provide generally higher-level functionality in a more standard, coarse-grained, object-oriented “black box” design philosophy. Although the OSKit’s “components” are also packaged into ordinary link libraries by default, their structure represents more of a component-oriented design than a traditional C function-oriented design. In contrast with the OSKit’s function libraries, the component libraries typically export relatively few public entrypoints in comparison to the amount of functionality provided. For example, in the Linux and BSD device driver component libraries (see Chapters 43 and 44), each entire device driver is represented by a single function entrypoint which is used to initialize and register the driver as a whole. The client OS generally interacts with these components through the OSKit’s object-oriented COM interfaces, allowing many components and instances of components to coexist and interact as defined by the OS developer. This design strategy has proven to be most appropriate when incorporating large chunks of existing code from existing systems such as BSD and Linux, where it is more important to hide the details of the original environment than to provide the most flexibility.
Following is a summary of the component libraries currently provided by the OSKit along with the chapter numbers in which each is described:
The Simple Process library uses the NetBSD UVM library to provide support for implementing simple process, including running a thread in user mode, system calls and loading an ELF executable file.
This section describes some of the general principles and policies we followed in designing the components of the OSKit. This section is most relevant for people developing or modifying the toolkit itself; however, this information may also help users of the toolkit to understand it better and to be able to use it more effectively.
The downside of this policy is that exposing the internal composition of the libraries this way leaves less room for the implementation to change later without affecting the client-visible interfaces. However, we felt that for the purposes of the OSKit, allowing the client more flexibility in using the library is more important than hiding implementation details.
The OSKit follows the GNU conventions for configuration, building, and installation; see the INSTALL file in the top-level source directory for general instructions on using GNU configure scripts. In short, you need to run the configure script that is in the top-level source directory of the OSKit; this script will attempt to guess your system type and locate various required tools such as the C compiler. You can configure the OSKit to build itself in its own source directory, simply by moving to that directory and typing ./configure, or you can build the OSKit into a separate object directory by changing to that directory and running the configure script from there. For example, using a separate object directory allows you to put the object files on a local disk if the sources come across NFS, or on a partition that isn’t backed up. Additionally, you can have multiple configurations of the OSKit at once (with different options or whatever), each in its own object tree but sharing the same sources.
To cross-compile the OSKit for another architecture, you will need to specify the host machine type (the machine that the OSKit will run on) and the build machine type (the machine on which you are building the toolkit), using the --build=machine and --host=machine options. Since the OSKit is a standalone package and does not use any include files or libraries other than its own, the operating system component of the host machine type is not directly relevant to the configuration of the OSKit. However, the host machine designator as a whole is used by the configure script as a name prefix to find appropriate cross-compilation tools. For example, if you specify ‘--host=i486-linux’, the configure script will search for build tools called i486-linux-gcc, i486-linux-ar, i486-linux-ld, etc. Among other things, which tools are selected determines the object format of the created images (e.g., ix86-linux-* tools create ELF format, while ix86-mach-* tools create a.out format).
The OSKit’s configure script accepts various standard options; for a full listing of the supported options, run configure --help. In addition, the configure script also supports the following options specific to the OSKit:
Before you do the actual OSKit build, there are some steps you can choose to take to make the build go faster, by not compiling parts you do not need.
Building the OSKit currently requires these tools and versions:
To build the OSKit, go to the top-level source directory (or the top-level object directory, if you configured the toolkit to build in a separate object directory), and run GNU make (e.g., just ‘make’ on Linux systems, or ‘gmake’ on BSD systems). Note that the OSKit requires GNU make. Don’t even think about trying to get it to work with another version of make. You’ll be better off porting GNU make to your system. Really. To avoid confusion, the OSKit’s makefiles are named GNUmakefile rather than just Makefile; this way, if you accidentally run the wrong make utility, it will simply complain that it can’t find any makefile, instead of producing an obscure error.
To build or rebuild only one specific part of the OSKit, such as one of the libraries, you can simply go into the appropriate subdirectory of the object tree and run GNU make from there. The top-level GNUmakefile essentially does nothing except recursively run make in each subdirectory, so it is always safe to run any OSKit makefile manually. A few OSKit directories depend on others being built first--for example, the example kernels cannot be built until the OSKit libraries they use have been generated--but most of the OSKit libraries can be built completely independently of each other.
Once the toolkit is built, you can install it with ‘make install’. By default, the libraries will go into /usr/local/lib and the header files into /usr/local/include, unless you specified a --prefix on the configure command line. All of the OSKit header files are installed in an oskit/ subdirectory (e.g. /usr/local/include/oskit), so they should not conflict with any header files already present. Although the libraries are installed in the main library directory (e.g., /usr/local/lib) and not a subdirectory, all the library names are prefixed with oskit_ to avoid conflicts with other unrelated libraries you may want to place there. For example, the OSKit’s minimal C library is named liboskit_c.a rather than just libc.a, allowing you to install a “real” C library in the same directory if desired.
The standard make variables such as CFLAGS and LDFLAGS are used by the OSKit’s build rules but are not actually defined by any of the OSKit makefiles; thus they are available for use on the make command line. For example, you can type ‘make CFLAGS="-save-temps"’ to cause GCC to leave its intermediate files in the object directory, such as the preprocessed C source and the assembly language compiler output. (Internally, the OSKit makefiles use OSKIT_ prefixes on most standard makefile variables.)
The OSKIT makefiles also support a variable OSKIT_QUIET_MAKE. If this is set to “yes”, as in ‘make OSKIT_QUIET_MAKE=yes’, then the makefile will spit out less verbiage than by default--it will tell you what is going on without telling you how.
To use the OSKit, simply link your kernel (or servers, or whatever) with the appropriate libraries. Detailed information on how to use each library is provided in the appropriate chapters in this document. For initial experimentation with the OSKit, you can simply hack on the example kernels or create new kernels in the same directory (see Section 1.6.1 for a tour through our examples); however, once your own system grows beyond the stage of a simple demo kernel you will probably want to set up a separate source tree and link in the OSKit components from the installation directory.
Linking libraries into a kernel may seem strange at first, since all of the existing OS kernels that we have encountered seem to have a strong “anti-library” do-everything-yourself attitude. However, the linker can link libraries into a kernel just as easily as it can link them into application programs; we believe that the primary reason existing kernels avoid libraries is because the available libraries aren’t designed to be used in kernels; they make too many assumptions about the environment they run in. Filling that gap is the purpose of the OSKit.
All of the OSKit libraries are designed so that individual components of each library can be replaced easily; we have taken pains to document the dependencies clearly so that clients can override whatever components they need to, without causing unexpected results. In fact, in many cases, particularly in the function libraries, it is necessary to override certain functions or symbols in order to make effective use of the toolkit. To override a library function or any other symbol defined by a library, just define your own version of it in your kernel or other client program; the linker will ensure that your definition is used instead of the library’s. We strongly suggest that you use the linker to replace components of the OSKit, instead of making changes directly in the OSKit source (except, of course, to fix bugs in the OSKit). Maintaining a clean separation between the OSKit and your kernel will make things much easier when upgrading to a new version of the OSKit.
If you are starting a new OS kernel, or just want to experiment with the OSKit in a “standalone” fashion, an easy way to begin is with one of the example “kernels” in the examples directory. These examples build up from the kernel equivalent of a “Hello World” application, demonstrating how to use various facilities separately or together, such as the base environment initialization code in kern, the minimal console driver code, the minimal C library, the remote debugging stub, and device, filesystem and network COM interfaces. The code implementing these examples is almost as small and simple as equivalent ordinary user-level applications would be because they fully rely on the OSKit to provide the underlying infrastructure necessary to get started. The compilation and linking rules in the GNUmakerules files for these example programs demonstrate how to link kernels for various startup environments.
The following example “kernels,” and many more, are provided. See the examples/README file for a list of most of them, and their source for more detail.
The example kernels, as well as custom kernels you build using the OSKit, can be booted from either the GRUB, Linux, Mach, or BSD boot loaders, from MS-DOS directly, or from the NetBoot “meta-kernel.” (NetBoot is described in Section 46.) GRUB and NetBoot can boot the kernels as-is, since they directly support the MultiBoot standard, whereas the other boot loaders need the kernel to be in a different format. This conversion can be done with the mkbsdimage, mklinuximage, and mkdosimage “boot adapter” scripts, which are automatically built and installed with the OSKit when configured for the appropriate host. See comments in each script for the argument syntax.
For example, the following command creates a bootable BSD-style image named ‘Image’:
% mkbsdimage hello
the mktypeimage scripts can also do more complex things, such as combining an arbitrary number of additional files or “boot modules” into the image. See 15.14 and the scripts for more info.
For details on the MultiBoot standard see Section 15.14.12.
The various boot adapters convert their respective command-line formats, such as the boothowto word in the BSD boot loader, into the string format used by MultiBoot-compliant operating systems. The OSKit expects this string to be in a certain format, which looks like:
progname [<boot-opts and env-vars> --] <args to main>
Note that if no -- is present then all of the args will be passed to main.
The default OSKit MultiBoot startup code then converts this string into a C-style argv/argc pair, an environ global array, and a set of booting-options in the oskit_bootargv/oskit_bootargc global variables.
The argv/argc pair and the environ array are passed to main, the latter as the third parameter commonly called envp. The booting-options in oskit_bootargv/oskit_bootargc are interpreted by the default OSKit console startup code and the following flags have special meaning:
These flags are decidedly BSD-centric, but that is because at Utah we most commonly boot OSKit kernels from the FreeBSD boot-loader.
In addition, if the NetBoot booting program is being used, then an additional parameter will be present in oskit_bootargv: