Writing kernel in Windows with Visual Studio C/C++

Most hobby osdev projects prefer *nix+gcc combination these days, primarily because there are a bunch of nice tutorials and examples available for them. Considering the flexibility, I too personally think it is a good choice. But if you are a heretic by nature and are curious about how you can osdev with just the express edition of Visual C++ and some other free software without leaving windows (or forced), this post is for you.

In this post I will simply demonstrate the toolchain setup. Complete package can be downloaded from the lower 'Downloads' section. The actual 'kernel' that is written in this tutorial is the lame hello-world one. We will get it working with PXE boot in VirtualBox (replace with any emulator of your choice). It works with default Windows Boot Manager (WBM) without modifications (GRUB equivalent, just lamer) and can easily be modified to work with mbr/vbr, booting with stages. Let's face it - even osdev noobs these days are so over floppies.

Kernel loader (stub)
For a kernel loader stage, there are many ways of entering protected mode, loading the actual kernel image, relocating and passing control to the kernel. Here is one that I think is simple enough to fit in a blogspot post.

The image loaded by PXE/WBM consists of two sections - the stub and the kernel executable image. Both PXE and WBM (are supposed to) load our image at physical address 0x7c00, or 7c0h:0h in real mode. This means that the stub must be 16bit real mode code that is statically linked to run at 7c00h. The kernel executable output by Visual C++ compiler cl.exe on the other hand is 32bit protected mode code. Also, the kernel executable needs to be relocated before it can be run. Thus, the stub must set up pmode, relocate the kernel to the address it is supposed to run and then pass control to it.

The stub in this tutorial is written in assembler for ease. I have used yasm, primarily because it supports both At&T and Intel syntax and it can output a flat binary.

stub.asm can be compiled as
This should create a flat binary for stub.

If you reached here, you probably know what you can/should and can/should not use in a kernel. I am assuming that you have some experience with writing such code in other toolchains.

The kernel for this tutorial is hilariously simple. It can be C or C++.


Here is the tricky part. We must compile kernel.cc in such a way that it can run standalone, without any ties with Windows hosted environment. For Visual C++ 2010, I have seen the following switches working. I am going to save on typing as MSDN has explanation for each of those switches in detail.

Creating the boot image
This is the simplest part.

Yay! Now you have a PXE/WBM bootable image!

PXE boot setup instructions (or lack thereof)
Unless you are using Windows Server as the host OS, chances are that you will have to work with a non-Microsoft solution for DHCP and TFTP servers with pxeboot support. I have used tftpd32 in the past as well as in this tutorial, and have had no complaints thus far. Setting up DHCP and TFTP servers is out of scope of this tutorial. tftpd32 configuration is a walk in the park. Just attach the DHCP and TFTP servers on local interface, set the TFTP root and file name as the boot image. Fire up any virtual machine emulator of your choice that supports PXE boot ROM in its NIC. I have used VirtualBox. Start the VM, boot from LAN. You should see some activity in tftpd32 and moments later you should see the message.

It is fairly easy to get your kernel alongside Windows entry in the Windows bootloader. There are tutorials [1, 2] to create new BCD entries. Just add your kernel as a BOOTSECTOR application. EasyBCD is an option in case you are not comfortable with bcdedit.

Using Visual C++ IDE
Sure, these build rules can be set in VC++ projects. Enjoy one-button-hit builds with VC++'s Intellisense. Eventually when you get a debugger stub in your kernel, you could also get source level debugging working within WinDbg.
32bit versions of yasm and tftpd32 are included.

Using Makefiles
I never felt comfortable when an IDE came between me and my source code custom build rules. If you too think that makefiles are the way to go, nmake works out of the box. I personally think CMake is a better choice if your project gets serious.
32bit versions of yasm and tftpd32 are included.

Please let me know if decide to use Microsoft compilers for osdev.

Good luck!