I’ve started working on a compiler for my programming language, see m3.git. Since code generation is boring, it targets LLVM. And anything related to “M++” is gone, and so are all object oriented features. This is simply an imperative language with modules, and maybe some type-system features.

That hasn’t been moving for some time though, due to my replacement project: an operating system. This is going to be a dirt simple but (at least theoretically) fully functional operating system for x86-64/amd64/em64t, written in Assembly. I’ve decided that it’s not going to be running C code until I finish support for loadable processes and/or kernel modules. As a general rule, I’m aiming for a microkernel structure. In part because microkernels are cool, in part because Assembly is hard to write so I really want to keep the “kernel” part as small as possible!

Now, the kernel thingy needs to do some memory management internally to keep track of some data structures where most of them are not as big as a page. I have a simple page-frame allocator, but now I need virtual memory management (which will reside inside the microkernel part of the OS) and processes, and to do that conveniently I need a kernel malloc to allocate my metadata.

There goes the third interrupting project: the malloc. In order to have something working in the kernel, written in assembly, I thought it best to start by even implementing malloc at all. So this is a malloc that you build into a .so (you can also link to it statically, if you compile it for that yourself) that loads into any program using LD_PRELOAD to replace the C library’s malloc. That is, on Linux. Haven’t tried it on anything else

I’ve stolen most of the ideas from a paper about phkmalloc, so I guess it’s not very interesting in terms of malloc design. And it’s single-threaded (protected by a single lock, if thread safety is enabled) so there will be no scalability contest between this and e.g. jemalloc! Then again, it shouldn’t be excessively hard to take this allocator and essentially have one heap per thread/core, with a shared pagepool in the bottom somewhere.

My malloc currently “works” with any programs I’ve thrown at it, except that it doesn’t actually free anything back to the OS. Properly implementing that seems to require a few changes to the underlying data structures. Oops! Which is precisely the reason I’m doing it at all – to find these problems out so I can redo it correctly in a write-once language like assembly

Anyway, after the malloc works, the next step is to translate it into the kernel code, implement virtual memory management in the kernel and to use it instead of the hardcoded page tables that it currently sets up for user processes. Then it’ll be time to implement dynamically allocating processes based on an empty or existing address space, starting them up, scheduling them, ending them, and waiting for them! That’ll be exciting

Compiler usage

I’m thinking it would be left up to the compiler to keep track of dependencies and determine which modules need recompilation. Rather than taking a list of source files to compile into objects, run the compiler for each source to make a .o file and then link your hundreds of .o files into a shared library or executable, you’d tell the compiler to either build module Main (which can have another name) into an executable, or to build a library exposing one or more given namespaces. The names declared in these namespaces would be exported as if in the global namespace (following the target C++ ABI, most likely – I would like to keep binary compatability as far as possible).

Example 1: Compile the program contained in module Main and all dependent modules into a.out.

m++ --main Main -o a.out

Example 2: Compile modules A, B, C into a shared library.

m++ --export A --export B --export C -o libstuff.so

As mentioned in the previous posts, modules referenced in the source (e.g. Net::HTTP) would be automatically looked up as ./Net/HTTP.mpp in the current include path. It is up to the compiler to apply necessary magic to that file in order to extract the information it needs to compile the current module. I’m thinking there’d probably be a local file containing a parsed representation of the module source which is automatically updated if the source file is outdated.

Due to the way this works, I think mutually recursive modules are basically impossible to write in this new language. C++, using headers with declarations separate from implementation files, allows to get around this problem in some small way by e.g. using forward declarations in the header. It may be possible for the compiler to apply a similar workaround automatically by parsing declarations and implementations separately, but I actually think it is a good thing not to be able to build mutually recursive modules.

Code generation would be driven entirely by the need to output the exported names (or just main() in the case of –main), so only names used recursively by those functions would be code-generated at all. In ordinary C++, it is very hard to control the set of functions you’re exposing to the world. In M++, it should be very easy and very explicit.

Syntax

I originally thought using the syntax of C++ would be a good thing. After all, what I secretly want this language to do is replace C++ entirely, which I thought more likely to happen using a syntax familiar to the old fashioned C/C++ tradition. Worked wonders for e.g. Java, C# and JavaScript, didn’t it? Too bad all of them punted on the opportunity to actually replace C++, rather than take some small niche where you never really needed C/C++.

However, since this is a C++ dialect anyway, why not go the extra mile and just throw out all the syntax, and all that ugly legacy that would come along with the syntax? For example, there’s this Modest Proposal for a new syntax for C++. Quite interesting read, and using a syntax like this would probably eliminate a lot of the quirky issues you run into when trying to parse C++. The proposal also suggests fixing a few quirks of the C++ language, such as making this a reference rather than a pointer. One of the most wonderful parts of SPECS (Significantly Prettier and Easier C++ Syntax) is the complete re-working of type syntax – even a complicated type signature like that of a pointer to an array of pointers to functions returning a pointer to a function is easily readable in SPECS:

type ComplicatedType : ^ [7] ^(int -> ^(int -> int));

To read the SPECS typedef, just start at the left and read:

pointer to array of 7 pointers to a function taking int and returning a pointer to a function from int to int

Contrast the equivalent C++:

typedef int (*(*(*ComplicatedType)[7])(int))(int);

There is a somewhat reliable technique for reading this kind of nested type definition (start in the middle and go outwards? something like that…), but in my opinion: don’t bother. Give the original author a proper spanking and make them rewrite it as a set of typedefs instead

SPECS would be so much nicer than C++ syntax, but then I’m much more talking about an entirely new language than a C++ dialect (even though some semantics would be similar).

So: keep C++ and the world of pain it represents or write something using a new (and therefore scary and controversial) syntax?

I’m not entirely sure about this, but maybe I’ll call it M++ as in “C++ with Modules”. In any case, this will become a C++ dialect as there is no hope of not breaking any code. There will also probably never be a very easy transition path for existing large-ish C++ codebases. (That’s not even a goal until this thing is sufficiently awesome to motivate someone to convert a large body of C++ code to it…)

What is M++?

Basically, it is C++ with “modules”, where modules are somewhat like ordinary C++ translation units, but with special magic for how names are imported between modules.

Importing a module imports a well-defined set of names into the local scope. Contrast with C/C++ where module importing is implemented by inclusion of header files. (You know this already, but I’ll repeat it for the sake of rhetoric) Header files may #define just about anything and wreak whatever havoc on the environment of following files. This means that any kind of higher-level reasoning on the effects of changes in header files on the actual code is pretty much moot. Any compiler must recompile every included header file from every including source file every time any part of this pool of mud changes. A wonky define in module A may produce error messages in system headers included from module B, while compiling module C. Modules A and B may not even be your code, or code you can’t change. World of woe!

With a proper module system, declarations from different modules will never clash, except maybe in the module that is importing conflicting names from more than one module. But that would be caused by code in your module, and you would have the tools to solve the problem!

Guiding principles

• Keep as much as possible of the syntax and semantics of C++
• Remove the need for preprocessor inclusion
• Keep the preprocessor around for e.g. importing external interfaces
• Replace the current text-level importing with a symbol-table-level import
• Provide good means of separating unrelated components by:
  • Limiting the set of exported symbols from components
  • Providing easy means to cherry-pick subcomponents (using namespace::name;)
  • Providing robust non-conflicting importing of whole components (using namespace;)

Basic proof-of-something example

For context, this module would reside in a file Main.mpp (maybe even just call these files cpp?), and exports the class Main::Foo, the method Main::Foo::Bar and the function Main::main(…).

// This is where the *really* cool part is. These modules (namespaces) are
// automatically imported by searching the system module path. Either they
// are defined by local source tree cpp (mpp?) files named e.g. Win32.cpp
// and Net/HTTP.cpp (Net_HTTP.cpp would also work), or they could be
// binary self-describing modules. Nothing said here on how that binary
// self-description would look. That's the magic left as an exercise for the reader.
using namespace Win32;
using namespace Net::HTTP;
 
// These are some provisional ideas on how to wrap the C/C++ standard library in
// M++ form. More on importing legacy C/C++ functions and classes later on.
using stdlib::atoi;
using stdio::printf;
 
 
// This is a private function. It would not be linkable from other translation
// units (default linkage in the top-level is static), but is usable from all
// definitions in the Main namespace below.
void Foo_Bar()
{}
 
 
namespace Main
{
    // Normal classes here
    class Foo
    {
    public:
        void Bar();
    }
 
    // Maybe the main function could move into the Main namespace from the
    // global namespace like this:
    int main(int argc, char *argv[])
    {
        Foo foo;
        foo.Bar();
        Foo_Bar();
        printf("Baz\n");
    }
}

Unresolved/other issues

• How do you expose things in the global namespace from M++ modules (these exposed names should follow the relevant C++ ABI and mix-and-match with C++ code.)

This one is slightly harder than the latter question. Many solutions here are bad. So I think I’ll just let this one noodle for a while

• How do you import external C or C++ classes/functions into M++?

I’m thinking you’d #include the external headers in one module, then use using ::asdf; to import the top-level declarations into the namespace exported from that module. This means that a M++ compiler must be able to understand the full wonderful ambiguity of C++. But hopefully, the modularization through use of namespaces would mean that only one of a large number of modules need to go through the hassle of actually parsing all that crud in order to build a small symbol table.

One remaining issue is how to distribute the kind of macros that are required/useful (i.e. file/line-tracking allocation functions, asserts, debug/release-dependent code). Would you be including some small number of headers into each module to do that, would the module system somehow get involved in preprocessing and let macros be imported as part of a namespace?

Getting rid of macros in the first place is a pretty good idea anyway, but exactly how far can you practically take it? Some things like stdint.h define a large quantity of useful macros. In an ideal world, these would be constant varibables const int INT_MAX; etc defined in a suitable namespace (perhaps something like stdint, since they come from stdint.h).