stz incremental compilation
I'd like the stz compiler (stzc) to be very very fast. So fast you don't have to think about waiting to try out your changes. So fast it might as well run in the background monitoring file changes and recreating the program as it goes.
For small programs that is not a pipe dream, but for large programs there are some bottlenecks that get very complicated. The biggest one being link time optimisations. But before we get there we can talk about the things we can cache during compilation that will speed things up.
Dependencies between files are known at compile time as each package needs to either have a using: statement or an import: statement to use other packages. I am tempted to require #package name: 'Foo' at the start of each file so that the filename convention can be tossed out the window.
This isolation of packages means a package can be compiled in its complete form and only need be recompiled if its files or a package it depends on have been changed. Incremental compilation means the cost of compiling with an interpreter as part of the process is almost entirely gone (for some measurement of gone).
No fancy build tools should be necessary to achieve this. The compiler should be smart enough to keep track of all this itself. Each packaged up intermediate package file should store its dependencies too.
The next step - assembling - is where things get more expensive. We should now have an easy to load memory blob of classes, methods, imports, exports, and parse trees. This information is handed to the backend once it's confirmed nothing needs to be rebuilt. The act of inlining might happen before giving this information to the backend - or we might expect the backend to do inlining for us (qbe does not yet, but there is discussion of it doing so). We might even do discarding of unused bits and pieces before we pass to the backend.
At this point the backend must compile the information in to executable instructions - either opcodes in the case of the interpreter or machine code in the case of a backend that produces an executable.
Adding dependencies to external files could be achieved by adding a dependency: call to the #package object. If you need to (at compile time) process some data from a binary file, for instance, then the tools should aid in that too.
It's a coin toss between running interpreted code (without a JIT) or assembling the instructions in to machine code; which will be the slowest. With the packaged package caching at least when we decide to compute 10,0000 primes it'll only have to happen once for the majority of developers using stz.
