I was about to say, don't bother submitting a patch to fix this, as last week rustc completely moved away from make, deleting all its makefiles in the process: https://github.com/rust-lang/rust/pull/39431
I'm curious how OP even ended up trawling old versions of rustc's makefiles from 2014. I bet there's an interesting story there. :P
I don't think using noops to get the parser to co-operate is that strange at all, especially not something like a makefile where whitespace is significant. It's the whole point of the `pass` noop in python: the Python grammar expects there to be something inside a loop, so if you want an empty one, you have to throw a noop in there.
> the Python grammar expects there to be something inside a loop
I think this is the underlying issue - the "well, it's a list, so there's got to be something inside the list too" mindset, which makes sense in the real world but is stupid in programming. (Some bad memories at old Visual Basic times come to mind too)
Maybe there was something else there before, and it stopped working when they removed it, so they tried adding back the most minimal thing they could think of as a sanity check.
"Why doesn't this work?". Add debug statement. "Huh, now this works". Take out debug statement. "Huh, now it doesn't work". Try no-op. Table flip. Commit.
Maybe they even knew that white space would work. However some people (including me) wouldn't even look twice before they remove white space. An explicit no-op on the other hand would get my attention.
So they switched from make to something called rustbuild, it appears. A quick search didn't turn up any details on rustbuild for me. Is it something very rust specific, or could I use it for other projects?
It's specific to rustc, to satisfy the unique demands of bootstrapping compilers. Unlike the makefiles it leverages Cargo as much as it can, which means that the compiler will gradually become less insular and will also benefit from future Cargo development.
Right now the non-Cargo bits are written in Python (which was already required to build LLVM, so this doesn't add any new dependencies), but those bits could pretty much be written in anything, if needed.
rustbuild is actually pure-Rust. The entire Rust build is written in Rust (Rust is bootstrapped all the way down!). The python scripts are a slim wrapper that downloads the bootstrap compiler to get into the Rust world.
Anyone know how Debian is handling this? Debian packages are not allowed to access the network on build, and (normally) are not allowed to depend on non-Debian binaries.
rustbuild can optionally build from vendored (in-tree) source of all cargo dependencies, where it does not touch the network, and that's how we expect packagers to build Rust. I believe Debian has their own bootstrap lineage where they bootstrap from Debian's own prior builds, and rustbuild supports this mode; but they also have a fallback for rebootstrapping, with the corresponding policy exceptions.
rustboot, the OCaml Rust implementation, was removed in 6997adf7 on 13 May 2011. It looks like it was fairly actively maintained until then.
A file named src/snapshots.txt showed up in the repository in cb53065a on 2 May 2011, and remained there until 02538d46 on 13 April 2016. During that time, rustc was buildable from specific snapshots of previous versions of rustc, which were listed by both source commit and hashes of binary compilers.
From 13 April 2016 onwards, stable rustc compiles on previous stable Rust, and specific versions are listed in src/stage0.txt in a similar fashion.
So, you can probably take the last version of rustboot (it didn't change between 2 and 13 May 2011), compile that with some version of OCaml, then go from 2011 to 2016 by repeatedly building the newest snapshot possible with your current snapshot, then build stable Rust 1.10 onwards, thereby reducing it to the previously solved problem of verifying that Ken Thompson has not compromised your OCaml compiler.
They have to be done in order. Only the first version that was in Rust could be compiled from the OCaml compiler. Then each one in series only builds with the specific parent compiler.
> (normally) are not allowed to depend on non-Debian binaries.
It turns out that Debian is more permissive than Fedora when it comes to bootstrapping compilers with out-of-archive compilers. Fedora has an actual policy and it allows one-time out-of-archive bootstrapping.
1. A python script to make sure you have the correct environment, and that kicks off cargo to compile the program used in step 2. ("rustbuild" proper.)
2. A Rust program that manages the full build (remember, Rust is bootstrapped, so there's three stages...)
3. That build kicks off a number of "cargo build"s for all the components and glues the final result together.
For any project written in Rust, pure Cargo should be enough.
I've always been a bit confused about Rust's need for so many build stages. The project I work on (crystal) only requires you to install a working build of the latest compiler release to build. (the makefile is pretty simple). As far as I know, rust can't bootstrap without downloading a rust release, so can you explain why rust has compiler stages.
1. This stage is a compiler built with the compiler from stage 0
2. This stage is a compiler built from stage 1.
If the process worked, then stage 1 and stage 2's output should be identical. You can skip stage 2 if you want to skip that verification, and that sounds like what Crystal does.
I always mess this up, but here's something from memory: I believe there's ABI things that motivate stage 2. The compiler built in stage 0 has historically had special ifdefs to deal with "features that don't exist in the old snapshot" (we aggressively adopt our own features in the compiler and stdlib, because that's a great way to test them). This implies it can potentially emit binaries with a different ABI from the compilers built in stage 1 and 2. So the compiler that's output from stage 1 can actually have a different ABI from the binaries it produces.
Normally this wouldn't matter, but Rust also supports compiler plugins written in Rust. These link directly to the compiler that compiles them. With the stage 1 compiler's quirky ABI, the plugins it compiles might not be able to interact with it.
The stage 2 compiler, on the other hand, uses the same set of ifdefs that the stage 1 compiler has. So it was built by a compiler with the same output-ABI as itself, and so plugins have the same ABI.
So:
* snapshot: Built with snapshot ABI, produces snapshot ABI (snapshots are old stage2 compilers)
* stage0-output: built with snapshot ABI, emits stage0 ABI.
* stage1-output: built with stage0 ABI, emits not(stage0) ABI.
* stage2-output: built with not(stage0) ABI, emits not(stage0) ABI.
Ah yes, crystal doesn't have compiler plugins, which simplifies things a bit. Say, if you specified that you had to be able to build rust master with the latest nightly release, could you use only 2 stages (latest nightly, built master)?
I'm currently working on crystal's CI infrastructure so that we can have nightly crystal builds (+ CI) for all architectures we support. Currently we release features incrementally such that the compiler can always be built with the latest release, and I'm wondering how much effect relaxing that constraint to the latest nighty would have on development speed.
Additionally, if one is building a compiler for other targets (both a compiler that runs on another target, and just a simple cross-compiler), the makefiles/rustbuild need to make sure the appropriate things get cross-compiled in the appropriate stages. Managing this was a source of a lot of complexity in the old makefiles.
Is it possible to bootstrap from a compiler compiled from C, or another language? Or is there another way to ensure there aren't any sneaky compiler backdoors?
In university our classes required us to make a printout of our code to submit. My friends and I had a running joke that we'd hand in a stack of blank pages and claim we solved the problem in Whitespace :)
Ah good, now the whole world gets to enjoy the greatest Makefile ever. For other curiosities of Rust's history and bootstrapping process, I recommend brson's annotated stdlib:
Thanks, that sucked me in. Starting after line 93 it gets downright fascinating. Have written commercial compilers but not self-hosted so it was illuminating
Reminds me of the R4300 multiply bug, and the lazy compiler fix that just always emitted a nop between them if it saw two mults in a row. It was a real, hardware bug tho.
Reminds me of a lab in college where we were doing MIPS assembler. This particular lab involved output to the console using trap handlers (basically a glorified hello world). I was the only one in the lab to get it to work at the time, and I could explain how. I had a random assignment of a strange hex number to, I think, a register (it may have been a memory location) preceded by a comment to the effect of: "Do not touch this. I'm drunk. I don't know why this works or how, but it does. Sober you will not understand." And, true enough, I didn't. And this was before google was a thing, so I'm still not sure how/where I found that magic number to make things work.
Very often hardware bugs can be worked around by well-placed NOP.
Last time I did that was on MSP430, where it seems that some integrated peripherals set completion interrupt flag one cycle before the operation really completes. (I'm not sure whether it really is bug, I didn't find this documented anywhere, but for TI's example this does not matter and produces marginally better performance)
Which MSPs, if I may? I use some of their 2xxx hobbyist line (since I'm a hobbyist), and this sounds like the kind of PITA that I'm not nearly enough of an embedded dev to avoid tearing my hair out over.
The MSP430x5xx and MSP430x6xx families have this problem. From the user-guide [1]:
> Include at least one instruction between the clear of an interrupt enable or interrupt flag and the EINT instruction. For example: Insert a NOP instruction in
front of the EINT instruction.
> Include at least one instruction between DINT and the start of an code
sequence that requires protection from interrupts. For example: Insert a NOP
instruction after the DINT
Luckily if you use the MSP compiler's intrinsic functions to enable/disable interrupts it takes care of this for you.
What I described happened on G2553 with USCI_A in SPI mode. Given the fact that various code samples on the net do not handle this behavior it seems that it is either happening only for some configurations or only on some versions of the HW. But: when you see even bytes on SPI dropped or replaced with previous ones, add NOP.
SPARC and other classic RISC pipelined architectures require you to put a nop "Bubble" in the pipeline after a branch if you can't use the instruction slot that's always executed right after the branch.
The democoder in me thinks of "that's not a bug, it's a feature" and wonders if, like the branch delay slot, the delayed memory effect was used to squeeze in an extra instruction and save a cycle or two in optimised code.
A similar trick can be found in this interesting article:
Looks like they're using make as way to much of a programming language instead of the IMO much better declarative approach, way too many loops and function calls rather than dependencies. I'm not sure but it also looks like it's building too many things instead of different build for different configurations.
I dunno, I've never seen a readable Makefile in any system above say 300 files. If you could write readable declarative Makefiles getting the job done, would there be autotools? This is not to say that autotools get the job done beautifully, just that their existence demonstrates how the problem gets us all contort violently to terrible effect like a wounded animal screaming and running into trees.
I admit that I despise make and maybe I don't get the beauty (I get the beauty of the idea but I've never seen it result in actual beautiful Makefiles), and maybe if you linked to what you consider a good open source Makefile, we could discuss this in more concrete terms.
> I dunno, I've never seen a readable Makefile in any system above say 300 files.
The build complexity should be a reflection of the project complexity, not the number of source files. Having to run in multiple environments and separating things into libraries would add complexity for instance, adding a new source file shouldn't. When you start adding calls and loops however, that complexity grows exponentially instead of linearly. This isn't make specific.
> I admit that I despise make and maybe I don't get the beauty (I get the beauty of the idea but I've never seen it result in actual beautiful Makefiles), and maybe if you linked to what you consider a good open source Makefile, we could discuss this in more concrete terms.
Here is one I put together (http://flukus.github.io/2016/11/30/2016_11_30_Rediscovering-...), I tried to make it somewhat real world but there are obviously many caveats, it's dealing with c# (easier to build), dependencies come via nuget, but I think it shows the idea.
The most common anti-pattern I see (again, not just in make) is thinking that there has to be one build that does everything, it will iterate over target OS's and/or environments and produce a build for each one for instance. It's much cleaner to have the OS/environment be one or more parameters and to execute the build script multiple times with the desired parameters.
Developers have a tendency to not realize build scripts are a seperate discipline worth learning.
You use $(shell) and sed in there which I don't think is much different from $(foreach), and perhaps it's gnarlier. Perhaps my counterpoint is "a build system is a program using a library/framework for scheduling work based on dependencies" and make's insistence on builds being about the dependency graph and delegating programming to shell commands and afterthoughts like $foreach, plus its very painful evaluation model with unclear order make for a terrible programming environment.
A big difference is that using sed is producing a single known output that is clearly evident, it's much less clear what all the loops and calls is producing, it may even be an example of premature abstraction.
I don't think relying on shell commands is a bad thing though, it's just part of the unix philosophy, every build tool in existence will call external programs like the compiler itself. I doubt you could find another build tool that can replace values as simply as sed. The shell command would actually be unnecessary on newer versions of make.
Well, I guess I agree with Rob Pike in that the "tools doing one job and doing it well philosophy is dead and the eulogy was delivered by Perl." I think you should write build rules in a programming language and a programming language is something that doesn't make you upgrade to a new version to avoid shelling out to find. Let's agree to disagree :-)
Yes, the Makefiles were an abomination. Nobody could understand them. The reason the makefiles are so heavily macroized like they are is because of cross-compile bootstrapping, where the build rules are the same in many configurations, with slight differences. Effectively, large swaths of the makefiles were parameterized over 4 different values.
Sincere question - why not use autotools? It's a lot easier writing a Makefile.am declaring the parts of the project and having platform specific stuff handled by autoconf.
I find autotools very complex and bloated, but it tends to make builds for different platforms, cross compilation, dependency checking and other tasks easy for me. It's often easier than dealing with (g)make directly IMHO.
The answer here is probably "bootstrapped compilers are hard". Rust compiles itself thrice in a typical compile (most devs do fewer stages when hacking), and the makefile needs to distinguish between these, and it gets more complex while cross compiling where your host/target change halfway through the process. So it's quite possible that it's easier to make work in pure make.
Reminds me of a bug we encountered building mobile devices.
Turns out if a certain instruction was executed on a page boundary the CPU would lock up (the gist of the issue anyway, might have had to do with being in an elevated mode as well). Appeared to be a problem in silicon. Unfortunately I can't remember enough details but a NOP or strategically placed variable solved it.
Reminds me of the time my team found a bug in the .Net C# compiler. A whole branch of an if/else statement was being ignored, as in, it simply wasn't present in the emitted IL.
Adding a little `&& 1 == 1` to the conditional made the compiler do the right thing again.
Edit: preemptively, no there wasn't anything special or weird about this conditional.
A better comment would describe a test case that would allow to quickly confirm that the command can be removed when the compiler bug is fixed. Because otherwise it can be there for ages, because everyone will be afraid of touching the code.
There's a story I've heard that ends with the real reason being that the author thought it would be funny to have people asking him why there were three consecutive nops years after the fact.
This is a makefile, so no. What you could do is replace the $(eval $(call PREPARE_LIB,libname-glob)) lines wherever they are with $(info $(call PREPARE_LIB,libname-glob)) and see what actually expands. But I suspect they've either tried that, or the usage of this macro in conjunction with other macros is so complex that it's not super-easy to parse the output and find the problem.
Make is so useful that I'm willing to live without things like a debugger and useful error messages in order to have the flexibility to implement a build system.
Per Moore's law, the number of nops must double every 18 months. For codebases originally written in 1977, the year of make's origin, this now measures in the dozens of meganops.
I'm actually working on a browser extension that will use natural language recognition to detect when you're typing a sarcastic comment and automatically take a photo with your webcam and upload it to imgur. :)
A common result of moving instructions through early processors at too high a rate to maintain laminar flow. Early cryptographic algorithms in particular were severely limited by such considerations, until the advent of modern pipelining algorithms vastly reduced susceptibility to data cavitation.
As a point of historical interest, the "dataflow" paradigm in programming originated as an alternative approach to solving this problem, which attempted to avoid it entirely by having everything happen at once all over the place through magic. Between the misnomer and the heavy drain on the limited market for wizard labor, it's easy to see why the paradigm, though intriguing and not without useful lessons, failed to gain meaningful traction in the industry - perhaps ironically, there was just too much friction for it ever really to take off.
https://github.com/rust-lang/rust/commit/fca66702a44038b14d8...