Harry Jeffery

Book review: Modern x86 Assembly Language Programming

2022-01-27

Over Christmas, I got a copy of Daniel Kusswurm’s Modern X86 Assembly Language Programming, 2nd edition. It promises to cover the fundamentals of x86 64-bit assembly, with coverage of the modern extensions such as AVX-512.

I was quite excited to read it, as I’ve been enjoying reverse engineering quite a bit lately, so also wanted to try doing some software development directly in assembly to sharpen my skills a bit. Unfortunately, I’d have to describe my experience of reading it as disappointing. Within 10 minutes of reading I had found several errors, and sadly the trend continued through the rest of the book.

An example of one of the errors I found was in a table on page 16, listing common instructions: cupid - Query CPU identification and feature information, which should read cpuid.

I also didn’t gel with the teaching style of this book. I was hoping for and expecting a thorough guide of the fundamentals of x86 64-bit assembly language. Perhaps starting with a high level look at how functions from C or C++ are represented in assembly, what a stack frame is, what the most common calling conventions are, before drilling down into topics like floating point operations, common string manipulations, best practices, and so on.

Instead, this book shares about a page of C++ code, followed by a hand-written equivalent in assembly (with comments) and then some prose describing what the program does. Any new instructions introduced are given barely a sentence of explanation. There’s no consistent synopsis for new instructions being introduced, just what feels like hand waving.

Perhaps for some that’s a useful format, but to me it seemed to add very little value to just reading the output of gcc -s by itself, and using google to look up any instructions I was unfamiliar with.

The comments in the assembly code also doesn’t seem to have been proof read very thoroughly, containing numerous inconsistencies.

Just by flicking through the book now and inspecting random pages I can find examples easily.

On page 56: inc edx; edx = i + i. Actually inc increments a register, it does not add i to itself.

On page 76:

mov rax,r8          ; rax = 'src'
mov rdi,[rax+r11*8] ; rdi = src[i]

Why is src in quotes? rax isn’t pointing to a string containing src. Elsewhere src isn’t quoted.

Other errors I remember, but can’t be bothered to back and find were the comments using + and += interchangeably. That is, sometimes you’d see count += 1 to mean increment count, and sometimes you’d see count + 1 for the same operation. Sometimes the comments would even refer to different variable names than the code was even using.

Perhaps this sounds fussy, but I didn’t go into this book with a nitpicking attitude at all. In my casual read I just started spotting errors, and couldn’t stop spotting them. I can’t emphasise enough that this is a second edition, with experienced and respectable engineers listed as the author and technical reviewer of this book. I don’t really understand how so many errors made it in, survived review, and survived to the second edition.

Moving on from the topic of errors, I’d like to comment a little more on the way that key concepts are covered. I’m by no means an expert on assembly programming but I’d still consider calling conventions to be an absolutely critical area to cover, lest a beginner be confused by contradictory examples found across different books and the internet.

This book casually introduces them as an aside on page 31, buried under the header of Advanced Integer Arithmetic. It’s mentioned because the author is giving an example that accepts a non-trivial number of parameters and has realised the way they’re passed to the function might be non-obvious to a beginner. So on page 33 the author gives a brief summary of which parameters are passed in which registers in Visual C++ and that for other compilers and operating systems the reader should go and read their respective manuals. What?. At the very minimum the author ought to provide the most common calling convention around: the SysV x86-64 ABI, but they don’t. In 600 pages not one paragraph is spared to summarise any x86 calling conventions other than Visual C++’s.

Consider this, if someone is writing assembly language by hand what context are they going to be using it in? Most likely to write server software that is almost certain to be running on a unix-like operating system. Have you heard of any high frequency trading shops using Windows Server? I haven’t. Or perhaps they’re working on a AAA video game that’ll indeed be running on Windows 10 x64, but also very likely on a console like the PlayStation that runs on a FreeBSD derivative.

But no, this book leaves the most common environments high and dry on such a critical aspect of assembly programming. “That’s not fair,” you might say, “the author is assuming a more experienced audience.” No, they’re not. On page 21 the author felt it necessary to explain to the reader what a Visual Studio solution and project are, and how to open one by double clicking on it. I cannot reconcile the decision to explain how to use Visual Studio in an assembly book but not to cover a non Windows ABI.

Another concept that I thought was introduced and explained far too late in the text is stack frames. Stack frames are taught on page 143. Page one hundred and forty-three, under the subheading of Calling Convention, which yet again only describes Visual C++. Another thing you won’t find an explanation of in this book: __stdcall vs __cdecl vs __fastcall. Despite being extremely common calling conventions on Windows that you will encounter and have to deal with, they’re not covered. It’s all implicitly __fastcall here. I appreciate that this book is focused on 64-bit programming but giving the reader some awareness of that potential difference could save them hours of confusion in the future.

I wish I could have written something nicer about this book, but I really did feel let down by it. It doesn’t cover core concepts and best practices in a clear, methodical way. Instead it dumps sample programs on the reader with dubious code comments and hand-waving prose. I just can’t see myself recommending it to anyone. It’s not clear or thorough enough to recommend to beginners. Meanwhile I suspect experts would be better off improving their knowledge by just reading the manuals published by Intel and AMD.

If someone knows of a better book covering this topic I’d love to hear recommendations.