Notes on Refactoring Exponential Macros in Common Lisp

Abstract

I recently consulted for a very big Common Lisp project having more than one million lines of code (including comments). Let's call it "System X" in the following. System X suffered from extremely long compilation times; i.e., a full recompile took about 33:17 minutes on a 3.1 GHz MacBook Pro Intel Core i7 with SSD and 16 GBs of RAM, using ACL 10.1. It turns out that a number of macros were causing an exponential code blowup. With these macros refactored, the system then recompiled in 5:30 minutes - a speedup by a factor of ~ 6. In this experience report, I will first illuminate the problem, and then demonstrate two potential solutions in terms of macro refactoring techniques. These techniques can be applied in related scenarios.