This is really where unit tests will save your hide. I never did any truly dramatic rewrite, but there was one project (fairly new, even) that had one very badly written, very badly understood module.
I started out writing unit tests for it. There weren't any, so I wrote them until I had unit tests that tested everything. Those unit tests also document the behaviour of the module. So then I started asking people what it was really supposed to do. Then I refactored it into something more maintainable. Only after that did I change the unit tests to reflect the behaviour the module was supposed to have, and based on that I finally wrote the code we needed.
The unit tests were my anchor in the storm. They keep me grounded. They mean I can do anything to the code, while keeping tabs on the impact my changes have.
The trouble with unit tests is their small scale. You write 3 Mb of source code filled with unit tests, does that give you any guarantee the program will even start when you call it ? (A point of frustration that I have had with other developers is that they deliver "a product", which immediately sigsevs upon execution, and then claim that's not possible, all tests pass. Same with memory leaks, which are never caught by unit tests, not in C++ and not in Java either).
First thing to do is to get the program having a backend and a frontend separately, then introduce a system test. Something that does what the main customer(s) do with your application, and does it while seriously stressing the system. You need it to send out mails ? Make it send out 10000000 mails while only using the backend calls the webserver calls while activating a sleep(1) in the mock database routines, and impose a memory limit of half a gig, checking if memory usage actually drops when you stop pushing the application. Make it use an actual database, and make it send actual emails to a fake "smarthost" email server. If it doesn't complete in 5 minutes, something is wrong. I've been meaning to implement this test atop docker. I tried to generate VMs to test the full application, start the app in a simulated network and run most of the actual production code, but making the VMs took ~25 minutes (and they weren't even on the correct machine by then), which made it unusable. I wonder if docker can do better.
Yes this test will be flaky, yes if it does indicate failure figuring out what's wrong is bloody hard, yes it means abandoning several holy cows of software development, yes it means some program will tell you where to focus attention, yes it will force you to actually think about debugging production failures BEFORE shipping, it forces you to have test systems for the program's components, it generally makes you deal with reality. You will have discussions with this test if you do it right ("No that hash-table MUST be faster than that linked list ! Aaaargh !"). And that is something sorely needed in software development.
Second thing I use this for. Boss-man wants a new feature. Ok, sure. I add a system test that uses it. Needless to say, it will fail. But it allows me to have a good think about what the high-level changes needed are, without unit tests constantly badgering me that some method now accepts different parameters. It will tell me if I got it right. And when I deliver it, the freaking program will start, and won't crash the first time someone uses this feature. You tell me how to make unit tests that deliver 10% of those guarantees. Forgot the big picture after spending 2-3 hours on getting a module fixed after adding new functionality to it ? Just single step through the system test and see where it blows up, and hey. There's the next thing needed for the high-level change.
Unit test programmers are terminally afraid of changing module interfaces to suit higher-level objectives. Given the amount of unit tests needed to even approach 95% coverage, I fully understand the resistance. But if they choose to write unit tests and I need the module change, I just go in and do it, disabling the unit tests. I tend to get some attitude in code reviews.
It also means that replacing the database with something else is something you can just do in an afternoon. Changing the core logic of the system, from a set of rules to a rule interpreter ? Changing that rule interpreter to jython ? No problem, the test will tell you if common use cases of your product are actually still working. And that's what the business/boss (should) care about.
Ops will love you if you do this. You will consistently deliver a working product that operates well within known parameters, as opposed to a system where the tiny wheels all turn, just not in the same direction.
Unit tests optimize for the wrong thing : they make sure programmers' jobs become easy, because the modules behave the way they "should", and can be changed with "predictable" results, which of course turn out not to be predictable in the real world. Furthermore the interactions between the "should"s of multiple modules ... turn out to be less than simple. Unit tests allow you to say "not my fault" (e.g. if your method is merely slow, but suddenly gets called billions of times because some other system changed and makes the system suddenly crash when too much is going on, but not due to a logic error). They make it easy to change systems by only touching the smallest components, which of course also massively limits the changes possible in a system.
Unit tests impose strong limitations on the changes that you can make to code (in reasonable time). Therefore, for the large majority of code, they are a burden, not a boon. There is one (big) exception. Algorithm and data structure code should be thoroughly unit tested. In most projects however, you have maybe 2-3 methods that fall into that category. If you have more, find an open source project to replace 50% of them, or figure out how to use more general data structures to do your thing.
I started out writing unit tests for it. There weren't any, so I wrote them until I had unit tests that tested everything. Those unit tests also document the behaviour of the module. So then I started asking people what it was really supposed to do. Then I refactored it into something more maintainable. Only after that did I change the unit tests to reflect the behaviour the module was supposed to have, and based on that I finally wrote the code we needed.
The unit tests were my anchor in the storm. They keep me grounded. They mean I can do anything to the code, while keeping tabs on the impact my changes have.