It's quite possible this is old news to people, but it took me more than a couple of minutes to find the solution. I was using the FromFile method on the Image class to create a new Image object. When I was done with the Image, I needed to move the file. But it turns out that the file was still 'being used by another process'.
The solution is simple, but it didn't come immediately to mind because I'm so used to basically ignoring the scope of variables. The trick is to call Dispose immediately rather than waiting for garbage collection to do its thing. Once Dispose has been invoked, you can manipulate the underlying image file as you need to.