As enthusiasts, we often attribute far more value to a car than it's actually worth. Body work is expensive, that's just a fact. Unless you can do it yourself, chances are very good that there's no way that it will ever be worth doing.
Years ago, I started with cars that weren't worth a lot of money, and I spent far more than they were worth on materials and tools to do my own body work. I knew that the car "wasn't worth saving" but I went ahead and did it anyway, so that now when I have a car that is worth saving, I can do it.
Well, except that with a rational analysis, the newer car isn't really worth it, either. It's either worth it to have a body shop do it (Typically because the damage is insured) or it's not really worth doing because another one can be found in better condition.
Really though, who cares about a rational analysis. My car is my hobby, so I will spend the time to do things to it.
So I would say that doing the body work is probably not worthwhile. I'm sure that there are more cars out there like mine, which look okay, and just need "simple" work (Like the fender that I replaced) and are cheaply available (I paid $200) which will give a much better return on investment.
If you want to learn to do body work, then this is a good chance, but be warned, the car will not increase in value if you do it, and may actually decrease, depending on how well you do. Same with the motor swap. It probably will not increase the value to another person, and may even lower it, because someone can no longer just go to a shop and have them find parts by VIN number.
If this car is your daily driver, and you do not have funds available for another car, you do not have funds to "fix this one up." Unless you can afford to be entirely without a car, your priority must be maintenance and upkeep of the car you have, as it is. Body work (Beyond basic safety) is a luxury, and one that will eat money like nobody's business.
The whole reason that I have an Accord is because I need it in order to be able to do fun things to my Supra. I won't be able to do fun things to the Accord unless I have something else that I can drive.
As a younger person, I did "fix up" my daily driver. It's a slow experience, since I could never do anything that had the car out of commission for more than the weekend, often not more than one day. I ended up doing the same job 3 times, because I had to put it back together before I got to everything that needed to be done, and needed it to get to work Monday. So I fixed up the body panels. Drove the car in primer for a while. Then sanded off the primer because it was too dirty, and got a bit of surface rust, and primered it again, and painted with spray bombs. Then sanded that down, re-did some of the bodywork that I'd rushed through, and spray bombed it again. Then one more time when I could get good paint. In total, I probably could have bought new fenders and hood, and had the whole car painted at a shop for the same total money. Either way, it probably would have cost the same to just look for another car in better condition.
So if you're asking if you should just let go, the answer is probably yes. I know that you have an emotional attachment to the car, that's the nature of an enthusiast. That's worth something. Keep the car, keep it in decent mechanical condition so that you can continue to drive it for as long as you can. If in the future finances permit you to have a "beater" that you can drive while you fix this one up, then go for it. Until then, either wait, or go looking for another car that you'll love just as much.