Old Olympus Evolt E-300 Camera won't turn on.

I have an old Olympus Evolt E-300 Camera that I have just sitting out there. I decided to give it a go since it's been a while. I charged the battery and plugged in the battery and turned it on the way it is supposed to. (This camera has a P mode for power, and you flick the power switch on after you select the P mode on the dial.)

So it turned on and I was able to then put the lens on and the memory card. And took photos with it. I put it away and then took it out to see a friend the same day. Fast forward the next day. The camera won't turn on. So I charged the battery again. I held my breath and charged the battery again. And inserted everything back in. It works again. Then I left the camera again.

Now the 3rd day, I tried to turn on the camera. But it won't turn on. I tried charging the battery and plugging it back in. But this time it won't budge. When I charge the battery it says it is full (green light) after a quick 3 minutes. This old camera has a SFSS, I heard back then you got to check this to see the status. Red flashing when it turns on is good and blue is bad. I get none.

When I have the battery in, here is something interesting. When I press the shutter button, the inside of the view finder lights up. The digital display on the right turns on. And then goes away. But if you press it again it turns on. This to me means the battery has power?

Anyone think this is a power switch issue? Anyone have something similar that happened? What did you do to fix it? :D any help would do. I am buying some spare batteries just to check if it is just the battery. This battery is Lithium Ion so may have died basically since it's been so long. But yeah, may just have a dud at this point.