It sounds like this cat is not feral, but not completely socialized, either. I rescued a stray like that many years ago. He knew what the sound of a can opener meant, and he seemed to want to get petted, but he also couldn't stand human touch.
If he's coming out to look at you, he is slowly getting used to you. A lot of patience will be needed.
One thing that helped greatly was interactive toys. Anything where you hold a string or a wand and make the toy move. Have one with you when you are reading in the sitting room. When he comes to the doorway, don't look at him, but make the toy move0--bounce it, wiggle it on the floor. Eventually, he will pounce.
Don't make too much eye contact--that can be seen as a threat. Remember, to him, you are the Human Predator. He needs to learn to trust you. And that could take some time.
When I said patience--it was 10 months before he got into my lap. He stayed there for about 20 seconds that first time. But eventually, it got to the point where he could tell that I was going to settle down and read a book, and he'd hop in and out of my reading chair until I sat down, and then he'd curl up in my lap and prevent the other two cats from getting anywhere near us. Reading time was his lap time. But I think that took another year or so to happen.
There's an organization, Tiny Kittens, that works with a lot of feral cats and has even socialized some of them. If you contact them, they might be able to give you some pointers. http://www.tinykittens.com/
And Maravel and Neelix are two of the feral cats Tiny Kittens rescued. Neelix is having a much slower time adjusting to humans than Maravel. You can find out about them here: http://www.tinykittens.com/cases/maralix and follow their continuing adjustment to indoor life on Facebook, where it is clear that Maravel is fast becoming an indoor cat and Neelix is much more hesitant about trusting humans, but is slowly getting there: https://www.facebook.com/Maralix-Tak...9357103741252/ (Yeah, I look at a lot of cat cams on the internet. I can stop any time. Honest.)