Most I am sure will say New Vegas. I can see where you are coming from. It is a great game.
I liked Fallout 3 personally. It was my first entry into the series. I was alive and gaming when the first 3 hit the market but wasn't on my radar. Then Fallout 3 came out. I bought it on launch day and spent more hours on that then I care to admit. It caused me to play the 1, 2 and tactics. It was what I wanted, RPG/FPS hybrid. If another comes out I will buy it as well. This is the only franchise I will buy a game from no matter what.
I've played New Vegas and the story is great, but for me, nothing is going to change that first time I realized I could blow up megaton and did it. You could play the game completely different from the start of the game. You can't sequence break Fallout 4, but you sure as hell can New Vegas and Fallout 3.
I happen to be a fan of Fallout 4 as well, but it isn't my favorite.