Having had a number of similar experiences in various places in my own game, I had a closer look at this problem
It turns out that this is how the previs-precomb stuff works in this game. If a major fraction of one mesh in a group is hidden from view, the engine culls the whole group of objects instead of just that single mesh. It doesn't happen at a close distance though, only to meshes that are farther away. If you go looking for this consciously, you'll find many places in the game world where this is happening. For example in the ruined building opposite to the Red Rocket Station at Fort Hagen (the ruin where a random encounter is taking place every once in a while). Go there and look at the Red Rocket Station through one of the windows, then move around slightly. As soon as some part of the station gets hidden from viiew by the wall on the side of the window, the whole Red Rocket Station mesh disappears at once (and pops in again if you move in the opposite direction) - although you have an undisturbed view on most of the building.
Since this is all handled at engine level, there's nothing we can do to improve it.