Its not the goblins showing up that is the plot hole, but the goblin slayer guy himself as he comes into the show in the same direction that the Priestess and the injured girl came from. Which is the entrance of the cave. As in the anime we only get shown the 1 cave entrance and the only other route/path in the cave is the side one where the totem is placed slightly ahead of, to distract the adventurers so the goblins can attack from behind out of the side path.
But for some unexplained reason the Goblin Slayer comes in from that same direction of the cave entrance, therefore he must have walked passed the dead guy, the side path and the ambush point for the goblins to reach the Priestess and the injured girl. Which makes no sense for him and the Goblins to not notice each other or attack each other.
I don’t mind some plot holes, but to me it was rather big one really early into the show considering its episode 1.