First, you have to have the quest to be able to go in.
Once you're in, you need to read the book near the door that is closed, and it ultimately, your final goal. You need to do what it says. Go to each boss in the dungeon. They must be defeated first, before the door to the final boss opens up. Solve the puzzles.
