How can I make it so that the directory display doesn’t show the sub-categories?

You’ll need to edit the Summary Display for Categories: Directories –> [Your Directory] –> Content Types –> Category –> Summary. You can delete the Child Terms Display Element from here.