The first thing is to decide on WHERE you are going, then its the WHEN. Which is often the hardest decision as there are so many places to go.
The WHERE and WHEN is closely linked to the ..... COST part. Unfortunately when you have kids WHEN can be dictated by school holidays, exams etc. Guess what, those school holiday flights are....not the best deal around.
Going off season or shoulder can give significant savings, its quieter but you may be exposed to things like undesirable weather conditions.