Also make sure you're using drill bits made for plastic/metal, and not wood.
Metal bits are THE worst things to use actually. They cut by "slicing" the metal. Doing this to rigid plastic can only lead to the bit grabbing the plastic and breaking it. Especially in holes larger then 3/8". The best bits to use are Forstner Bits and Spade Bits which are made for wood. Forstner Bits are the best as they don't "bite" the plastic in chunks but rather chip it away and grind it. Spades are pretty good but you have to go in perfectly straight of you will go end up catching the backside. Always use a wood backer such as a spare block of 2x4.
Harbor Freight has a small Forstner set for under $10 thats perfect for modding Ataris.
With Forstner Bits I can cut up to 1 1/4" of a hole (my largest one) through the Atari Jr plastic (the crappiest and most brittle).