In a saute pan over medium low heat, caramelize the onion until golden brown in a little olive oil, and season with salt and pepper. Stir occasionally, about 20 minutes.
Meanwhile, mix all other meatball ingredients in a large bowl, including onions when they are done. Clean hands are the best tool to really mix well!
Form into meatballs and set on a sheet pan. They should be about ping pong to golf ball size, use an ice cream scoop to portion and then roll them with your hands. Makes 60 to 70 meatballs, depending on size.
At this point you can freeze or refrigerate meatballs, some or all, for later use in any different sauce. Freeze extra raw meatballs in containers or freezer bags, in portions right for your family (frozen meatballs should be thawed overnight in fridge before cooking).
Preheat oven to 400° F.
Over medium high heat, reheat the pan from the onions or use a clean large pan, add 1 Tablespoon olive oil. Brown the meatballs on one side, then flip and brown the other side (about 2 minutes per side, don't try to flip them too early or they will stick and break). Cook about 8 to 10 at a time in batches so you do not crowd the pan. When done browning the outside, return them to the sheet pan (don’t worry that it had the raw meatballs on them, you will be baking the whole thing).
Bake meatballs in the oven for about 10-20 minutes to finish cooking through, cut one open to check.
Add marinara sauce to the pan you browned the meatballs in to warm through, adding the meatballs in when they are done baking. Excess fat can be discarded if you like. Simmer about 15 minutes or on very low until ready to serve.
Assemble subs on hoagie buns topped with cheese. Bake at 400° on a clean sheet pan until cheese is bubbly and bread is toasted, about 10 minutes.