brokers use online system to clear customs for importers. so they can clear customs wherever in the US if they have the permit to clear for outport entries. (that means brokers with permit can clear the freight going to new york even they are located in los angeles.) what you should concern is if there is something happen, the broker should solve the problem as quick as possible.
since you are located in virginia, i suggest you to have your freight discharged in newark, nj, and then truck down to your warehouse. inland trucking charges are expensive if you want it goes to los angeles.
you do not need to go to port, just find a good freight forwarder, they should have good relationship with customs broker. let them take care of it.