According to agreement, you can accumulate the credits under the employements in both countries. So your credits you accumulated for working in Swizerland also counts for the credits wored in USA. But you probabaly should use those credits towards the SS benefit in one of the countries you worked for, not both. That is something the SS agency may overlook.
http://trac.syr.edu/immigration/library/P1077.pdf
As shown in Tables 1 and 3, alien workers and alien dependents/survivors may
receive payments while living outside the United States if they are a citizen or
resident of a country with which the United States has a totalization agreement.23
Section 233 of the Social Security Act authorizes the President to enter into a
totalization agreement with a foreign country to coordinate the collection of payroll
taxes and the payment of benefits under each country’s Social Security system for
workers who split their careers between the two countries. For example, without a
totalization agreement, an individual who is sent by a U.S. company to work in a
foreign country (and his or her employer) must contribute to the Social Security
systems in both countries, resulting in dual Social Security coverage and taxation
based on the same earnings. With one exception (Italy), totalization agreements
allow workers (and their employers) to contribute only to the foreign system if the
worker is employed in that country for five or more years, or only to the U.S. system
if the worker is employed in that country for less than five years.
Totalization agreements also allow workers who divide their careers between
the two countries to combine earnings credits under both systems to qualify for
benefits if they lack sufficient coverage under either system.24 While a worker may
combine earnings credits to qualify for benefits under one or both systems, his/her
benefit is prorated to reflect only the number of years the worker paid into each
system. The same treatment applies to foreign workers in the United States.
