everything lies in totalization agreement with Swizerland

According to agreement, you can accumulate the credits under the employements in both countries. So your credits you accumulated for working in Swizerland also counts for the credits wored in USA. But you probabaly should use those credits towards the SS benefit in one of the countries you worked for, not both. That is something the SS agency may overlook. 




http://trac.syr.edu/immigration/library/P1077.pdf

As shown in Tables 1 and 3, alien workers and alien dependents/survivors may

receive payments while living outside the United States if they are a citizen or

resident of a country with which the United States has a totalization agreement.23

Section 233 of the Social Security Act authorizes the President to enter into a

totalization agreement with a foreign country to coordinate the collection of payroll

taxes and the payment of benefits under each country’s Social Security system for

workers who split their careers between the two countries. For example, without a

totalization agreement, an individual who is sent by a U.S. company to work in a

foreign country (and his or her employer) must contribute to the Social Security

systems in both countries, resulting in dual Social Security coverage and taxation

based on the same earnings. With one exception (Italy), totalization agreements

allow workers (and their employers) to contribute only to the foreign system if the

worker is employed in that country for five or more years, or only to the U.S. system

if the worker is employed in that country for less than five years.

Totalization agreements also allow workers who divide their careers between

the two countries to combine earnings credits under both systems to qualify for

benefits if they lack sufficient coverage under either system.24 While a worker may

combine earnings credits to qualify for benefits under one or both systems, his/her

benefit is prorated to reflect only the number of years the worker paid into each

system. The same treatment applies to foreign workers in the United States.