The term dates back to...

the 1930s, originally from military slang. It’s a crude metaphor suggesting someone is so close to someone’s rear end (figuratively speaking) that their nose would be “brown.” Not exactly subtle—but definitely memorable...