Dorff was born in Atlanta, Georgia, the son of Nancy and Steve Dorff, who is a composer and music producer. His father is Jewish, and his mother was Catholic, and Dorff has stated that he was "kinda brought up half-Jewish". Dorff's brother Andrew (1976–2016) was a country music songwriter. He was raised in Los Angeles, where his father worked, and began acting as a child, appearing in commercials for Kraft and Mattel. Dorff attended several private schools, and was expelled from five of them.