Rosalie Silberman was born on July 1, 1946, in a displaced persons camp in Stuttgart, Germany, to Jacob Silberman and Fanny Krongold Silberman, both Polish Holocaust survivors. Her father, born in 1910 in Sienno, Poland, had studied law at Jagiellonian University in Krakow during the 1930s, completing eight years of legal training (four academic, two articling, two clerking), but was barred from practising due to antisemitic restrictions including the numerus clausus Jewish quota. He managed his in-laws' factory in Ostrowiec Swietokrzyski before the war. Her parents were separated into concentration camps in 1942, each surviving three years of internment — her father at Theresienstadt and her mother at Buchenwald. Their first son, Julius, aged two, was left with a Polish caretaker who brought him to his paternal grandparents; he was deported to Treblinka along with his grandmother and uncle and murdered there. Jacob Silberman's parents and three younger brothers were also killed at Treblinka.
After liberation, Jacob Silberman went to Stuttgart where Allied authorities hired him to help establish the system of legal services for displaced persons. He taught himself English. While at the camp, he introduced Eleanor Roosevelt during her visit in 1948. The family immigrated to Canada in 1950, settling in Toronto. Jacob was again barred from practising law due to non-citizen status and credential requirements; Allied authorities had falsely certified him as a shepherd, then a tailor's cutter. He eventually retrained as an insurance agent, working at Queen and Spadina in Toronto. He died of cancer at age 60, before Rosalie completed law school. Her mother, Fanny, became a real estate agent specializing in writers and artists after Jacob's death, and lived to age 92.
Rosalie grew up in Toronto's Jewish community. She decided at age four to become a lawyer after learning her father had been denied that right. She was required to read three library books per week and was an exceptional student, scoring eleven first-class honours in secondary school. She had a younger sister, Toni, born in 1948.