Ruben was born on August 18, 1929, in La Verne, California, to Juana and Jose Castro. The Castro family moved to Moorpark, California just after Ruben was born. At an early age, Ruben began to learn the lessons of service to God and community that became the example of his life.
As a boy, Ruben worked with his father and grandfather in the family store on High Street. Ruben played football for Moorpark Memorial Union High School where he graduated in 1948. Ruben was married to his high school sweetheart Dolores Arenas in 1950.