The club was created as Thornhill United. For many years the leading team in the region was Rotherham Town, who spent three years in the Football League while Thornhill United were still playing in the Sheffield & Hallamshire League. By the turn of the century, however, Rotherham Town had left the Football League and briefly gone out of business prior joining the Midland League. Meanwhile, Thornhill's fortunes were on the climb to the extent that in 1905 they laid claim to being the pre-eminent club in the town and changed their name to Rotherham County. For an era both teams competed in the Midland League, ending first and second in 1911-12. When the Great War finished, it was decided to lengthen The Football League by creating four new places in Division Two. Rotherham County, who had been winners of the Midland League in 1915 before the competition was suspended, were elected to one of the vacancies.

The Second Division was rather too hard for the Yorkshire outfit and in 1923 they were demoted to Division 3 (North). In 1925 the club finished bottom and had to apply for re-election. Rotherham Town were themselves suffering and it was clear that to have two professional clubs in the town was not sustainable. Talks had commenced in February 1925 and in the beginning of May the two clubs merged to form Rotherham United. Days later the reformed club was formally re-elected under its new name.

In a new amber and black strip, United may have started with optimism but the new team fared little better than the old one. The now familiar red and white was adopted around 1928 but there was no improvement in the teams fortunes: in 1931 they again had to apply for re-election. Immediately after the Second World War things looked up. After adopting Arsenal-style white sleeved shirts, United finished as runners-up three time in succession between 1947 and 1949 and then were champions of Division Three (North) in 1951.

Rotherham reached their highest ever league position of third in the Football League Second Division in 1955, when only goal average denied them a position in the top flight after they finished level on points with champions Birmingham City and runners-up Luton Town.

The club held on to its place in Division Two until 1968 and then went into a fall that took them down to Division Four in 1973. In 1975 they were promoted back to the Third Division winning the title. In 1981/82, Rotherham were once again in contention for promotion to the top flight but eventually finished 7th, some 4 points off 3rd place.