First of all, hough transformation can be used. just parametrize the representation for rectangle, but the parameter space is 4D. Second, line detection, followed by checking corner degree. In practice, I would use opencv's coutour fitting function to fit for quadrilateral, then check the angle. This works quite well.

I would scan the image with a basic edge detection mask. Then I would scan to count the number of lines, keeping track of end points in pairs. Cases such as curves would also be found in this step and returned as not a rectangle. An analysis of basic trig with the points could then be performed in order to determine if the points form a rectangle. Once a rectangle is confirmed then you could use those points to display on the image the recognized rectangle. Or do whatever the reason for finding the rectangle was.

By a custom rectangle detector I mean how would you write your own function to detect rectangles. Your input would be the pixel data and your outputs would be something like the x,y locations of the rectangles.

I suggested something about matching intersecting hough lines.