The finite element analysis software (ANSYS) is used to carry out the modal analysis of the common brace and buckling restrained brace. The advantages and disadvantages of the two kinds of support modes are compared. On the basis of modal analysis, the buckling mode of the structure is analyzed. The critical load changes and instability position are also studied. The friction coefficient of the buckling restrained brace and the strength of the filled concrete are analyzed. The results show that the bending of the Y axis is easier to occur in one core, and the stiffness and the natural vibration period of the frame structure can be obviously changed by installing a steel shaped hybrid buckling brace.