Competing edge structures of Sb and Bi bilayers generated by trivial and nontrivial band topologies

One-dimensional (1D) edge states formed at the boundaries of 2D normal and topological insulators have shown intriguing quantum phases such as charge density wave and quantum spin Hall effect. Based on first-principles density-functional theory calculations including spin-orbit coupling (SOC), we show that the edge states of zigzag Sb(111) and Bi(111) nanoribbons drastically change the stability of their edge structures. For zigzag Sb(111) nanoribbon, the Peierls-distorted or reconstructed edge structure is stabilized by a band-gap opening. However, for zigzag Bi(111) nanoribbons, such insulating structures are destabilized due to the presence of topologically protected gapless edge states, resulting in the stabilization of a metallic, shear-distorted edge structure. We also show that the edge states of the Bi(111) nanoribbon exhibit a larger Rashba-type spin splitting at the boundary of Brillouin zone compared to those of the Sb(111) nanoribbon. Interestingly, the spin textures of edge states in the Peierls-distorted Sb edge structure and the shear-distorted Bi edge structure have all three spin components perpendicular and parallel to the edges due to their broken mirror-plane symmetry. The present findings demonstrate that the topologically trivial and nontrivial edge states play crucial roles in determining the edge structures of normal and topological insulators.