Abstract: There are over 6000 rare diseases in the world, affecting more than 300 million people. Early and precise diagnosis of rare diseases has always been the goal in clinical medicine. Emerging computer vision technology now greatly enhance medicine and healthcare and shows the potential in assisting the diagnosis and treatment for rare diseases. The technology can be a useful tool for extracting disease-relevant patterns from medical imaging. However, the effectiveness of its application depends on the complexity of the medical cases. In this paper, we summarize the challenges and emerging solution for the application of computer vision in diagnosis, rehabilitation as well as management of rare musculoskeletal diseases.