Legal Question Answering (LQA) aims to automatically answer questions in the legal domain, which is a challenging and meaningful task for Legal Intelligence. Most previous work modeled LQA as a retrieval task, which could obtain irrelevant answers due to the limitations of pre-constructed databases. In addition, it is costly to maintain an up-to-date database as legal knowledge is constantly being updated. To address these issues, we propose the Generation-based Method for Legal Question Answering (GLQA). GLQA instead models LQA as a generation task to flexibly produce new relevant answers specific to each question. To further make the answers more controllable and informative, GLQA incorporates laws as external knowledge into the answer generation process. Specifically, our method contains a retriever and a generator. The retriever is used to retrieve applicable law articles as external knowledge and the generator aims to generate answers with the help of external knowledge. The retriever and the generator are integrated into a single T5 model using multi-task learning to make them mutually reinforcing. Our design enables the model not only to produce highly relevant new answers but also to keep the knowledge in the answers up-to-date by modifying the law database at a low cost.