The detection of malicious software is an essential activity for ensuring the ongoing security of the Internet of Things (IoT). Malware, often known as malicious software, is computer code designed to steal private information, corrupt data or files, or otherwise interfere with the functioning of a computer system. A number of different methods, including signature-based detection, heuristic analysis, and behaviour-based detection, are utilised in the process of identifying malicious software. The process of searching for specific patterns or signatures that are characteristic of known malware is what is referred to as signature-based detection. The success of this method is dependent on the upkeep of a database containing known malware signatures. Heuristic analysis, on the other hand, uses rules and algorithms to detect suspicious behaviour that may indicate the presence of malware. This approach does not rely on specific signatures but instead analyzes the code or behaviour of a file or program. If the behaviour exhibits characteristics commonly associated with malware, it is flagged as potentially malicious. Behaviour-based detection focuses on analyzing the behaviour of software programs to identify anomalous or malicious activity. Utilizing machine learning algorithms for malware detection helps to identify potential threats and prevent them from compromising valuable data and information. In this paper, we propose a robust approach for the detection and analysis of malware attacks in the communications of the IoT. In the proposed protocol various machine learning-based models, i.e., k-nearest neighbours (KNN), random forest and neural networks are used. The proposed protocol is also compared with the other similar existing protocols and it has achieved better accuracy as compared to the other existing protocols.