The design of an anomaly/outlier/novelty detection system, for a given application, fundamentally requires, on the one hand, the construction of a numerical score from available datasets, to rank data points as to their closeness to being normal, and on the other hand, the statement of a policy, based on the score, for declaring anomalies. The task of designing an anomaly score is challenging because the probability distribution of what may be considered normal cannot always be assumed a priori, especially in dynamic environments like monitoring a computing system. Hence, most anomaly scores in use presently are based on some form of a direct distance between the data points. The use of the Kullback-Leibler (KL) divergence, for probability distributions, along with a windowing scheme, is explored in this paper, for the design of anomaly scores. Distributions are built from the frequencies of a metric in a given time window. For context, KL is used to compare the distribution of the current window with that of a linear combination of several preceding windows. Ensemble-like scores may be built by using the method on independent metrics. Experiments are conducted to test the proposed method on a University of Victoria dataset and a synthetic dataset.