Thus the drug might be tested on 1000 patients and the results might not show a statistically significant positive result for a given problem. This is done by taking a representative sample and collecting huge number of parameters related to the test subjects, so that the drug can be claimed and correlated to the problem in some form or the other.ĭata fishing can also be done by narrowing down the sample size to include those results that bear out the intended hypothesis. Therefore the company uses data snooping to project claims that are not actually true, even though the data confirms the claim. However, it needs to market the drug in order to make profits from it. Industries that are heavy on data mining are many times involved in data dredgingįor example, a drug company might spend millions of dollars on a drug but it may not show the kind of results that were initially expected. In such cases, data dredging is said to have taken place.

However, owing to the sheer number of hypotheses on a limited data set, it is possible to see some very highly correlated data that are statistically significant. If the data is totally random, then say all the hypotheses are actually false.

Suppose there is a given data set and there are a huge number of hypotheses about this data set. The Way Data Snooping Works is as Follows:

There are a number of reasons for data snooping and it is a matter of grave concern as it uses statistical principles for the purpose of drawing misleading and false conclusions. Data dredging is usually followed by the researcher who wants to try and ‘prove’ a point of view that might not hold or might not be shown in by the actual data.

Data dredging meaing