June 21, 1999

U.S. Senator John McCain
241 Russell Senate Office Building
Washington, D.C. 20510

BY MAIL AND BY FAX
TO (202) 228-2862

Re: S.97, Secure Computing Corporation (SmartFilter)

Dear Senator McCain:

I write about your visit last Friday to Secure Computing Corporation, and their demonstration of SmartFilter, their Internet filtering software product. It is my belief that the company may have misled you about their product, and particularly about an extensive evaluation of it by The Censorware Project <http://censorware.net> of which I am a member.

[JST note: I have deleted from the web posting a personal note included at this spot in the fax. Other than that, the text is identical to what was faxed.]

To the matter at hand: In March, The Censorware Project released a report analyzing 31 days of actual user data of SmartFilter as deployed by the Utah Education Network (UEN) for public schools (and some public libraries) in Utah. We obtained the data (with individual user information redacted) under the Utah Government Records and Management Act, Utah’s version of the Freedom of Information Act. UEN electronically file-transferred to us logs of approximately 54 million separate web accesses, with the logs showing, among other things:

Our report was highly critical of SmartFilter, finding that it blocked access to materials which are not only Constitutionally protected, but which should be part of the curriculum for school children. Such blocked material included sites containing the following, among many others:

Further, our report demonstrated that SmartFilter and other similar products are, to a large extent, (flawed) solutions in search of a problem. As we wrote in a portion of the report:

"Banned accesses made up less than 1% of overall accesses, most of which were banner ads presumed (by the software) to be sexually explicit. Very few people used the internet to access sexually explicit material, and students were the least likely to do so. It thus appears that the stated problem of minors accessing sexually explicit material (inadvertently or deliberately) is considerably less than some organizations would have the public and the Congress believe."

The full report is online at <http://censorware.net/reports/utah/>.

Secure Computing’s immediate reaction to our report was to do what censors do: they banned not only the report itself from users of their product, but all of The Censorware Project’s website. Fortunately, the Salt Lake Tribune had mirrored the report, and UEN later overrode the SmartFilter ban, so the citizens of Utah were able to read the report.

Of particular note is that the front page of the report always has stated that anyone who wanted access to the data we analyzed could have it for only the cost of duplication. Yet no one from Secure Computing ever requested the data. Nor could they have obtained it from UEN, since UEN’s electronic records retention policy was such at the time that they would have deleted the logs from their systems by the time our report was published. Consequently, Secure Computing had nothing to work with except for our published report. And with nothing other than the published report with which to work, they said absolutely nothing in public until last Friday, not coincidentally the day you visited their offices.

On June 18, they issued a press release entitled "Censorware Project Unequivocally Confirms Accuracy of SmartFilter™ in State of Utah Education Network." The release is online at <http://www.securecomputing.com/C_PR_SF-censorware.html>. In pertinent part, it says:

"San Jose, CA, June 18, 1999 -- The Utah Education Network (UEN), which oversees all Internet Access for the state of Utah's 500,000 K - 12th grade students, is nearly 100 percent accurate in its judicious oversight of students' access to sites on the Internet, according to a recent report by the CensorWare Project. The UEN has standardized on Secure Computing SmartFilter™, the leading Internet filtering and systems bandwidth management tool. Using SmartFilter, the UEN is able to deny student access to sites that are deemed inappropriate, based on where they appear in one of the 27 SmartFilter Control List Categories - which include pornography, hate speech, gambling and criminal skills. The report details findings of its exhaustive and thorough review of student Internet use during a two-week period in October of 1998. During that period of time, there were over 54 million web access attempts and of those, according to the report, less than 300 were denied access because the site contacted had been mis-categorized. This represents a stunning accuracy rate of 99.9994 percent."

"‘The comprehensiveness of the research of The CensorWare Project and its subsequent findings validate just how valuable a tool SmartFilter can be for schools, businesses and any organization that is looking to implement an Internet Use Management Policy,’ said Gus Maldonado, product marketing manager at Secure Computing."

A full copy of the press release is enclosed with this letter.

In one respect, we agree with the press release: our review of the logs of student Internet use was indeed "exhaustive and thorough". To our knowledge, no person or organization ever has undertaken a similarly large review of actual user data, or anything which even approximates what we accomplished with the Utah data. Thus, it is particularly galling to see Secure Computing claim, three months after the fact, and with no basis in reality, that they have a "stunning" accuracy rate of 99.9994 percent; even more so because, given the non-coincidental timing of their release and your visit, it may be presumed that Secure Computing attempted to convince you that they were so accurate.

What Secure Computing did was simply to take the total number of web accesses (approximately 54,000,000) and divide it by the number of wrongfully blocked sites (a little over 300), the result of which is their 99.9994 percent "accuracy" rate. Mark Twain (some of whose writings are blocked by SmartFilter) once remarked upon lies, damn lies and statistics, and Secure Computing’s creative math would do him proud.

Suppose that one surfs to, say, http://cnn.com. Each time one accesses that page, one also accesses all of the .gif, .jpg and other files associated with that page. Thus, a single hit on CNN’s front page easily could generate dozens of “web accesses”. Further, the 54,000,000 web accesses we analyzed obviously include multiple hits to the same page by multiple users. Consequently, for example, if one hit on CNN’s front page generates 36 web accesses (which is exactly the number as I write this -- 1 access to text plus 35 image files), and if the CNN front page was hit 10,000 times during our review period (a very conservative number for all Utah public schools and eight public library districts over a one-month period), then the surfing to that one page alone would generate 360,000 web accesses in the logs we analyzed.

In marked contrast, the 300 or so sites we identified as being wrongfully blocked are, in fact, separate and unique sites. Regardless of whether there was only one access attempt to a wrongfully blocked site or fifty attempts, we counted the site only once. Thus, dividing 54,000,000 by 300 yields an impressive-sounding number, but it is one which bears no relation to any meaningful analysis.

A more meaningful analysis would be to take the 122,700 separate web pages which were blocked by SmartFilter during our review period and divide by the 6,465 directories wrongfully blocked by SmartFilter — since a site can, and often does, consist of multiple directories or pages, this is the "apples to apples" comparison. That more meaningful math results in one in 19 pages being wrongfully blocked, an error rate of slightly more than five percent. (The real error rate may well be higher, since we are a small group without the resources to examine each of the 122,700 blocked pages, but we will stand on these numbers for purposes of this letter.)

Please note that Secure Computing has not challenged any of our categorizations as wrongful blocks. To the contrary, they have praised our thoroughness. Also, please note that we have not characterized blocks on sites such as www.hustler.com as being wrongful. Although the speech at that site is constitutionally protected, we recognize that UEN’s primary audience is school, and we do not argue that either Hustler magazine or its web site should be in schools. Thus, our one in nineteen figure is limited to materials which would be appropriate for UEN’s main audience.

In short, if Secure Computing tried to sell you on their claimed accuracy rate, as seems highly likely, they tried to sell you a bill of goods. They know not of what they speak, and they could not even be bothered with requesting our underlying data before inventing false and frivolous statistics.

Finally, I should note that SmartFilter’s accuracy, or lack thereof, is not exceptional. Since The Censorware Project was formed 1½ years ago, we have examined a number of filtering software applications in addition to SmartFilter, including Cyber Patrol (the industry leader), WebSENSE (used by some federal courts) and X-Stop Librarian II, (the software used in the Loudoun County, Virginia Public Library). In each case, we have found significant flaws with the software, hundreds or thousands of highly inappropriate blocks of perfectly innocuous web sites. The reason for this commonality is simple: the claims of the vendors notwithstanding, no filtering software company has, or could have, the resources to human-review every new entry to their list of blocked sites. The web consists of hundreds of millions of pages, and vendors typically have blacklists of more than 200,000 entries, constantly changing, with many of those individual entries serving as a block on hundreds of thousands of web pages. There simply is no filtering software company which employs even a fraction of the number of people necessary to keep up with their own lists, let alone with all of the Internet. Some day, the technology to get it right without substantial human review may exist, but that day will be in the future.

I ask only that you not let yourself, your staff or others be misled by the false claims of Secure Computing and others when you take further action on S.97. There is a wealth of factual material available at The Censorware Project website. The other Project members and I would be more than willing to discuss with you or your staff at length our investigations into and findings about filtering software. Thank you for your consideration.

Very truly yours,

JAMES S. TYRE

JST:hs
Enclosure

(Note: the Censorware Project's domain name has changed since this letter was sent; this copy was updated to the new domain.)