A Crackpot Email Archive Design for 21st Century City Halls

How should a city archive its employees' emails in a 21st century information technology environment?

As an astrophysicist, I deal with large datasets which can be updated daily with 10s or 100s of GB every day; the next generation of projects will create TBs of data daily. We need archives in astronomy, and are constantly searching ("querying") them in order to select out data of interest. Some datasets go public immediately; others are always private; while others have a proprietary period (such as the standard NASA 1-year) after which the data become public to anyone. And we store lots of additional parameters in the archives, not just the data themselves, which allow for robust searches for unusual astrophysical objects, date ranges, keywords, etc.

Archiving at city hall shouldn't be much different -- it's just a lower bandwidth problem.


Delete-Delete-Gate

Michael Kineavy, the chief of policy and planning in the administration of Mayor Thomas Menino, daily deleted all of the emails in his inbox and then deleted them again from his trash folder before the automated computer backups would run at midnight every day. The problem with his deletions is that the emails are public records which are illegal to destroy under the state's Public Records Law; Kineavy has taken an indefinite, unpaid leave of absence until the matter is resolved by Secretary of the Commonwealth William Galvin and Attorney General Martha Coakley.

On his Media Nation blog, Dan Kennedy pointed to some archival ideas in an article in the lifestyle and tech magazine Blast. The city has already announced that they are no longer solely relying on nightly backups of individual employee desktop computers in order to save copies of all the email; instead, they have started an archival system where email that is sent or received on the city's servers is instantly saved in a centralized database:

The city implemented a comprehensive program that immediately creates a record of all user messages sent and received no matter what each individual does with them. All journaled messages are kept in the archive for three years.

While I don't know the details of the city's newly deployed archival system, it sounds relatively similar to the Blast article's suggestion.

Lost in this technological solution, however, is how the storage and retrieval system can be designed in a manner that serves the public interest while reducing governmental waste.

The design of the archive is dictated by the "requirements document," i.e., the text of the state's Public Records Law. The whole current system is arcane, to put it mildly. In this 21st century, we could easily require through state law that electronic public records are archived and served to the public in a seamless, cost-effective, and transparent manner.


An Archive Design to Serve Public Documents in Real Time

A few points about the state's Public Records Law, and how it might be addressed in designing a transparent electronic archive:

• Some public records can be withheld from release according to 16 exemptions listed in the PRL. These exemptions range from personnel matters to personal information (home address of a public employee) to pending litigation. Any archival system should include a classification tag as to whether the document is fully public, or if it is exempt (along with the particular classification).
  
• Documents (letters, emails, etc.) become public records the moment they are created or received. The logical time to classify those documents as either public records is when they are first read by a person. The email application (MS Outlook, Thunderbird, Evolution, etc.) could be configured so that a user must always classify the email as public or exempt before allowing him to move to the next email in the inbox; that classification then gets stored in the archive.
  
• Once a document has been made public it is always public. If someone accidentally released a document, then the person must continue to release it. (No Uruguay Round of copyright law where works in the public domain can be reverted back into copyright.) Once a document is released, it can be tagged in the archive as such, freeing it from ever needing another review.
• State law does not require electronic documents to be provided in electronic form. The city has delivered Kineavy's 5,018 emails in paper form, then scanned the paper versions, and finally put the PDF scans on the internet. This is time-consuming, wasteful of paper, and annoying since the PDF documents are no longer electronically searchable -- a 20th century way that government offers the public its middle finger.
  
• The cost of searching and reviewing documents in response to a public records request can be charged to the requester. This is the most frustrating part of the PRL, since some records requests generate estimated costs of thousands -- or hundreds of thousands -- of dollars mostly in the charge to review all those documents one-by-one. If the search could be done using automated tools available to anyone -- inside or outside government -- then many man-hours could be saved; the member of the public now wastes his or her own time using the search tool, while the government employees are left to go get a cup of coffee instead. Or do their other job duties. The search tools are easily designed so that they only release documents that are public, and can generate summary lists of emails that are being withheld because they are exempt.
  

What does this mean in practice?

A government employee: receives an email; clicks on it to read it; when he clicks on the next email a dialog window asks him to classify the previous one first as either public or exempt; and so on. (The same goes for outgoing email: the employee has to classify it before the mail system will send it out to its recipients.) When the email was received by the city hall server it is immediately archived; when the employee classifies it, the classification is immediately entered into the city hall archive. If the email is a public document, then it is served up on the internet immediately.

A member of the public: wants to obtain some public documents; goes to the city's website; runs a search using a set of criteria (date range; email to/from/cc name[s]; keyword search; etc.); inspects the two sets of documents returned by the search engine (the public records resulting from the query and a summary table of the exempt documents); and downloads the documents he wants or starts a new query. No government employee has to waste a single moment of time responding to the public records request. Now that is efficiency in government!

What are the obvious holes in such a 21st century system? It has to be designed with the idiot government employee in mind. I don't mean to imply that all or most government employees are idiots, but that any system has to be designed in a way that the least clever employee can't release everyone's health records onto the internet. I can also imagine that some employees might think twice about sending so many three word emails when they have to classify each one before it is sent.

All the employees will complain at first about how much time it takes for them to read their email and classify each one when they're done -- although it will become quick and easy-to-do in no time. Nearly all the emails are public, not exempt, so it's usually pretty routine. (97.5% of Kineavy's emails that were found were released, with only 2.5% being withheld due to one or more exemptions listed in the PRL.)

And employees who like to make rude comments about members of the public will think twice before doing so in an email. Actually, that's a good thing.


Reality?

I am under no illusion that any politician would be so crazy as to vote yes for a law that would require government electronic records to be served up in real time on government web servers. Remember: the state legislature has already exempted themselves from the state's Public Records Law. But it is a neat thing to think about.

In this Year of Astronomy celebrating the 400th anniversary of Galileo pointing his telescope at the heavens -- and President Barack Obama gazing at Jupiter from the White House lawn -- city hall might strive to reach the heavens by creating a transparent government whose electronic public records are served seamlessly to the citizenry.


Image of a desk inside the National Archives in the 1950s by C. Getts provided by The U.S. National Archives, and image of the National Archives by marttj, both provided through a Creative Commons license.

Missing the Boat About Missing Emails

At Tuesday night's candidates forum, and several times at Thursday night's candidates forum, challenger Alex Selvig repeated his request that incumbent District 9 City Councilor Mark Ciommo release all his emails to or from Michael Kineavy, Mayor Thomas Menino's chief of policy and planning, who is involved in the Delete-Delete-Gate scandal.

Selvig responded incredulously to Ciommo's estimate that providing the documents would take some time -- saying instead that it should take only 30 seconds to satisfy his request. And challenger Abigail Furey said that Ciommo and his staff "shouldn’t be looking through [Ciommo's] emails, deciding what to release and what not to release.”

On both counts, these two challengers over-reached. And in saying that constituent service work would be compromised by having to provide the documents, Ciommo over-reached, too.


Ciommo Says Response Will Take Time

Ciommo responded Thursday to Selvig's document request by saying that he would respond and provide the documents "as if it were a public records request." (An odd statement given that verbal requests for public records are as valid as written requests, according to the Secretary of the Commonwealth's Guide to the Massachusetts Public Records Law, p.2.)

Ciommo noted, however, that he has approximately 8,500 emails on his computer, such that the process of delivering the public records may take some time -- presumably for he and his staff to search for the records, and then review them.

That explanation didn't satisfy Selvig -- who twice criticized the time estimate by arguing that the computer search should only take 30 seconds.


Exemptions in the Massachusetts Public Records Law

But Selvig missed entirely the potentially labor-intensive side of responding to a public records request: inspecting the individual documents in order to determine if each one should be withheld due to legal exemptions to the disclosure requirements of the Public Records Law; and possibly redacting specific parts of each document when that text should not legally be released under the PRL.

The Secretary's Guide identifies the exemptions to public records disclosure -- 18 exemptions listed in the law as (a) through (s) [(k) has been repealed]. The exemptions range from redacting specific kinds of personal information to non-disclosure of documents that might jeopardize public safety.

Most of the exemptions are highly unlikely to apply here. But the privacy exemption (c) may require redactions in some emails, and the deliberative process exemption regarding developing policy (d) could require withholding some documents. (Kineavy's job, after all, is chief of policy and planning.) If Ciommo chooses to invoke (d) to refuse to release any document, however, he must provide a full explanation of it -- such as provide the time/date/to/from/cc/subject for each individual document along with a brief explanation of why it is being withheld.

And Ciommo has 10 days from the receipt of the request for public documents to provide his response -- which might not be the documents themselves, but instead a description of the work effort (and possibly the estimated cost) required to provide the documents.

I suspect that Ciommo will read the writing on the walls and waive any costs related to the work as well as minimize -- if not avoid altogether -- invoking the deliberative process exemption (d). The public interest is rarely served when a legitimate request for public information is withheld pending a substantial fee, or when its release is delayed inappropriately.

The implication in this fall's campaign for city council is clear: Ciommo has every right under the Public Records Law to take days to respond to the request for documents, as well as every expectation to review those documents for privacy information that should be redacted or documents that could legally be withheld. (He may also choose to release the latter documents, anyway.)


Everyone Over-Reaching in the Heat of the Campaign

Selvig over-reached by disbelieving the possibility that Ciommo's time estimate might be reasonable -- yes, it might actually be reasonable -- and Furey over-reached by saying that Ciommo "shouldn't be... deciding what to release and what not to release," because he has those rights and responsibilities as specified under the law itself.

And Ciommo's claim that constituent service work might be compromised by the time it takes to respond to a public records request? Here Ciommo over-reached, too. He and his staff are paid by the public to do both constituent service work and respond to public records requests. If his office can't do both at the same time, then he is giving his opponents on a silver platter a reason that he should not be re-elected.

It must be emphasized that there is a substantial public interest in Kineavy's emails: recovering documents that appear to have been inappropriately deleted; providing documents responsive to a federal investigation into former state Senator Dianne Wilkerson; and providing documents responsive to challenger Abigail Furey's request, as part of the civic organization Brighton Neighbors United, into the process by which the Boston Redevelopment Authority reviewed and approved Boston College's Institutional Master Plan. There is little question that Ciommo's office should release these documents, it's really only a question of how, when, and exactly what.

Let's hope that cooler heads prevail so that the public interest can be served.


Image derived from [redacted] by stallio, provided through a Creative Commons license.

Delete-Delete-Gate

More questions were raised by a follow-up story in the Boston Globe that described the procedures that Michael Kineavy, chief of policy and planning in the administration of Mayor Thomas Menino, used to delete nearly all his emails over one particular six-month period, in apparent violation of the state's Public Records Law:

[Boston Corporate Counsel Bill] Sinnott said that all e-mails are supposed to be backed up in the city’s computer systems, but a glitch, which he said the city was fixing, allowed users to delete messages and then empty a “deleted items folder,’’ thus circumventing the nightly backup process...

Kineavy told [city officials] that he deletes all his e-mail on a daily basis and does not allow his e-mail to be saved on city backup computers, according to officials.

Note how Kineavy was not just deleting emails in his email application's inbox -- he was also going into the "deleted items folder" (or the "trash folder") to delete those backed-up versions of the emails.

And, on top of that, Kineavy was so systematic that he managed to delete all but 18 emails over that six month period so that they would not appear in the automated, nightly backups made of city hall computers. That deletion record is impressive: Kineavy must have had an incredible attention to detail, because he deleted both his inbox and his trash folder regularly and thoroughly at the end of each day before he left work for home, otherwise one or the other would have appeared in the backups.

I dub this scandal "Delete-Delete-Gate," since the issue is not just Kineavy deleting emails but then deleting them from his trash folder so that they would not be archived in city hall backups.

The Boston Herald is playing second fiddle catch-up on the story, missing how Kineavy also deleted "deleted items folder" content:

In an effort to tidy his inbox, Kineavy was deleting his e-mails every day before he left work, unaware that those e-mails were gone for good, the city insists. Until the Globe’s request, the city archive system didn’t copy records until well after the end of business each day.

Peculiarly, WBUR's Morning Edition today reported that city officials say that the deletion of emails was "unintentional."

Calling delete and then delete again unintentional? That explanation does not pass the sniff test. I smell a stinker here.


Image of "Miscalculation" by .::HIMU:: provided through a Creative Commons license.

Hey Globe, Where Have You Been for the Past 16 Years?

The Boston Globe buried a story on page 20 of their paper that is probably the biggest shot of this election campaign: that several of Mayor Thomas Menino's top officials have been deleting all their emails on a daily basis, thereby likely violating state law by destroying those public records.

A quick look through my own email shows nineteen emails either to or from two specific city officials cited in the Globe article -- and a twentieth email on mayoral candidate Kevin McCrea's blog -- that should have been archived during the past two years, but appear not to have been. On the face of it, those are possibly twenty specific violations of the state Public Records Law that each carry a potential penalty of up to $500 and/or one year in prison.

After reading the story, I am left with a jaw-dropping question: Why did it take the Globe 16 years to figure out that Menino's closest adviser was deleting all his electronic records?

I am dumbstruck that the paper of record in Boston does not appear to have requested such electronic public records from Michael Kineavy or Thomas Tinlin during all this time -- or at least realized what was going on when all their public records requests came for naught.

Hey Globe, where have you been all this time?

The Globe deserves some credit, nonetheless, for doing some serious investigative reporting this year leading up to the mayoral election later this month. It is important for residents to know what their city and elected officials have been doing so that they can make an informed choice to re-elect Menino or choose someone else.

The Globe mentions Michael Kineavy, the city's chief of policy and planning, and Thomas Tinlin, the commissioner of the transportation department, as producing zero emails in a request for all emails from a specific time period -- i.e., no incoming or outgoing emails. The Globe story states that the city has admitted the problem, and the corporation counsel, William Sinnott, is in the process of addressing it.

Public records of all kinds must be maintained for a minimum of two years under the state's Public Records Law. While it might appear at first to be difficult to prove that a person has deleted public records -- because the files are, by the nature of the violation, deleted -- there are independent ways to determine if such public records should exist, because other people are on the list of recipients or senders of those emails.

A quick look through my email indicates:

• Thirteen emails, written by other people, addressed to Tinlin in the time period 7/23/07 - 8/24/08;
• Two emails written by Tinlin on 8/11/08; and
  
• Four emails, written by other people, addressed to Kineavy between 12/14/07 and 5/15/08.
  

That's 19 specific cases of emails that should have been archived as public records under state law, but appear not to have been; McCrea's letter makes it an even 20. I'm sure that other people around the city can produce their own lists of emails, sent to or received from these two city officials, that ought to be archived by the city as public records. An easy way the Globe could find more such emails would be to submit a public records request, say, to the Boston Redevelopment Authority for all emails written by or received by BRA employees that include Kineavy and/or Tinlin in the To/cc/From header fields.

Left unanswered in the Globe story was the motivation of both Tinlin and Kineavy in systematically deleting these email public records. Were they trying to hide the trail behind controversial (or worse) actions on their part? Until we hear more from them directly to explain their actions, this story will not likely go away.

There is no question that city employees should be saving their emails. An easy way? Move them into folders when you are done reading them in your inbox. That's what most other people do.

As for the Globe? When you finally get a scoop like this, why do you bury it on page 20 of the Sunday paper?


Update: Boston City Councilor-At-Large and mayoral challenger Michael Flaherty has announced that on Monday he will request state Attorney General Martha Coakley and Suffolk County District Attorney Daniel Conley to investigate the allegedly improper or illegal deletion of electronic public records by officials in the Menino administration.

Update 2: Flaherty will be joined by both of the other mayoral challengers, Kevin McCrea and Councilor-At-Large Sam Yoon, in calling for the investigation during a Monday morning press conference.


Image of a paper shredder by Shane Vigil provided through a Creative Commons license.