Big Data in ICT4D – Examples from Ukraine’s Libraries

Yesterday (July 11, 2012) the UN Global Pulse hosted a roundtable on Big Data and Global Development. This follows the release of their white paper – Big Data for Development: Opportunities & Challenges – on May 29, 2012. In this same spirit, I would like to present some products of IREX Ukraine’s quest to become more data driven. The last paragraph of their blog post announcing the white paper is worth quoting directly:

“It is important to recognize that Big Data and real-time analytics are no modern panacea for age-old development challenges. That said, the diffusion of data science to the realm of international development nevertheless constitutes a genuine opportunity to bring powerful new tools to the fight against poverty, hunger and disease.”

The white paper presents a number of key dialogue points for the movement, all of which are very applicable to global library development. The first asks what types of new, digital data sources are potentially useful to the field of international development? In this case the data source is from computers in libraries in Ukraine. Each of the computers that the Bibliomist – Ukraine program provides to libraries in Ukraine has software included to provide impact data back to our team. Upon deployment, the librarian completes a short questionnaire, including information on the library’s location and the size of the community in which it resides. Also, the software reports to our database every 15 minutes; a 1 if the computer is actively being used, a 0 if it is on but not in use, and Null if it is switched off. This simple ‘hand raising’ exercise from each of our 700 libraries provides a wealth of data. Over the course of the first half of 2012, Bibliomist has certainly accumulated a big data source that provides insight on public computer usage in international development settings.

The second question posed in the UN’s white paper is what kind of analytical tools or methodologies for analyzing Big Data have already been tried and tested by academia and the private sector, which could have utility for the public sector? Like most big data analysis, I use a combination of tools to retrieve, format, and visualize our project data. The raw data is stored in a MySQL database, and through a variety of views is exported into a csv file. From there, I use a few Perl scripts I had laying around from graduate school to clean things up. I then produce a monthly workstation usage report to our project team, which ranks libraries according to Oblast. This Excel spreadsheet allows the whole team (representing a wide range of technical expertise) to be comfortable making more data driven decisions. For the fun stuff, I bring my data into R. Using the wonderful IDE, R Studio, I get down to visualizing the activity of over 2000 computers in libraries across Ukraine. Specifically I use the ggplot2 visualization package written by Hadley Wickham to produce the following graphics. The following animations are not meant to be comprehensive data driven representations of the program, but provide what I feel are compelling displays of the beauty to be found in visualizing the data in our development work.

Click each of the images to watch the animation in full size.

(WordPress resizing the gif files removes the animation, but I think they lose too much visually if sized down)

This first animation shows the evolution of computer usage by month based on the size of the library’s community. Workstation usage is taken only during working hours for a library (8am – 8pm). As can be seen, the vast majority of the computers are in rural locations. There is a surprising amount of change from month to month.

This animation is the same as the first, only each of the 25 Oblasts of Ukraine are graphed separately. The variation between Oblasts is striking; some have many more computers than others, others are much more urban. In this visualization you can almost feel the project breathing from month to month. The binary report from each computer comes together to paint this full picture.

In this final graphic, we again return to the location size vs. performance. Each of the dots represents a library. The box and whiskers is used to mark the nature of the distribution in workstation usage. Over the months, libraries move fairly significantly. Some very noticeably separate themselves from others, most noticeably in the village community size. We can now reach out and specifically find out what aspects of programming are making these libraries successful and how can they be emulated?

Read Full Post »

Libraries Bring E-governance Services to the Public

A patron connects with her government online at the local library.

by Megan Volk, Bibliomist Senior Program Manager

Ask Ukrainians about the last time they’ve been to their local library and some may tell you about their days as a student bent over an open encyclopedia. But ask others and increasingly they will tell you how a visit to the library earned them money, saved them time, or just made their lives easier. Valentyna in Kirovograd may tell you how a librarian helped her find an error in her pension calculation, which she corrected to see an increase in her monthly payment. Viktor, a farmer from the village of Izmailivka, may tell you how he went online to use the Ukrainian Agricultural Portal to find a buyer willing to pay top dollar for his corn.  Iryna from Sumy may tell you about how she used the state employment service’s website to find a job at her local post office.

The Ukrainian government is taking steps to make the lives of citizens easier by placing access to services and information online. Unfortunately these efforts are undermined;  according to World Bank data from 2011 less than 6% of the population has a fixed Internet subscription. In late 2011, we conducted a survey that indicates that a large number of public libraries across the country are helping to bridge this divide by providing not only internet access, but assistance to patrons in utilizing e-governance services.

Based on completed questionnaires from 246 librarians working in libraries with computers and internet access across the country, 73% reported offering some kind of e-governance services. Of those offering such services, 97% help users access links to national and local level government and 48% report providing training to users on searching for government information online. Librarians report that patrons are most interested in information closely affecting their lives, with pensions topping the list: 97 percent of librarians responded that this was an issue of particular interest among their patrons.

Many libraries are not only helping patrons find the information they are looking for, but taking their efforts a step further and helping the government to develop and promote its new services to the public.  68% of those libraries offering e-governance services report cooperating with government officials in helping to disseminate information, and 17% report working with the government in the creation of online sites and tools. In some libraries the exchange of information is a two-way street, with 25% of those respondents reporting that they routinely provide government with feedback and suggestions from their patrons.  Of these, 18% (45 respondents) said the government acted on at least one of their suggestions. 44% of respondents report that their work in e-governance has improved their relationship with local officials.

In addition to serving as access points to e-governance information and tools, libraries also continue to serve as a physical meeting place, connecting citizens to their government officials and elected representatives. 72% of all respondents report holding roundtables in the library attended by government representatives, and 32% report holding seminars on different government policies and procedures led by experts in the respective field.

We’re excited to have the opportunity to share this information with our partners and demonstrate how powerful this collaboration between libraries and government can be as both work to connect citizens with the information they need.

Read Full Post »

Frontline:SMS to Google Spreadsheet Reporting Tool Part 2

This is the second post about our new reporting tool in Ukraine, which uses Frontline SMS and GoogleDocs to track information that our regional representatives collect in the field.

Here is the link to Part 1

You can download the completed and documented script here.

I will now walk through the code itself and describe the purpose of each piece.  Unfortunately our blog can not currently imbed inline code, so I will just post images for the time being.  This is going to get into some basic computer code; just a warning.  If this sort of thing is new to you, these resources may be a simpler and more basic introduction to the same sort of tool that I built (first and second).  To get this tool running you will need to download a few things.  Thankfully they are all free.

1. Download and install the latest version of Frontline:SMS
2. Make sure you have the latest version of Python installed.  We will be working with the Google Data Python Library.  Follow the instructions found here to get things running.  You will need to have both Python and the Google Data Library.
3. I recommend making sure that you can get a simple Python script to communicate with a Google Spreadsheet by following the hello world example, or by running some of the example programs that come with the Google Data Python Library.
4. I also recommend making sure that you can install Frontline:SMS on your computer and get it sending and receiving text messages with your GSM modem before moving on.

Now that Frontline:SMS is up and running and you can get a Python script to connect to the Google Spreadsheet API, lets work on coding something that links the two together.  This code represents just one way to accomplish this, and is by no means perfect or optimized.  However, it does get the job done.  Open up my script (found here) with your favorite editor and lets walk through the different portions.

Read Full Post »

Frontline:SMS to Google Spreadsheet Reporting Tool Part 1

Coverage Map Courtesy of MTS Ukraine

The drive for data is a central motivation in large, dispersed ICT4D projects such as Bibliomist in Ukraine. The distances and number of project sites for Bibliomist are both vast and challenging, especially given the infrastructure realities in the less developed regions of the country. When it comes to data reported from the field, too much is never enough in my book. I constantly look for tools to improve and streamline our reporting mechanisms. It might be tempting to think that the best route to accomplish this is to simply gather more data, however in many cases the best approach is to simplify the data gathering process.

Currently, our regional staff usually reports to our central team either via email, updating a shared google document, or by direct cell phone call.  The problem here is that data can be delayed a few days by the reporting process, or by a lack of Internet connection.  When visiting libraries in rural locations, reps must travel back to a location with an Internet connection to report on what they found at the library site.  Reporting data can vary significantly, but this particular instance was to improve the reporting efficiency for pre-installation surveys.  A regional rep must verify a number of conditions for each library that applies to our program to ensure that facilities meet our requirements.  For example, this particular survey contains questions such as ‘what is the available Internet connection?’; ‘are there burglar bars on the doors/windows?’; and ‘is there enough physical space for the computers?’  I wanted a technological solution that would allow the reps to report on this survey without having to transcribe answers into an email, or worry about the availability of an Internet connection.  I also wanted the survey responses to be organized and parsed automatically and not have to be done so by hand by our monitoring and evaluation team.  Currently, reps would wait until they were home to work on the actual reporting of the survey.  This tool will allow them to report on the survey results from the library site itself.

With this goal in mind, I wanted to make it easier for my regional staff to report data to myself and our impact team. Some of my key constraints:

1. I did not want to reinvent the data organizing tools already in place by our impact team. (Don’t fix what isn’t broken)
2. I did not want to mandate new technology to the regional staff. (Outfit them with smart-phones or require learning a new program)
3. I wanted the tool to be accessible to all regional staff regardless of technology comfort level and language preferences. (Keep it simple)
4. I wanted reporting to not be dependent on an active internet connection.
5. I did not want to spend (much) money.

Taking these things into account, I began researching tools that are already out there. Constraint #1 meant feeding reports through google docs, which requires playing around the Google Documents API. This works in this case as we do not require the power of a stand alone database, and the cloud hosting inherent to gdocs solves a lot of the backup and access worries. #2 meant  I wanted to bring data gathering to a technology and process our regional staff already use everyday. In this instance, my regional reps are my ‘customers’ and a quick user-needs study brought back a clear direction. It became apparent that the only common tool available to all 25 representatives is a basic mobile phone with only SMS (text message) and voice capabilities. Even java based form reporting (such as this example) would require buying new phones for some of the staff. #3 kind of fell in line after #2; I knew I’d be crowd-sourcing SMS’s at this point. #4 amounts  to the fact that it is very difficult to mandate that staff inform you that the internet is down at a particular library if the only way they can effectively report this is through an internet connection. While internet connections can be unstable in rural Ukraine, cell phone coverage is more or less ubiquitous. For #5 I was half way home by using Google Docs, there are plenty of open-source tools laying in wait out there, but which to choose? I settled on a trusty-veteran of the text message world: Frontline:SMS.

I decided to build a reporting system that would tie together the  ubiquitous functionality that Frontline:SMS puts in the hands of each of my field staff with the cloud ‘database-on-the-cheap’ that is Google Docs. I wanted reps to be able to text message in the status of library locations by parsing the message contents. I used the following items in building my tool:

• Frontline:SMS
• Google Spreadsheets Python API
• A USB GSM Modem

Frontline has a very nice feature where an incoming SMS can be used to trigger an external command. I wanted this command to be a call to a python script I would write, and the command line arguments would be the data that the script passed along to a Google Spreadsheet. To do this, I would need a few helpful Python tricks.

• gdata to give me all the google api calls and authentication commands
• optparse  to get data from the FL:SMS command line call into my script
• urllib to clean up an encoding issue so that I can properly parse the text message’s contents

In my next post I will walk through my code and give a tutorial on how I built this tool, and how to install it. In the mean time, you can download the completed, and documented, script here.

Read Full Post »

Open government: the demand side in Moldova

Yesterday, I had the fortune to moderate a salon at IREX’s office in Chisinau on the theme of libraries and open government. As we are getting ready to start the Global Libraries program in Moldova, we hosted the informal gathering in order to start a discussion about where libraries fit as more of Moldova’s government services and information goes online. We were joined by colleagues from civil society, USAID, Moldova’s e-government center and the library community.

There’s considerable momentum behind e-government in the country now, and the e-government center is getting ready to roll out nearly two hundred new online services. The challenge is ensuring demand for these services so that they become institutionalized – streamlining government and making it more transparent and responsive, while strengthening the sustainability and viability of e-government as an approach for Moldova. While there was discussion of hiccups with past efforts, the group was optimistic about the viability of libraries as a delivery mechanism. Much of the focus in the past has been on supply – the creation of services – and there is recognition that it’s equally important at this point to consider the demand side of the equation.

There was discussion around ideas on the types of services that would serve as ‘easy wins’ on both sides – both easy to create and sustain from the e-government side, but also easy to explain and use from the user side. There was acknowledgement that such early services must be carefully considered to fill a gap in information and deliver a tangible benefit. Paul-Andre Baran, the director of the Biblionet program in Romania, shared the example of libraries’ cooperation with the Ministry of Agriculture in his country, which is connecting farmers to online subsidy applications through libraries. Whereas this application process used to take up at least two days of a farmer’s time in travel to the local government office to get information and then return with a completed application, the online version completed at libraries saved Romanian farmers more than 34,000 days in 2011. More than 20,000 farmers collected more than €15m in subsidies by visiting their libraries.

Participants in the salon related to this example as illustrative of the efforts that need to be made in Moldova and thought agriculture might be a good place to start. The Ministry of Agriculture has a wide-reaching extension service as well as a payments agency similar to the one in Romania, but access to information for farmers is still a challenge. With agriculture making up a significant part of the Moldovan economy, services that can improve farmers’ lives are high on the agenda when considering public support and uptake. Other services prioritized by the group included online access to documentation and registration, broader tax filing services (limited online tax filing is already available), communication with local government and education information. One point made was that the most important services are the ones currently most captured by corruption. With Moldova currently in 112^th place on the Transparency International Corruption Perceptions Index, this is a salient issue.

A representative from the Moldovan Library Association expressed that Moldova’s libraries are ready to step up to the challenge. Previous projects have brought computers and new services to libraries based on the initiative of librarians, though in many cases, equipment has languished due to lack of local government support. In the upcoming program, she believed that government support was the key to sustainability – an aspect that’s baked into the Global Libraries approach.

There is clearly a long way to go in making open government and e-government a reality for Moldova. Discussions like these, where stakeholders and implementers with resources gather to exchange opinions and learn from each other are essential to ensuring the effort succeeds for the benefit of all Moldovans.

Read Full Post »

Libraries in Kenya

Kenya has become an impressive lab for ICT4D initiatives. In the Nairobi airport’s arrival hall, the glow of orange and white Safaricom signs is a reminder of one of Kenya’s most successful innovations, M-Pesa. Safaricom introduced M-Pesa in 2007, a mobile system that allows users to track their finances, make payments, and transfer money with the ease of a text message. While Kenya resonates with the mobiles for development community, Kenyan libraries do not have such recognition.

This week, I’m in Kenya for discussions about libraries and what role they can play in an environment with little of a reading culture, low computer literacy, and 99% of internet traffic taking place through mobile devices. I’ll be blogging about the release of EIFL’s report on perceptions of Kenyan public libraries as well as discussions with members of the Kenyan government, development, technology, and library communities involved in lowering barriers for accessing information.

Somewhere along an aimless amble around Nairobi yesterday, I found myself walking in stride next to James. James is attending university to become a tour director and was eager to speak to someone who appeared to be connected to Kenya’s tourism industry. After we played the guessing game about where I’m from, I explained that I’m here not for a safari, but to study Kenyan libraries. He was perplexed why I would want to go to a library if I was not a student, but he proudly explained that his home village does have a library. He has visited his town library once.

In the context of EIFL’s report, James demonstrated common attitudes towards libraries but does not represent the 72% of Kenyan males who are library users.  Furthermore, 87% of library users visit the library at least weekly.  James associates libraries with students, just as over 90% of Kenyans see educational purposes as the primary reason for visiting a library. Perhaps he was being eager to please someone who clearly had a connection to libraries, but James’s pride over his town’s library would be typical as well. Libraries are considered essential or important to the community by 98% of users and 93% of nonusers, but nonusers see the library as less essential to them as an individual.

The fact that James’s town does have a community library is fairly significant if it is one of the 58 libraries associated with the Kenya National Library Service (KNLS). To compensate for the limited reach of KNLS, libraries and community centres have sprung up across Kenya thanks to the dedication of local organizations. The total number of Kenyan libraries, including those not networked with KNLS, is unknown.

I’m looking forward to tomorrow’s event – a discussion of the EIFL findings – to learn more about libraries in Kenya.  And to make sure I’m interpreting the statistics correctly!

(November 7 @ 1:15 PM EST Updated with corrected statistics.)

Read Full Post »

Tech Salon / Sustainable Public Access to Info (event alert!)

The upcoming November 9 Technology Salon will focus on sustainable approaches to public access to information and will feature Sandra Fried, program officer in the Global Libraries program at the Bill & Melinda Gates Foundation, and Catalina Escobar, director of Makaia, part of the Digital Medellin project.

From the event description:

Access to information has been part of the development discussion since the Internet arrived. Previously, many saw community telecenters as the way to bring technology to the developing world. Yet telecenters are not sustainable without donor funding and the concept of public access hasn’t kept pace with advancing technology.

The global penetration of mobile phones calls into question the need for public Internet access at all. Until you realize that mobile devices are limited in functionality and there is more development information than is convenient for a phone screen – such as government open data and transparency initiatives.

So the question remains: how can people participate? It is time to reconsider the question of public access. What works today? What makes most sense for the future?

Full details and RSVP information.

Read Full Post »

Libraries and Open Government

We’ve followed with enthusiasm the new global focus on open government. Barack Obama launched the Open Government Partnership in New York in September, an international initiative which aims to highlight the importance of public information and transparency in democratic governance. Among the countries who have applied to join the partnership include Ukraine, Romania and Moldova – three countries where we are working on library development programs.

Our colleague Anna Taranenko from the Bibliomist program in Ukraine recently attended the Open Government Data camp in Warsaw, where she networked with activists from around the world on the issues and challenges surrounding open government. At the camp, she encountered amazing people from around the world working to help governments become more transparent and engage more with citizens. She noted some common challenges, however, especially in developing countries. Uptake is an issue. Despite the resources being put into public data and websites that give unprecedented visibility to government spending and statistics, there’s some disappointment and confusion about how to get people beyond the few activists making use of the new tools.

This matches what we’ve been noticing about much of the discussion around open government. It focuses on technology, on tools, and on data. Governments often see releasing public data as the final step, or hackathons and tech camps as a solution for the question of how information is used. There’s no questions all these pieces have their place, but there’s a gap in the discussion. What about access? If we’re talking about developing countries, where most people don’t have their own computers or smartphones, how are they being included in these efforts?

Opening a new public access point at Plebanivka library in Ukraine. Young patrons and a local official surf the internet.

We’re thinking about how libraries belong in this discussion. In countries with public library systems, institutions dedicated to the idea of access to information already exist. With the right technology and training, we’ve seen how libraries can become community centers where those otherwise excluded can become aware of, access, and learn how to use the information that’s available. They get a chance to participate in development process that they’ve never had before. If you’re talking about open government, you can’t just be talking about the hackers and the data. You have to be talking about access.

Read Full Post »