<h1><a href="http://programminghistorian.org/lessons/data-mining-the-internet-archive">Data Mining the Internet Archive Collection</a></h1>

My name is Daniel Ruten, and I have just finished my undergraduate studies majoring in History at the University of Saskatchewan. During my last term, I took a course on Digital History (HIST396) with Dr. Jim Clifford. In it, I became familiarized with the various emerging digital tools and methodologies that are becoming increasingly important for historians to learn. The course also required that I create some kind of digital history project myself. For my project, I took inspiration from one lesson in particular featured on the Programming Historian website: [historian Shawn Graham’s lesson on data sonification](http://programminghistorian.org/lessons/sonification). Building off of what this lesson taught me, I developed my own method to represent and analyze textual data through sound, which I have termed Sonic Word Clouds. In this post I will briefly explain this method of sonification, while reflecting a bit on the learning process that both inspired the idea for this project and allowed me to make it a reality.

At first, I was somewhat at a loss of what I should do for my digital history project. While scanning the various lessons on the Programming Historian website in order to get some ideas, one lesson in particular caught my eye. It was entitled “The Sound of Data (a gentle introduction to sonification for historians),” by Shawn Graham. I hadn’t really known that data sonification was a thing, much less a method that could be used by historians, and so my curiosity was piqued. In [the lesson](http://programminghistorian.org/lessons/sonification), Graham provides an introduction to the theory of data sonification, before detailing some of the sonification methods and resources that are available to historians. In particular, he discusses some methods to convert historical data into MIDI notation that can then be mapped to instrumentation. Some of these tools, such as [Musicalgorithms](http://www.musicalgorithms.org/3.2/) and the [MIDITime package for Python](https://pypi.python.org/pypi/miditime), were particularly designed with time-series/quantitative data in mind. But Graham also provides an example of the potential of MIDITime to analyze historical texts, as he uses it to sonify topic modelling data from John Adams’ diaries. By [mapping the resultant MIDI data to different instruments in Garageband](https://www.youtube.com/watch?v=ikqRXtI3JeA&feature=youtu.be), he offers us a means to hear the relative occurrence of different topics in the diaries over a 50-year period. Being able to listen to the relations between these different concepts over time this way provided a very unique and intriguing representation of a textual narrative. As Graham emphasizes, the choices one makes regarding how to represent data sonically in this fashion reveal the ways in which we privilege, condense and transform information as historians. The lesson gave me an idea: what if there was a way to sonify this kind of textual data in a more readily intelligible way? This set the stage for my own project.

Here at the Programming Historian, we have a number of lessons focused on "[distant reading](https://programminghistorian.org/lessons/?topic=distant-reading)." These lessons pull from a variety of fields to demonstrate different ways to computationally surface patterns across a large collection of digital objects. But how do you build on those patterns as part of a research project? That question of what to do next is what the authors of this post have set out to answer.

If you are interested in learning how to use collocations and keywords in your own research, we recommend starting with [Corpus Analysis with AntConc](https://programminghistorian.org/lessons/corpus-analysis-with-antconc) by Heather Froehlich. In this lesson, Froehlich introduces techniques from corpus linguistics, showing how to identify significant patterns of language use within and between sets of texts. And, as always, if you have an idea for a lesson or want to get involved with the *Programming Historian,* please visit our [contribute page](https://programminghistorian.org/contribute) for more information.

[CLiC](http://clic.bham.ac.uk) (Corpus Linguistics in Context) is a web app specifically designed for the corpus linguistic study of literary texts. While CLiC shares much of its functionality with other corpus tools — similarly to what is described in the [Programming Historian’s lesson ‘Corpus Analysis with AntConc’](http://programminghistorian.org/lessons/corpus-analysis-with-antconc) — it also contains additional features that are particularly relevant to literary analysis. These include the ability to search subsets of the text – such as character speech – and a sorting function that goes beyond alphabetic sorting: the ‘KWICGrouper’, which this post focuses on. The CLiC web app has been developed as part of the [CLiC Dickens project](http://www.birmingham.ac.uk/schools/edacs/departments/englishlanguage/research/projects/clic/index.aspx) for the analysis of patterns in 19th century fiction, particularly novels by Charles Dickens. CLiC currently contains 15 Dickens novels and 29 novels by other 19th century authors and a corpus of 19th century children's literature will soon be added.

Since its creation, the Programming Historian’s [Spanish Language Team](https://programminghistorian.org/posts/announcing-new-team-spanish-language-editors) has translated [28 tutorials](https://programminghistorian.org/lessons/) into Spanish with the support of a network of collaborators from countries such as Argentina, España, Colombia, and México. Building on our commitment to diversity and access, the Programming Historian is seeking a new team member to contribute to the Spanish Language Team. The editor will contribute to the ongoing translation efforts and will support the community building strategies to increase access and collaboration in the Spanish-speaking world.

_Programming Historian_ authors are nothing if not ambitious. That's great, but sometimes we receive pitches for lessons that simply want to do too much, such as teach an entire programming language, or cover everything one needs to know about a category of research. Some of these ideas would make better textbooks than tutorials. If you are too ambitious with the scope of your lesson, chances are you'll have trouble writing it. A good lesson does one thing well. Take a look at our [published lessons](http://programminghistorian.org/lessons/) for a sense of what's possible.

Interest seems to be focused on the tutorials that provide Python programming skills ('[Counting Word Frequencies with Python](https://programminghistorian.org/lessons/counting-frequencies)', '[Working with Text Files in Python](https://programminghistorian.org/lessons/working-with-text-files)', and '[Creating and Viewing HTML Files with Python](https://programminghistorian.org/lessons/creating-and-viewing-html-files-with-python)') top the list, with a lot of interest also in '[Introduction to Mobile Augmented Reality Development in Unity](https://programminghistorian.org/lessons/intro-to-augmented-reality-with-unity)' and '[Building a static website with Jekyll and GitHub Pages](https://programminghistorian.org/lessons/building-static-sites-with-jekyll-github-pages)'. This suggests two things: first, that a cohort of individuals in or around the Indian "tech industry" are using our lessons to develop their programming skills; and second, that open access Python tutorials fill a gap in the market.

[live version here]: http://programminghistorian.org/lessons/automated-downloading-with-wget#lesson-goals