Web Science and Digital Libraries Research Group

It is hard to believe that the time has come for me to write a wrap up blog about the adventure that was my Masters Degree and the thesis that got me to this point. If you follow this blog with any regularity you may remember two posts, written by myself, that were the genesis of my thesis topic: 2017-01-20: CNN.com has been unarchivable since November 1st, 2016 2017-03-09: A State Of Replay or Location, Location, Location Bonus points if you can guess the general topic of the thesis from the titles of those two blog posts. However, it is ok if you can not as I will give an oh so brief TL;DR;. The replay problems with cnn.com were, sadly, your typical here today gone tomorrow replay issues involving this little thing, that I have come to , known as JavaScript. What we also found out, when replaying mementos of cnn.com from the major web archi...

I have written posts detailing how an archives modifications made to the JavaScript of a web page being replayed collided with the JavaScript libraries used by the page and how JavaScript + CORS is a deadly combination during replay . Today I am here to announce the release of a suite of high fidelity web archiving tools that help to mitigate the problems surrounding web archiving and a dynamic JavaScript powered web.To demonstrate this, consider the image above: the left-hand screen shot shows today's cnn.com archived and replayed in WAIL, whereas the right-hand screen shot shows cnn.com in the Internet Archive on 2017-07-24T16:00:02 . In this post, I will be covering: Updates to WAIL Release of node-warc Release of node-cdxj Release of Squidwarc WAIL Let me begin by announcing that WAIL has transitioned away from using Heritrix as the primary preservation method . Instead, WAIL now directly uses a full Chrome browser (Electron provided) as the pres...