In part 2 the chart app was successfully loaded with historical data using appcfg.py’s bulkloader tool and the remote_api module. However as queries over time series data quickly become expensive when we begin to deal with months and years, we should use some kind of caching mechanism to prevent too many datastore operations from slowing down our chart display.
App Engine’s powerful built-in object caching framework is memcache. With memcache we first check if the data we’re looking for is in memory and if so we will retrieve it from there, removing the need for any datastore querying. If not we will retrieve it using GQL as done in part 2 of the tutorial.
so for our 5 day data our datastore code in main.py would become:
Here I will walkthrough pre-populating the Google App Engine datastore with historical data in CSV format using the python bulkloader tool to do the data transformation and import.
In the last tutorial we seen how to create a financial chart with Google App Engine & update it periodically via some css-based web scraping. The bulkloader tool that comes in the appcfg.py script will enable us to visualise long term trends *right now* without needing to wait on our datastore growing one day at a time. Again the 3 Month EONIA Index Swap from the European Banking Federation’s Euribor site will be used as sample historical data.
So we need to grab the EONIA historic data. It’s available in CSV already but additionally I needed to:
This is a short 3 part tutorial series that will guide you through how to create & host your own financial charts on Google App Engine.
To begin we’ll see how simple it is to create a web scraper that uses CSS selectors and string manipulation to grab whatever data you want from a website. You can adapt the code to target whichever financial instrument you require. We will use a python app instance on Google App Engine to host the code and its scheduled tasks API to grab the required data periodically and persist to the datastore.
lxml is a python web scraper that uniquely uses CSS to query HTML documents. This makes it ideal for web developers and designers who are already familiar with the selector syntax.
If you haven’t already done so, sign up to Google App Engine, create a new python 2.7 application and download the SDK. Also ensure you have python 2.7 installed locally together with the 2.7 version of the lxml library.
For the purposes of this example we’ll seek to regularly poll the Euribor site and scrap the daily EONIA (Euro Over Night Index Average) 3 Month Swap fixing.
22 weeks in a Helmand Checkpoint.
One year ago this week I was lying face down in a ploughed field with 7 others while a Taliban ambush opened up on us from about 150m away. This time the rounds were falling a bit closer than usual. Both lads in front of me had near misses – the first one had a round graze the inside of his forearm while the guy immediately in front of me managed to stop one with his helmet, the round ricocheting harmlessly into the ground in front of him. Afterwards he said it was like getting hit with a sledgehammer.
“Paddy, Paddy, I think I’ve been shot in the head!” Was the memorable line as we hugged the dirt. I crawled towards him, trying to remember how exactly you were meant to treat bullet wounds to the brain. Feeling around his headgear for, well, cracks & blood but relievedly found nothing I told him to stop being a fanny and we took it in turns to get into hard cover. The Taliban’s fire had died off and we soaked in, but the feeling was one of resignation – another fleeting unseen enemy attack with no desire to take us head on in a firefight. Or so we thought.
Following requires fixing:
- 3D glasses.
- the complete fragile ineptitude of Sony earbuds.
- those social newspaper middlemen on facebook that try to force you to share that article on enhanced woman/donkey bonding before you can read it.
- The Google Analytics UI is the most unintuitive experience I’ve had to endure on the web.
- all white tech goods. Apart from original Game Boy.
- haptic feedback on my Sony Ericsson Mini Pro that doesn’t work when my fingers are sweaty or it’s wet. My phone is effectively indoors-only in Belfast.
- Google’s secondary search box has disappeared from the bottom of results page requiring additional scrolling for no obvious gain.
- lack of cross-device cut & paste.
- ‘Google Docs capturing of the right click event destroys Chrome’s ‘search for’ highlighted text function.
- ‘playstation certified gaming’.
We may be the only people in the world who can say our goal is to have people leave our website as quickly as possible.
Google Company Philosophy, Ten things we know to be true
With the launch of the ‘knowledge graph’ and the subsequent decrease of outbound links as a percentage of the total on Google search result page, is this still their stated goal?
Granted the two concepts are not necessarily mutually exclusive, but it is difficult to see in the general case how a user who starts off with a single query does not spend more time on Google.com if they are encouraged to view more search pages than they would have done otherwise.
Or reasons why Python is cool #273.
There are many snippets of code on the internet detailing how to go about dynamic class creation when the compiler (and thus, you the programmer) knows the name and module location of the class some time in advance, e.g. initial deploy time or via some parameter being entered in by the user when they want to instantiate such a class.
However if I want to create objects without knowing their module/class in advance a little bit more thinking is required. Here is a simple tutorial that will leave you with a package that when updated with new modules will introspect & instantiate their classes on demand.
I develop on my local Windows 7 (or XP netbook when out of town) and to test/deploy projects I need to send them to my Gentoo Linux server. While manual copying of files quickly becomes a pain in the arse a fully fledged continuous integration environment is needlessly complex for a single programmer project so I created the following batch script that once run checks my windows folders for new & updated files since the last time the script was run then sends them across the ether using scp1.
The script has been tested on XP & 7 successfully.
For the linux side of things I run a python paster instance with the —reload flag that means paster will pickup changes and restart automatically, making deployment from my windows box a single click action.
Initially I started out a little rusty in my command line knowledge
Dropping the old ICT curriculum with it’s much maligned emphasis on M$ Office is a brave move by education minister Michael Gove. It could be argued that from purely a learning point of view this one action is more important than any of the recent decisions on increasing education cost. Sending children back to where it all began – the formulation of grammars and statements thereof into machine-readable instructions that produce new computing tasks rather than just learn old ones by rote – is in many ways fundamental to getting a British engineering discipline back on track.
But it’s easy to get carried away here.