attribute: Phillie Casablanca

lessons (2014)



Mark Maron mentioned the Woody Allen quote, "80 percent of life is showing up". It stuck with me much of this year. Here's a bit more detail:

"I made the statement years ago which is often quoted that 80 percent of life is showing up. People used to always say to me that they wanted to write a play, they wanted to write a movie, they wanted to write a novel, and the couple of people that did it were 80 percent of the way to having something happen. All the other people struck out without ever getting that pack. They couldn’t do it, that’s why they don’t accomplish a thing, they don’t do the thing, so once you do it, if you actually write your film script, or write your novel, you are more than half way towards something good happening. So that I was say my biggest life lesson that has worked. All others have failed me." -- Woody Allen

There is a saying among jazz musicians:
"If you're not the worst musician in your band, you should immediately switch bands."

Lots of questions this year. Should I stay or should I go? Is there something left to learn? I have the room to learn on my own, but I've realized that that alone is not enough. There needs to be a shared purpose, a goal, to strive for. Left in limbo with no clear direction is something to be avoided.

As humans we live our lives through stories. The stories we tell ourselves, and the stories we tell the world. Stories drive our interactions, and are key to communication. In order to express your thoughts and ideas to the world clearly, storytelling is an important skill to have.

There is a depth to everything. A thing spiral away forever and can never be fully 'known'. At some point you need to make a decision how deep you want to go.

Diversity is what makes us great. It allows new insights and approaches to questions that can't be reached alone. We are limited in time by what we can experience. It is a benefit that others have had a different path to now. Our experiences are shared and combined lives enriched as a result.

Doing, even once, makes it easier the next time. That novel? It can only be written by putting words down. One more word puts you that much closer to the end. It is the not doing that keeps you from progressing. Progress requires motion. action. Without it your not going anywhere.

You determine your normal. Conscience or not you make decisions daily that determine what your normal is. Realize this is a choice, and make a good one.

Culture is something that together we create. It does not exist without people. Your actions define your culture. If you want the culture to change, start with yourself. Be the person of the culture you desire.

Clearly understand the goals. Work toward the goals. When you get lost, ask yourself, does this get me closer?

Always know that you may be wrong.

monkut // Dec. 6, 2014 // 11:35 a.m.

The continuum of good to great



It's a struggle. A balance really. Each time sacrificing something in order to achieve something else. Most often it's time, but that is only the high level obstacle. It's the foresight into how much time it would take to achieve the ideal. It's the choices taken along the path to the ideal that makes the goal more and more obtainable.

It's knowing what sacrifices are reasonable to take. The ones that allow you to push still closer to the ideal, but at the same time make the goal obtainable.

I wonder how to know when something is great. Perhaps, if your asking, you're not there. Good. Good is easy. It's that extra effort, that extra insight, that pushes something from good closer to great.

When is something great? Not when you find yourself reluctant to share it. At the same time, however, keeping something locked away from the world may keep it from judgement, but also from knowing greatness, or allowing you to discover a previously unseen path. A step, that moves you further down the continuum toward greatness.

Why, when good is good enough, should you struggle on toward greatness? I guess it all depends on the end goal you seek.

monkut // Dec. 6, 2014 // 10:45 a.m.


Safecast timelapse



Occasionally, I hear people still talk about the radiation in Fukushima. It's been over 3 years since the accident, and you rarely hear anything mentioned on the news anymore.

However, safecast.org has done a great job of collecting on-going radiation measurements, and publishes a number of maps. I've done similar maps over the years and decided to tackle making one in an attempt to answer the question, "how have the values changed over time". (I'm also taking the opportunity to do this as my first python3 django project)

So, the first step is to get the data.
Go grab the CSV at http://blog.safecast.org/data/

I'm then averaging the data monthly on load into geographic bins. For the timelapse, I'm using 1500m bins. For each month-area, the data is using simple average of all values in uSv/hour that fall in that area. I'm not sure if this is the best way to handle it, but the result appears to be close to the 'official map', so for now I'll stick with the simple average.

Note that some of the data is in CPM and not uSv/hour. For these values I converted using the following method:

def cpm2usv(cpm_value):
"""
Using chart at:
http://nukeprofessional.blogspot.jp/2012/04/geiger-counter-interpretation.html
"""
usv_per_click = 0.1/12
return cpm_value * usv_per_click

Once the data is loaded into the django Measurement models, it's time to generate the map tile png files. For this I created a python3 library, tmstiler (https://github.com/monkut/tmstiles) , which contains a class, DjangoRasterTileLayerManager, that generates the tile images from a django model containing raster data as a PointField.

Using the DjangoRasterTileLayerManager() minimizes what needs to be done in django project view code. I think the legend code ended up being bigger than the tile creation view code.

The first time I created a timelapse, the image was only showing the 'current' month's data, and made the image appear choppy. There are a number of ways to do some kind of interpolation to make this look better. I decided to try the easiest thing I could think of, hold the previous month data and show it in the 'current' month image. However, I also wanted it to be clear, in some way, that the previous data is not the same as the current data. At first I thought I could adjust the transparency and gradually fade out the older data. After looking into this, it seems that with PIL (pillow) this is a bit difficult to do, and can't be done on color assignment alone. So instead I'm adjusting the color saturation level. (For legends and color manipulation HSL http://en.wikipedia.org/wiki/HSL_and_HSV makes shifting colors much easier than dealing with RGB alone)

So I've defined a "months_to_actual" field in the Measurement model, and use this value in the legend to adjust the color saturation. The effect is that the color gradually shifts to gray before being removed. Visually, I'm pretty happy with the results. At the moment it's holding the past 6 months values.

For building the time-lapse, image capture is done using phantomjs (http://phantomjs.org/). I've used it in a previous project to capture data on a headless server, it's basically a programmable webkit browser without the need for a GUI. This time around I did notice that there is a python solution called ghost.py, but I think it's only python2 and the installation is more involved than using phantomjs.

Next, the images are combined together into a video using avconv. This is all done on ubuntu 14.04 server.

I'd like to get the map "live" and browsable here, but I figure that's at least a couple of nights work, and if I do that I really want to do periodic automated updates using the safecast api, which would be a bit more work...

In any case, if anyone is interested in trying it out or making it better, I've posted all the code to github.

https://github.com/monkut/safecasttiles

https://github.com/monkut/tmstiles


You can see the resulting time-lapse video at:

https://plus.google.com/114056874915345387620/posts/YkUsdcvejXY


monkut // Dec. 5, 2014 // 12:04 a.m.

OSMデータから他言語をpostgresにロードする方法



ubuntu 14.04を使用



Background


最近、マップ上にデータを表示するためのマップほしいと思ってきて、時間があったので、作ろうと決意した。


検索した結果、やっぱり、同じことを考えている人がいた!
https://www.mapbox.com/blog/designing-minimalist-openstreetmap-baselayer/


以前のプロジェクト、オフラインのマップサーバを作るために、OpenStreetMap (OSM)のデータをダウンロードして ( http://download.geofabrik.de/asia/japan.html )、使って、mapboxのTileMill ( https://www.mapbox.com/tilemill/ ) にロード、マップサーバのタイルを作成した経験があった。


上記のブログをみるとそのように作っているようですが、結果のデザインをGITHUBなどで残していないようだ。
よんでみると、他人のOSM Bright ( https://github.com/mapbox/osm-bright )をテンプレートとして作ったようだが、
あっちこっち色の設定が多いので、探してみたら、すでに白黒のものがあった!


https://github.com/aaronlidman/Toner-for-Tilemill


これが、オンラインマップデザインの王様Stamen Designの"toner" ( http://maps.stamen.com/toner/#12/37.7706/-122.3782 )を真似してつくったようだ。
使いたい目的には、ちょっと暗いが、これだったら、色調整が少なくて済むと思って、Toner For Tilemillをベースにして進んでいる。


さって、調整していくと気がついた。都市名がすべてローマ字。。。




OSMデータからマップ上に日本語の表示へ


OSMのデータをみてみると、日本語が入っているしているようだ。


とりあえず、準備する方法を下記


OSMデータを http://download.geofabrik.de/asia/japan.html からダウンロードしたら、osm2pgsqlを準備する。


https://github.com/openstreetmap/osm2pgsql


osm2pgsqlをGITHUBからPull:
(まだ、gitがインストールされていない場合、sudo apt-get install gitでインストール)


git clone https://github.com/openstreetmap/osm2pgsql.git


そこで”default.style”をコピーして、編集:


cd osm2pgsql
cp default.style osm2pgsql.style


osm2pgsql.style内に、下記の行のしたに



node,way name text linear

これを追加:



node,way name:ja text linear


Note


OSM tagについて下記のリンクへ参照
http://wiki.openstreetmap.org/wiki/JA:Map_Features



修正が終わったら、dbを作って、DBへロードする:



sudo -u postgres pgsql -c "CREATE DATABASE osm_db;"
sudo -u postgres pgsql -d "osm_db" -c "CREATE EXTENSION postgis;"

sudo -u postgres osm2pgsql -c -G -S ./osm2pgsql.style -d osm_db ~/maps/japan/japan-latest.osm.pbf -C 22000


これでDBに必要なデータがロードされる。
しかし、これを地図上に使うなら、簡単には、Tilemill内で、DBレーヤを追加することなんだが、
気がついたら、Toner for Tilemillには、都市名がきれいに表示されるように、既に処理して、ズームレベルごとに、SHPファイルに用意してある。


つまり、都市名が重なってしまう。


同じようなきれいな表示するなら、Dymo ( https://github.com/migurski/Dymo/ )があるようだ。
それを使って、結果のgeojsonを作ってみたんだが、なぜかファイルをロードしてくれない。。。


ちなみに、osm2pgsqlで作ったDBからdymoの入力CSVを作成してくれるスクリプトをつくってみた:


https://github.com/monkut/dymo_label_prep


(そのスクリプトにあるSQLQueryをレーヤとして、できれば、日本語レベルの表示問題が解決されるかも。。。。)



monkut // Aug. 23, 2014 // 10:42 a.m.

The New Open Data Revolution?



For years Tim O'Reilly has been saying that "Data is the Intel Inside". When I first heard this I didn't understand, but gradually I came to see the importance and strength of data, but from the business perspective. Companies, like google, use, process and transform data into something that provides value.


I've just read, "Day the Universe Changed", and I'm now seeing the statement in a new light. Since, O'Reilly's original Web 2.0 Manifesto, not only has the value of data become more realized, but data has also become more open. Companies like EveryBlock have lead the opening and availablity of public data, and even data.gov, a portal for government data, has been created.


What does this opening of data mean? It could be more than any of us can comprehend. I see this as a new opening of knowledge to the public. Just as the Gutenburg's printing press solidified "fact" and moved knowledge and awareness from elders and institutions of authority, open data extends the avaliable public knowledge.


With this tsunami of data, the public needs the knowledge to understand it. In the pre-printing press world, knowledge was spread by word of mouth, there was no choice but to understand and accept the world as we and our community perceived it. The printing press and the written word extended our perception of the world, so that we could picture a world beyond that of our life and our local social circle, increasing the resolution of our perception of the world. Now, that resolution is set to increase again.


With the availbility of data, Governments, analysts, and experts are no longer the solitary keepers of knowledge. When this happens data literacy becomes more important. The public will need basic statistical knowledge in order for meaningful conversation to occur. As data becomes more open, the number of "experts" will explode, and some of these new "experts" will, not unlike authority figures of the past, take advantage of those who lack the basic "data literacy" skills. While I suspect schools are struggling to teach basic math skills, in order to start these meaningful conversations, our students need to have a minimal level of data comprehension.


With the spread of data literacy, we may move from rehtorical debates where the keys of persuasion are emotional to debates of data analysis and validaty. These data debates are being had, but until now it was only by the keepers of the data.


I urge academics and researchers to open and share data and data keeping methods. I sense that there still persists a sense of ownership in the academic community. Understand with the opening of data, and more examination and discussion we are one step closer to a better understanding of the world around us.


monkut // Feb. 16, 2013 // 10:10 a.m.